EXAMINING THE ACCEPTABILITY, ADOPTABILITY, AND APPROPRIATENESS OF A SCHOOL-BASED EXTERNAL MENTAL HEALTH REFERRAL PROCESS: A QUALITATIVE PRE-IMPLEMENTATION STUDY By Erica Ashley Nordquist A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Human Development and Family Studies—Doctor of Philosophy 2019 ABSTRACT EXAMINING THE ACCEPTABILITY, ADOPTABILITY, AND APPROPRIATENESS OF A SCHOOL-BASED EXTERNAL MENTAL HEALTH REFERRAL PROCESS: A QUALITATIVE PRE-IMPLEMENTATION STUDY By Erica Ashley Nordquist Schools have been identified as a primary place where youth mental health issues are initially noticed. When schools collaborate with mental health service providers to refer youth to care, these youth experience improvement in both educational and clinical outcomes. However, such collaboration can be challenging. This dissertation explored the acceptability, adoptability, and appropriateness of implementing an external mental health referral process within several schools in an intermediate school district. Qualitative key informant interviews were used to identify barriers and facilitators to implementing an external mental health referral pathway. Key informants included school and agency personnel. Twenty-three themes and three subthemes were identified. Results indicated that implementing an external mental health referral pathway was mostly acceptable, adoptable, and appropriate. Implications for practice, research, and clinical practice are discussed. TABLE OF CONTENTS LIST TABLES LIST OF FIGURES CHAPTER ONE: INTRODUCTION Rationale Purpose Theoretical Framework Exploration Phase Outer context CHAPTER TWO: LITERATURE REVIEW Sociopolitical context Funding Client advocacy Inter-organizational networks Inner context Organizational characteristics Individual adopter characteristics CHAPTER THREE: METHODS Research Paradigm and Qualitative Approach Researcher Characteristics and Reflexivity Context Sampling Strategy Ethical Considerations Confidentiality Potential risks Potential benefits Data Collection Methods vi vii 1 1 2 3 7 9 9 9 10 11 13 14 14 15 17 17 17 18 20 21 22 22 22 22 22 23 23 23 23 23 24 24 25 26 29 30 Recruitment Informed consent Interview Demographic form Interview guide Data Collection Instruments and Technologies Units of Study Data Processing Data Analysis Techniques to enhance Trustworthiness CHAPTER FOUR: RESULTS Outer Context iii Sociopolitical context Barriers Referring to outside providers depends on insurance Grant funding has allowed for more resources Funding Barriers Inter-organizational networks Barriers Lack of provider capacity The “right type” of training Long waiting lists Current network of providers could be expanded Technology Different procedures between agencies Navigating confidentiality amongst partners Facilitators Meeting consistently amongst community partners Agency collaboration Agency personnel providing services in schools Some schools and agencies already have an external mental health referral pathway in place Inner Context Organizational characteristics Barriers Facilitators Changing a system Lack of understanding about mental health issues Lack of resources Interdisciplinary meetings to review students Administrative support Mental health staff as a resource Increased awareness of mental health issues Whole child approach Individual adopter characteristics Barriers Lack of connection with parents can prevent connecting youth with mental health services Facilitators Value their role in supporting youth and families Having professional networks enhances individuals’ ability to connect youth with mental health services Youths’ individual needs are used to determine appropriate services Stakeholders believe an external referral pathway would be helpful iv 31 31 31 32 32 32 32 33 33 33 34 35 35 36 36 37 37 38 39 39 40 41 41 41 41 42 43 43 44 44 45 45 46 46 46 47 47 48 49 50 CHAPTER FIVE: DISCUSSION 51 51 51 52 53 53 54 55 56 58 59 60 61 62 62 63 64 65 66 67 68 69 70 70 71 73 74 77 78 80 81 82 83 84 RQ1: What do stakeholders find acceptable about implementing a systematic external referral pathway? The idea of a systematic external referral pathway is “fabulous” Some schools and agencies already use an external referral pathway RQ 2: What barriers do stakeholders anticipate to implementing a systematic external referral pathway? Changing a system Technology Having up to date information about providers There is an overall lack of providers in the county Lack of understanding about mental health issues Navigating confidentiality Lack of resources Lack of connection with parents can get in the way of connecting youth with mental health services RQ3: What facilitators do stakeholders anticipate would aid in successful implementation of a systematic external referral pathway? Stakeholders believe an external referral pathway would be helpful Meeting consistently among community partners Interdisciplinary meetings to review students Grant funding has allowed for more resources Agency personnel providing services in schools Administrative support Mental health staff as a resource Increased awareness of mental health issues School and agency personnel value their role in working with families Having professional networks enhances individuals’ ability to connect youth with mental health services Implications for the field of Marriage and Family Therapy Limitations Conclusion APPENDICES APPENDIX A Telephone Consent APPENDIX B Demographic Questionnaire APPENDIX C Agency Personnel Interview Guide APPENDIX D School Personnel Interview Guide APPENDIX E Audit Trail Record REFERENCES v 6 25 29 LIST OF TABLES Table 1.1 Overview of conceptual framework, research questions, and interview questions Table 3.1 Demographics of participants Table 4.1 Summary of qualitative results vi LIST OF FIGURES Figure 1.1 EPIS Model adapted by Aarons et al., 2011 3 vii Rationale CHAPTER ONE: INTRODUCTION In the United States (U.S.), approximately 20% of youth between 13 and 18 years of age meet the DSM-V criteria for a severe mental health disorder (Merikangas et al., 2010). Of these youth with an identified mental health disorder, only 17% access and utilize mental health services (Kazdin, 2008). Consequently, the vast majority of youth with a severe mental health disorder are not accessing treatment. Outcomes for youth who have an untreated mental health disorder can be deleterious. Youth with untreated mental health disorders are likely to experience reduced school performance, truancy, engage in substance use, engage in risk-taking behaviors, and are at-risk for suicide (Brannlund, Strandh, & Nilsson, 2017; Holtes, et al., 2015; Larson, Chapman, Spetz, & Brindis, 2017; Meier, Hill, Small, Luthar, 2015; Moskos, Olson, Halbern, & Gray, 2007; Mundy et al., 2017). The impact of untreated mental health disorders in youth are likely to have effects into adulthood, especially for youth who have had adverse experiences. These effects can include depression, substance abuse, and increased risk of suicide attempts (Anda, Whitfield, Felitti, Chapman, Edwards, Sube, & Williamson, 2002; Chapman, Anda, Felitti, Dube, Edwrds, & Whitfield, 2004; Dube, Anda, Felitti, Chapman, Williamson, & Giles, 2001; Remigio-Baker, & Hayes, Reyes-Salvail, 2014). One important public health strategy to improve youth’s access to mental health treatment is the leveraging of schools (Rones & Hoagwood, 2000). Schools can play a key role in identifying youth with mental health service needs and connecting them with existing services (Fazel, Hoagwood, Stephan, & Ford, 2014). In this way, schools can significantly improve the referral pathways for these youth (SAMHSA, 2015). The Substance Abuse and Mental Health 1 Services Administration (SAMHSA) has even created a School Mental Health Referral Pathways Toolkit (2015) to assist schools’ use of referral pathways. Key components of the toolkit include assessing current referral approaches, building effective community partnerships, problem solving, and cultural and linguistic considerations (2015). Despite the potential for leveraging schools to improve referral pathways, such improvements require significant coordination due to the need for essential information sharing between providers, and the obstacles that prevent information sharing (Burns et al., 2004; Hamilton, Begley, and Culler, 2014; Husky, Sheridan, McGuire, and Olfson, 2011). For example, while federal legislation—including the Health Insurance Portability and Accountability Act of 1996 (HIPAA) and the Individuals with Disabilities Education Act (IDEA)—establishes confidentiality as a key practice in protecting student information, confidentiality is also a key barrier to effective communication among professionals (within, and outside of, the school) regarding students’ mental health needs (Stein et al., 2010). Purpose The intent of the proposed study is to explore key informant’s perceptions of the acceptability, adoption, and appropriateness of an external mental health referral process within an intermediated school district. The referral process is meant to connect youth who have been identified with a mental health concern to external mental health services. The guiding research questions for this study are: 1. What do stakeholders find acceptable about implementing a systematic external mental health referral pathway? 2. What barriers do stakeholders anticipate to implementation of a systematic external referral pathway? 2 3. What facilitators do stakeholders anticipate would aid in successful implementation of a systematic external referral pathway? Theoretical Framework Implementation science—an interdisciplinary field focused on studying and advancing approaches to the establishment of effective, data-driven practices in real world settings—can offer important insights to help researchers and practitioners identify and overcome the obstacles for the involvement of schools in referral pathways (Eccles & Mittman, 2006). This study will be guided by a conceptual model within implementation science--the Exploration, Preparation, Implementation, and Sustainability (EPIS) Framework (Aarons, Hulburt, Horwitz, 2001)— designed to guide the study and implementation of new practices in settings–such as schools and mental healthcare settings for children and families (Aarons, et al., 2011; see Figure 1). EPIS can be used to understand which contextual factors should be examined and monitored when implementing mental health innovations. Figure 1.1 EPIS Model adapted by Aarons et al., 2011 3 The four phases are of EPIS are: exploration, adoption/preparation, implementation, and sustainment. The exploration phase occurs when there is a growing awareness of an organizational challenges that needs to be addressed. The adoption/preparation phase occurs when an innovation is identified as a solution to the organizational challenge. At this time, the organization begins to prepare for implementation of the intervention. The implementation phase occurs when the innovation is implemented within the organization. The sustainment phase indicates continued use of the innovation. During each phase, a variety of factors in the outer and inner contexts of an organization influence the implementation of an innovation. Outer contextual factors can include the sociopolitical context, outside funding sources, advocacy efforts, inter-organizational networks, intervention developers and public-academic collaboration. Inner contextual factors include organizational characteristics, individual adopter characteristics, leadership, innovation fit, staffing, and fidelity monitoring (Aarons et al., 2011). Within this study, I used the EPIS Framework to guide interview question development and explore the acceptability, adoptability, and appropriateness of a referral pathway. Acceptability as stakeholders’ satisfaction with an intervention. Adoption refers to an organizations intention to try an intervention. Appropriateness measures stakeholder’s perceived “fit” of the intervention (Proctor et al., 2011). Guided by the exploration phase of the EPIS framework, in the proposed study I explored inner and outer contextual factors that may serve as barriers and facilitators to the implementation of a school-based external referral process. I implemented a design that closely follows the protocol of previous studies using EPIS for similar purposes (Aarons et al., 2012). I interviewed key informants to conduct a system level assessment, organization-level assessment, and provider level assessment. The information gathered during the system-level assessment 4 helped me determine the conditions necessary to implement a school-based external referral process. I used the organizational-level assessment to identify logistical and practical concerns related to uptake of the school-based external referral process. Finally, the provider level assessment helped identify individual factors impacting implementation (Aarons et al., 2012). Table 1.1 provides an overview of the conceptual framework, corresponding research questions, and sample interview questions. 5 Table 1.1 Overview of conceptual framework, research questions, and interview questions Theory Research Question(s) Sample Interview Questions EPIS- Outer context (inter-organizational networks) EPIS- Inner context EPIS- Outer context (Inter-organizational networks) EPIS- Outer context EPIS- Inner context/Outer context (organizational characteristics/inter- organizational networks) EPIS- Inner context (organizational characteristics) EPIS- Inner context (organizational characteristics) What do intermediate school district key informants find acceptable about implementing a systematic external referral pathway? What do intermediate school district key informants find acceptable about implementing a systematic external referral pathway? What do intermediate school district key informants find acceptable about implementing a systematic external referral pathway? What barriers do stakeholders anticipate to implementation of a systematic external referral pathway? What are your thoughts about implementing an external referral process? How do student mental health needs fit with your [school/school district/ agency’s] priorities? Can you talk about how you make decisions about referring to an agency or clinician? What might interfere with the implementation of an external referral process? What challenges do you expect/encounter? What facilitators do stakeholders anticipate would aid in successful implementation of a systematic external referral pathway? What facilitators do stakeholders anticipate would aid in successful implementation of a systematic external referral pathway? What facilitators do stakeholders anticipate would aid in successful implementation of a systematic external referral pathway? What processes are already in place to address youths’ mental health needs? What supports are in your [school/agency] to make implementation of an external referral process successful? What additional supports are needed to make implementation of an external referral process successful? 6 CHAPTER TWO: LITERATURE REVIEW School personnel often recognize youth who have a mental health issue (Phillippo & Kelly, 2014). While many schools provide mental health services (i.e., school counselors, school-based health centers, integrated school mental health services), there are many schools that do not have the resources to provide mental health services to all students or the capacity to provide mental health services in the form that would be most benefical to the student. In order to access mental health services, youth and their families may need to be referred to other mental health providers in the community. Referral pathways are the steps taken after a youth has been identified with having a potential mental health issue (SAMHSA, 2015). The School Mental Health Referral Pathway Toolkit (SAMHSA, 2015) identifies five key components of community-school mental health partnerships: 1) merging resources across sectors; 2) a full continuum of care; 3) promotion of healthy development for youth across system levels; 4) selection of services available to best fit the youth and their families’ needs; and 5) provider reduction of barriers to access mental health services. Youth referred to mental health services via referral pathways from schools may experience improved outcomes when compared to peers referred in other ways. Using data from a national evaluation of systems of care, Greif Green and colleagues (2016) examined educational and clinical outcomes for youth referred by schools (n=3012) for mental health services, as compared to other referral sources (i.e., mental health agencies, juvenile justice settings, welfare, self-referral, family physicians, caregivers, and other sources; n=5005). They found that youth referred from school settings experienced decreases in school failure and school absences when compared to peers referred from other sources. Additionally, youth referred from 7 school settings experienced decreases in externalizing and internalizing symptoms as compared to their peers referred from other sources. Research that has examined school mental health referral pathways in the past have focused on mental health and academic outcomes (Grief Green et al., 2016), racial differences in referrals (Alegria et al., 2012; Guo, Kataoka, Bear, & Lau, 2014), or gender differences in referrals (Maschi, Schwalbe, Morgen, Gibson, & Violette, 2009). In contrast, only one study to date has examined factors related to the implementation of these pathways. Hall and Wurf’s (2018) evaluation of the capacity of a school referral pathway piloted across four schools in New South Wales is an important exception. These researchers utilized a mixed-methods approach including surveys (n=135), focus groups (n=32), and key informant interviews (n=23) to understand key stakeholder’s experiences with the referral pathway. Their findings demonstrated an overall reduced workload for teachers and school administrators, as well as the schools increased ability to manage at-risk students. Furthermore, students reported feeling more comfortable speaking about their problems with a counselor at the agency rather than a teacher. While this study clarifies some of the inner contextual factors related to the implementation of school referral pathways, it provides limited guidance for the implementation of these pathways. Referral pathways are an innovative practice; yet, are under researched, both generally and in terms of their implementation. Therefore, my review of implementation factors related to referral pathways will be guided by the exploration phase of EPIS. In this review, I will first describe the exploration phase of the EPIS’ model, and then discuss its two primary components --the outer context and the inner context—with particular attention to their likely role in the implementation of school referral pathways. 8 Exploration Phase The Exploration Phase of EPIS can be used to frame the examination of an organization’s awareness that there is a need for an improved approach to a specific challenge (e.g., a school’s awareness of the needs for an external referral process for mental health needs). According to the EPIS model, there are two primary components that should be considered during this phase: the outer context and the inner context. The outer context refers to factors outside of an organization that impact its implementation of evidence-based practices (EBPs). The inner context refers to factors within the organization that impact implementation. Outer context. Outer context factors include the sociopolitical context, funding sources, client advocacy, and interorganizational networks. Sociopolitical context. The outer context factors include the sociopolitical context, funding sources, client advocacy, and inter-organizational networks. The sociopolitical context is a broad level factor that can influence exploration of promising practices or interventions (Hoagwood, 2003) through policies and legislation (i.e., HIPAA, FERPA, NCLB). For example, the reauthorization of the Individuals with Disabilities Act (IDEA) in 2004 identified Response to Intervention frameworks as a strategy to address learning and behavior challenges in students. Since that time, many tiered frameworks have been used in schools across the country (e.g., response to intervention, positive behavior supports and intervention, multi-tiered systems of support, interconnected systems framework). In most of these tiered frameworks, students are generally screened and then identified as needing a service, but more needs to be understood on how students are referred to those services. Literature reports that connecting youth with integrated school mental health services can require significant coordination (Burns et al., 2004; Husky, Sheridan, McGuire, and Olfson, 2011) due to essential information sharing between 9 providers (Hamilton, Begley, and Culler, 2014). Mental health service access for youth is a complex issue (Burns et al., 2004), involving many systems. Since evaluating systems can be complicated (Wolfe, Lemer, & Cass, 2016), focusing exclusively on parts of the service delivery process may be more practical to address the needs of youth (Rosenblatt, 2010). In order to examine and address barriers to mental health service access, it is imperative to evaluate referral pathways for youth (Whitson, Connell, Bernard, & Kaufman, 2011). Funding. Funding is another broad level factor that can influence implementation processes. Many researchers have identified the need to integrate implementation science and school mental health (Hoover, 2018; Kern et al., 2017; Sarno Owens et al., 2014; Shernoff, Bearman, & Kratochwill, 2017; Weist & Paternite, 2006). Funding opportunities through federal agencies (i.e., National Institutes of Mental Health (NIMH), Substance Abuse and Mental Health Services Administration (SAMHSA)) can be used to support implementation of Evidence Based Practices (EBPs) or to disseminate innovations. For example, The National Institute of Mental Health’s (NIMH) Child and Adolescent Treatment and Preventive Intervention Research Branch houses the Division of Services and Intervention Research (DSIR). Two high priority areas identified by the DSIR are to 1) identify implementation practices to increase the uptake of EBPs, and 2) participate in strategic community partnerships (NIMH, 2018). These high priority areas identify the need to understand implementation research in community settings. Services for youth and families have historically been siloed and provided in isolation of each other. This can cause many challenges, such as difficulty accessing appropriate services. Researchers could use this opportunity to examine mechanisms for effective school-community partnerships and disseminate findings related to improved service access for youth. 10 A federal funding source directly related to mental health in schools is through SAMHSA’s Project Advancing Wellness and Resilience Education (AWARE). Project AWARE has been funded through grants that promote youth mental health in schools and communities, primarily though the implementation of Youth Mental Health First Aid (YMHFA). YMHFA is an intervention that trains adults to identify mental health issues in youth and link them with appropriate services. In 2014, twenty state education agencies were awarded with grants (SAMHSA, 2017), including The Michigan Department of Education. In an evaluation of YMHFA across five schools over two years, Gryglewicz and colleagues (2018) found that school personnel had an increased awareness in youth mental health issues, felt confident in identifying issues, and used help-seeking strategies to link youth to services. Although funding sources have provided training to school personnel to better identify youth with mental health issues, there still begs the question: what next? Retaining funding for school mental health services is challenging (Neufeld, Jones, & Goodyer, 2017), and many youth and their families are referred to community providers for mental health services. However, many communities, especially in rural areas, may not have the capacity to provide an abundance of services due to a shortage of mental health providers (Mackie, Zammitt, & Alvarez, 2016; Thomas, Konrad, Holzer, & Morrissey, 2009). While there are federal funding incentives for mental health providers (e.g., loan repayment) to move to health professional shortage areas, many communities still lack the capacity to provide mental health services to youth in need. Client advocacy. Clients can advocate for change at the individual, community, state, or national level (Hoagwood, 2003). In terms of school mental health, youth may not know how to advocate for themselves. Those who exhibit externalizing behaviors may be seen as “bad kids” and receive disciplinary consequences rather than referral to services. When school personnel do 11 recognize youth with mental health issues and try to provide services or connect them with services, a lack of parent engagement is often seen as a barrier to service entry (Langley et al., 2010). Parent engagement can be challenging for many reasons; some parents work long hours, and lack resources, other parents struggle with their own mental health issues or substance use, and some feel judged or blamed by service providers (Baker-Ericzen, Jenkins, Haine-Schlagel, 2013; Langley et al., 2010; McCarthy et al., 2017). On the other hand, when parents are engaged their advocacy efforts may be misinterpreted at times. For example, Buckingham and colleagues (2016) gathered qualitative data through focus groups with parents (n=20) and youth (n=11) to understand parent engagement in their children’s mental health services. They found that parents and youth who advocated for themselves by taking direct action to influence treatment, such as getting a second opinion or discontinuing medication because of troublesome side effects, were labeled as “disengaged” or noncompliant. Research highlights further misunderstandings on advocacy efforts at the state level as well. Using data from a national survey of State Children’s Mental Health Directors and advocates for Mental Health for America, Cooper and Aratani (2014) compared participants’ perspectives of themes related to youth mental health. They found that while state directors and advocates understood the importance of integrating EBPs in practice, the processes of how that integration was actually happening was not understood. Additionally, advocates did not understand which service settings were able to receive reimbursement for services provided to children. Exploring client advocacy efforts in school mental health is a much needed area of research. Existing studies examining client advocacy have identified a pattern of misunderstanding advocacy efforts and the logistics of carrying out youth mental health services. 12 Schools can help promote client advocacy by fully informing youth and their families about services that are available, and providing options to various providers in the community when trying to connect them with services. Inter-organizational networks. Inter-organizational networks refer to the network of organizations that are working together or are involved with one another. Utilizing a system of inter-organizational networks is a promising strategy (Provan & Kenis, 2008; Provan & Milward, 2001) that many school systems are implementing (Russell, Meredith, Childs, Stein, & Prine, 2015). The larger and more diverse a network is, there is a greater likelihood of a variety of resources that are accessible (Russell et al., 2015) to youth and families. A wide network of referral sources can alleviate capacity limitations (Kober & Retner, 2011) when providing youth mental health services. Factors of inter-organizational networks can serve as a barrier or facilitator to EBP implementation to service youth and families. Palinkas and colleagues (2012) used semi- structured interviews (n=38) to examine collaborative efforts between systems serving youth (i.e., mental health services, child welfare, juvenile justice) when scaling up an EBP for youth in foster care. They found collaborative relationships among inter-organizational networks were key to EBP implementation. These relationships had four specific characteristics: focus, formality, frequency and function. Focus referred to whether the EBP served a broad or narrow array of needs. Formality referred to the formal or informal roles and methods of communication between organizations. Frequency referred to the frequency of contact between organization, based on need. Finally, function occurred when each organization understood their role, purpose, and how they were serving youth within the larger context of networks. 13 Similarly, if an organization, such as a school, implements an EBP other community organizations or schools are likely to implement that EBP as well (McIntosh, Kel, & Delabra, 2016). Similarly, principals are more likely to trust an EBP if it has been used in similar schools and a similar student population (McIntosh et al., 2016; Neal, Neal, Kornbluh, Mills, & Lawlor, 2015; Neal, Neal, Lawlor, Mills, & McAlindon, 2018). Additionally, a school mental health providers’ social networks can influence the success or failure of EBP implementation. For example, researchers have found that if clinicians knew of other respected colleagues were using that EBP, then they were more open to attending a training and implementing the EBP (Langley et al., 2010; Lyon et al., 2013). Inner context. The Inner Context of the EPIS framework refers to factors inside of a school that will impact uptake of an EBP. These factors include organizational characteristics and individual adopter characteristics. Organizational characteristics. Organizational characteristics include components such as the culture and leadership of the school, and have been found to impact implementation of EBPs (Aarons, Hurlbert, & Horwitz, 2011; Damschroder, Aron, Keith, Kirsh, Alexander, & Lowery, 2009; Schoenwald & Hoagwood, 2001). There are also several characteristics unique to the school environment to take into context when exploring implementation of an EBP in a school setting. For example, most schools operate on a nine-month calendar which includes scheduled breaks (i.e., winter break, spring break) along with periods of time where there are bursts in activities, (i.e., exam weeks; Sarno Owens et al., 2014). These characteristics can impact EBP trainings for school personnel, as well as implementation of the EBP itself. A school context supportive of EBP implementation helps professionals across the school setting understand the utility of EBPs (Cook et al., 2015). Additionally, EBPs focused on youth 14 mental health are likely to have higher implementation rates when school personnel—aside from only school counselors—are trained on the EBP or processes and procedures related to the EBP (Langley et al., 2010; Sarno Owens et al., 2014; Stein et al., 2010). School administrators are key to creating a school culture and climate supportive of EBPs. School administrators serve in key leadership roles that can influence school climate as it relates to mental health EBPs, either positively or negatively (Sarno Owens et al., 2014). McIntosh and colleagues (2016) conducted interviews with school administrators (n=10) to explore events that impacted whether principals supported implementation of a behavior intervention or not. They found that administrators were more supportive of intervention implementation when they were able to learn from other schools that had implemented the intervention and have discussions with those school administrators. Additionally, they were more supportive when they felt the intervention aligned with their philosophy of working with youth and identified the need for a behavior intervention at their school. Also, if informative trainings and technical assistance were available, principals were more likely to be supportive of implementation. Individual adopter characteristics. An individual’s characteristics such as values, beliefs, goals, and the perceived need for change can influence implementation (Schoenwald & Hoagwood, 2001). School mental health providers’ general attitudes toward training can impact implementation of EBPs. For example, Langley and colleagues (2010) found that if a clinician attended a training with an attitude that embraced active participation, they were more likely to listen. The opposite was true as well- if a clinician attended the training with the attitude that they were not going to participate, they were less likely to listen. Additionally, the emotional responses a clinician experienced during a training predicted follow through for implementation 15 — if clinicians felt overwhelmed or stressed with the EBP they were less likely to participate. However, if they perceived the EBP could easily be implemented, they were more likely to follow through with implementation. Finally, Sarno Owens and colleagues (2014) further identified the need to examine implementer motivations and perceptions in the field of school mental health and implementation sciences. Clinicians were more likely to support implementation of a practice if they believed it would help them be better at their job. They also were more likely to support implementation of an EBP if it would be a good fit with the school context and culture, past just the needs and demands of their current caseload, and more likely to attend trainings (Lyon et al., 2013). Furthermore, Lyon and colleagues (2013) found that clinician’s attitudes in general towards EBPs impacted whether or not the EBP had utility. For example, if clinicians had negative perceptions of EBPs they were less supportive of implementation. Many researchers have noted the need for training, supervision or support, and technical assistance during EBP implementation (Han & Weiss, 2005; Langley, Nadeem, Katoaka, Stein, & Jaycox, 2010; Livet et al., 2018; Romer, Green, & Cox, 2018; Tapia, Ocasio, Estrada, Pantin, & Prado, 2017) in order for school personnel to feel prepared delivering the intervention (Romer et al., 2018). A lack of feeling prepared or supported may lead to unsuccessful EBP implementation. It may be helpful to explore the options for support, technical assistance, and training available when implementing a referral pathway between a school and partnering community agencies. 16 Research Paradigm and Qualitative Approach CHAPTER THREE: METHODS This project was guided by principles from implementation science, the scientific study of how to promote the uptake of research findings and evidence-based practices into settings (Eccles & Mittman, 2006). Implementation science focuses on the processes and strategies required for the uptake of an intervention. Of relevance to this study, Proctor and colleagues (2011) have eight key outcomes factors of concern to implementation scientists, three of which involve methods for pre-implementation studies. This study used thematic analysis, a qualitative methodology and mode of analysis (Braun & Clark, 2006) used for “identifying, analyzing and reporting patterns (themes) within data” ( pp. 79). There are many benefits to using thematic analysis. First, it is flexible and can be used with many different methodologies. Second, thematic analysis is useful in applied settings-- such as health services-- when results are to be communicated with individuals who are not in academia (Braun & Clarke, 2014). Researcher Characteristics and Reflexivity Reflexivity occurs when qualitative researchers reflect on their own role in constructing data in order to understand—and account for-- how their own biases, assumptions, and life experiences impact data interpretation. Reporting on reflexivity allows readers, and others interested in the results of qualitative research, to assess the trustworthiness of results (Richards, 2005). Several of my own experiences and beliefs have influenced my role in constructing these data. I am a doctoral candidate, have a master’s degree in couple and family therapy, and am a Licensed Marriage and Family Therapist. The training I have received throughout my graduate 17 career has focused on using systemic treatment approaches to address mental and emotional health issues experienced by individuals, couples, and families. Additionally, I have experience providing treatment across various settings including campus clinics, community agencies, and private practice. It is important for me to reflect on my social location and clinical experiences to ensure that I do not project my own perspectives, or level of privilege, onto those from whom I will be collecting data. While continuing my work within this intermediate school district (ISD) and their community partners, it has been important to consider the level of privilege that comes with my level of training, education, and experience, while also remembering the participants I interviewed are the experts in mental health treatment and professional collaboration in their community. Context The proposed study took place in collaboration with an ISD in central lower Michigan. Erie County will be used as a synonym throughout the rest of this dissertation to refer to the home county of the ISD. The population for this county is 160,000. The median household income is $46,000. The median household value is $111,500, which is lower than the state average ($147, 100). The poverty rate is 17%. Racial and ethnic demographics are as follows: 87.5% Caucasian, 8.6% African American, 3.3% Hispanic. While these are the demographics for the county as a whole, disparities are highlighted when examining the demographics specifically located in the county seat. The population in the county seat is 33,000. The median household income is $28,000—nearly $20,000 less than the median household income for the county. The median household value is $62,000, nearly half that of the county median household value. The 18 poverty rate is more than double, at 36%. Finally, racial and ethnic demographics are as follows: Caucasian: 67%, African American: 23%, Hispanic 6% (Factfinder.census.gov, 2017). The purpose of the ISD is to provide “educational leadership, services, programs, and resources in partnership with local schools and the community.” This ISD houses twelve public schools and three public school academies, collectively serving over 24,000 students (jcisd.org, 2018). According to Michigan Profile for Healthy Youth (MiPHY) survey data, youth attending public schools within this ISD tend to experience mental health issues at an increased rate when compared to national averages. Thirty percent of 7th graders, and 25% of 9th and 11th graders within the ISD have seriously considered suicide (MiPHY, 2018), compared to the national average of 16% of students (CDC, 2017). Furthermore, approximately 20% of both middle school and high school students within this ISD reported having made a plan to attempt suicide (MiPHY, 2018) as compared to the national average at 13%. MiPHY collects data on student risk behaviors (i.e., drug use, tobacco use, violence, emotional health, physical activity, nutrition, sexual behavior), and risk and protective factors, reported by students in grades 7, 9, and 11 in Michigan. MiPHY data is housed within the Michigan Department of Education and the Department for Health and Human Services (Michigan Department of Education, 2018). This ISD is one of three county intermediate school districts that are part of a state-wide grant called Project Advancing Wellness and Resilience in Education (AWARE), awarded by the Substance Abuse and Mental Health Administration (SAMHSA). The primary goals of Project AWARE are: (1) “to increase awareness of mental health issues among school age youth; (2) train educators and other youth-serving adults to detect and respond to mental health issues; and (3) connect children, youth, and families who may experience behavioral health issues with appropriate services” (SAMHSA, 2018). The three goals are addressed through multiple 19 strategies, including but not limited to: implementing Youth Mental Health First Aid (YMHFA), collaborating with community systems to increase access to mental health services, using multi- tiered behavioral frameworks, implementing culturally and developmentally appropriate mental health services (SAMHSA, 2018). Of the twelve public schools and three public school academies served by JCISD, Project AWARE was piloted in five schools. The grant was received in 2014, with a five-year funding period. Prior to the beginning of the study, I had been attending monthly meetings with the intermediate school district and community stakeholders (i.e., school principals, agency mental health workers, health and human services representatives, etc.) for over one year. The reason for attending the monthly meetings was twofold. First, to build rapport and establish trust with the intermediate school district and community stakeholders. Second, to learn more about the initiatives that had been implemented across the intermediate school district and the ongoing efforts to address mental health issues in youth. Prior to attending these monthly meetings, I met with employees of the ISD to discuss their research and evaluation needs and the skills that I could offer, with the intention of using any data collected for my dissertation. During monthly meetings with community stakeholders, I was introduced as a graduate student who would be helping to implement the external referral process. Sampling Strategy A purposeful sample of key informants for this study were recruited for this study. Purposeful sampling is a strategy used in qualitative research to identify and select participants who will provide information-rich data in a specific context (Patton, 2002), that is both detailed and generalizable (Palinkas et al., 2015). Purposeful sampling is appropriate for implementation research, generally (Palinkas, Horwitz, Green, Wisdom, Duan, & Hoagwood, 2015), and pre- 20 implementation studies, specifically. For example, Marshall and colleagues (2008) used a purposeful sample when examining facilitators and barriers to EBP implementation with a sample of consultants and program leaders at various study sites. Key informants are individuals who have knowledge on the issue being examined (Patton, 2015). For this particular study, school counselors (n=2), community school engagement specialists (n=3), school principals (n=2), and partnering agency personnel (n=4) were interviewed. School principals, school social workers, and community school engagement specialists were interviewed because they are the individuals within the school that are involved with making external referrals to mental health agencies. Agency personnel who receive mental health referrals were interviewed because they have valuable insight as to what happens once referral are received, and if the referrals are being received in the first place. Finally, specific agency personnel of the intermediate school district who have been tasked with creating and implementing the external referral system was interviewed. This group of individuals was targeted for sampling because they were closely linked to the external referral process either by creating the process, maintaining the process, making referrals, receiving referrals, or following up with referrals. Ethical Considerations Study materials and procedures were submitted to the Institution Review board at Michigan State University and approved as an exempt study. Recruitment followed IRB approved guidelines. Participants were given contact information for the MSU IRB Office, if they had questions or concerns regarding this study. 21 Confidentiality. Participants were reminded their interviews were confidential, and that the information they provide during their interview will be de-identified. They were also made aware of how I planned to use the information they provided. Potential risks. Participants may be concerned their responses will not remain confidential, and may worry about the repercussions of their responses. To address this potential risk, participants were informed prior to the interview how confidentiality would be maintained and how collected data would be used. Participants may feel disempowered if the information they provide does not lead to improved mental health service access for students in the intermediate school district and larger community. Prior to the beginning of the interviews, participants were informed the results of the study may not necessarily lead to the adoption of an external referral pathway. Potential benefits. Participants may feel empowered by providing information that is used to inform practices at the ISD. Participants may feel they are positively contributing to solutions that can connect youth and their families with mental health services. Data Collection Methods Recruitment. The ISD Data Specialist (my primary contact) emailed stakeholders to let them know that I would be contacting them. The data specialist also made verbal announcements at the monthly Care Management Team meeting to announce that I will be reaching out to individuals to participate in the study, and to remind stakeholders to follow up with emails that I had already sent. She provided me with a contact list for stakeholders, whom I emailed to recruit into the study. Data collection began in October 2018 and concluded in December of 2018. Stakeholders who had not responded to me after three attempts did not participate in the study. 22 Participants were eligible for this study if they: (1) were identified by the ISD Data Specialist; (2) were an employee of one of the ISD schools or the ISD; (3) served in a role where they refer youth to mental health services, or (4) worked in a partnering mental health agency that accepts referrals from the schools; (5) were English speaking; and, (6) consented to participate in an interview. Participants were excluded if they did not meet these criteria. Informed consent. Eligible participants were verbally read the purpose and procedures of the study using an informed consent document (see Appendix A), which contained information about the purpose of the study, procedures to ensure confidentiality, and MSU faculty and staff to contact with any concerns regarding the research. After listening to the informed consent, people who were eligible for participation had the opportunity to ask questions and acknowledge their agreement to participate. Interview. Guided by the interview guide described in a subsequent section of this paper, eligible participants completed a phone interview. Immediately prior to the interview, I informed participants them that they could stop the interview at any time without penalty and also asked them to answer a series of demographic questions (see Appendix B). Eligible participants who completed any portion of the interview were given a $20 Amazon gift card as a means to recover costs associated with participating in the study. Data Collection Instruments and Technologies Demographic form. The demographic form included questions regarding the participants’ age, education level, and agency/role (see Appendix B). Demographic information was collected during the phone interview, after consent was given by participants. Interview guide. Semi-structured interviews were conducted with stakeholders. Broad, open ended questions were used to gather information about the stakeholders’ perspectives on 23 barriers and facilitators to implementation of an external referral process. Probes were used to ensure thick descriptions were gathered. I conducted all interviews. See Appendix C and Appendix D for interview guides. Units of Study My recruitment goal was to engage twenty-one key informants to participate in the study. Specific key informants, who have a role with the external referral process, were identified by the ISD Data Specialist. Specifically, I attempted to recruit 3 Community School Engagement Specialists, 6 School Counselors/Social Workers, 6 school administrators (e.g., curriculum directors, principals), and 6 agency representatives (e.g., mental health agency staff, JCISD staff). The following key informants participated in the study: community school engagement specialists (n=3), school counselors/social workers (n=3), school principals (n=2), and agency personnel (n=4). Table 2 details demographic information of the participants. I contacted each key informant a maximum of three times via email to schedule an interview. If no contact was made after the third attempt, the key informant was not included in the sample. Each key informant participated in one phone interview. After data analysis, participants were contacted a final time via email and asked to participate in member checks by reviewing the findings. Three participants responded to the email. A summary of the results was emailed to them. The three participants had no disagreements with the results, and had no additions to the results. Data Processing Interviews were audio recorded on two different recorders (one serving as backup in case the first one experienced technical difficulties). These audio recordings were uploaded to a secure server. Transcripts of the audio recordings were then made and also uploaded to the secure server. Transcripts were also uploaded to NVivo, a software package that facilitates 24 qualitative analyses (Bazeley & Richards, 2000). As an extra precaution in case of unanticipated issues (Richards, 2005), copies of the audio recordings and transcripts were kept in a secure Table 3.1 Demographics of participants Stakeholder group n Age range n Ethnicity n Education n 3 22-29 2 Caucasian 11 Bachelor’s 2 Community school engagement specialist School counselor/social worker Principal n 0 3 2 4 1 2 Yearly gross income $0- 19,000 $20- 40,000 $41- 60,000 $61- 80,000 $80,000 and above Prefer not to answer 8 0 1 3 30-36 3 African American 1 Master’s 2 37-44 4 Hispanic 0 0 0 Doctoral Professional degree Agency personnel 4 45-52 2 Asian 53-60 1 Native American 61 + 0 Multi- racial 0 research space. Finally, memos taken during the interview were kept in hard copy form and also scanned into pdf form and uploaded to a secure server. Data Analysis Data analysis was an iterative process, occurring concurrently with data collection (Richards, 2005). Data analysis was completed by myself and one research assistant. Consistent with existing guidelines for thematic analyses (Braun & Clark, 2006), phases of analysis were conducted. Phase I included familiarizing yourself with your data. During this phase, I began to individually immerse myself in the data. I then began coding by jotting down notes and initial ideas while reading through the transcripts (Braun & Clark, 2006). Generating initial codes 25 occurred in phase II, where I generated initial codes from the extracted data (Braun & Clark, 2006). During phase III, searching for themes, themes were identified from the initial codes from phase II. Phase IV, reviewing themes, includes reviewing and refining themes at two levels. During the first level, coded data extracts were reviewed by the initial researcher, and to ensure the data created a cohesive pattern. During the second level, the entire data set was reviewed by both researchers, to ensure the themes were coherent and made sense. It is not uncommon for data to be recoded (Braun & Clark, 2006). Defining and refining themes occurred during phase V. Each theme was reviewed and a detailed analysis was written for that theme. Additionally, sub-themes were identified during this time (Braun & Clark, 2006). Preliminary data was shared with ISD staff, and they were asked for their perspective (e.g., Do these results make sense?). Suggestions from ISD staff were incorporated during phase VI. Phase VI was producing the report. During this phase, the results of the dissertation were written. Examples and data extracts have been used to validate the identified themes (Braun & Clark, 2006). Techniques to enhance Trustworthiness Trustworthiness is a way to establish rigor in qualitative research (Lincoln & Guba, 1985). Quantitative research methods establish trustworthiness through reliability and validity; parallel criteria in qualitative research includes credibility, transferability, dependability, and confirmability (Lincoln & Guba, 1985). Practices to enhance trustworthiness during thematic analysis have occurred during each phase of the analysis (Nowell, Norris, White, & Moules, 2017). Credibility is the “fit” between the participants’ perspectives and how the researcher presents those perspectives (Tobin & Begley, 2004). Transferability is the degree to which 26 information from the qualitative study can be generalized and applied to similar contexts (Tobin & Begley, 2004). Qualitative researchers demonstrate dependability by clearly documenting their research process (i.e., decisions to change protocol, changes in themes during analysis, etc.) (Tobin & Begley, 2004). Confirmability occurs after credibility, transferability, and dependability have been achieved (Guba & Lincoln, 1989), and the results of the study are clearly obtained from the data (Tobin & Begley, 2004). In order to ensure credibility, peer debriefing and researcher triangulation occured between myself, the graduate assistant, and an advisor during analysis to build reliability (MacQueen, McLellan, Kay, & Milstein, 1998). Additionally, data was triangulated (Carter, Bryant-Lukosius, DiSenco, Blythe, & Neville, 2014) by comparing reports from the different groups of participants (i.e., school administrators, school social workers, agency staff, etc). Nowell and colleagues recommend using different forms of triangulation during qualitative research (2017). Transferability took place by gathering thick descriptions from key informants, so that the information gather could be applicable to similar settings. An audit trail was maintained throughout the duration of the study to establish dependability. An audit trail is a document containing written memos of the theoretical and methodology decisions, and justification for those decisions so that other researchers can understand why decisions were made (Koch, 1994). In this study, the audit trail documented theoretical thoughts, methodological thoughts, reflective thoughts, potential codes and themes, code generation, and notes on the development of hierarchical concepts and themes (Nowell et al., 2017). Changes to the research study or protocol followed recommendations made by Richards (2005): 1) noting that something has happened or changed; 2) recorded why the step was taken; 3) tell what the alternatives were and why they 27 were rejected; and 4) described the likely results for the final project (pp.44). Confirmability has been ensured by addressing credibility, transferability, and dependability. Additionally, member checking occurred to ensure that I interpreted participants perspectives as they intended (Lincoln & Guba, 1985). 28 CHAPTER FOUR: RESULTS Themes were organized by the EPIS model during the first level analysis. Within each category of EPIS, themes were identified as either a barrier or facilitator to implementation of an external school mental health referral pathway. Twenty-three themes and three sub-themes were identified during analysis. I acted as a primary coder for data analysis. A secondary coder conducted a confirmability audit in phase III of data analysis by reviewing the analysis, ensuring a fit between themes and codes. The secondary coder was a post baccaulerate scholar. Differences among themes and codes was discussed between myself and the secondary coder until an agreement was reached. Below, I will describe each of the twenty-three themes and follow with a discussion of how they assist in answering the research questions. The results are summarized in Table 4.1. Table 4.1 Summary of qualitative results EPIS primary category EPIS sub-category Barrier or facilitator designation Theme Sub-theme Outer context Sociopolitical content Barriers Funding Inter-organizational networks Barriers The “right type” of training Long waiting lists Lack of provider capacity Current network of providers could be expanded Technology Different procedures between agencies Navigating confidentiality amongst partners Facilitators Referring to outside providers depends on insurance Facilitators Grant funding has allowed for more resources 29 Source CSES, SW, SC, AP CSES, AP SC, CSES, AP AP, CSES CSES, SW, AP SC, AP AP, SC AP SW, AP Table 4.1 (cont’d) Meeting consistently amongst community partners Agency collaboration Agency personnel providing services in schools Some schools and agencies already have an external referral pathway in place CSES, SC, AP P, CSES, AP SC, CSES, AP, P CSES, AP, P, SC, SW AP, SW, P CSES, P, AP CSES, P, SC SC, CSES, SW SC, SW AP, CSES, SC, P CSES, AP AP, CSES, P, SC SC, CSES, AP SC, CSES, SW, P, AP AP, CSES, P, SC CSES, SC, P, SW, AP SC, SW, P, AP Lack of understanding about mental health issues Lack of resources Interdisciplinary meetings to review students Inner Context Organizational characteristics Barriers Individual adopter characteristics Barriers Changing a system Facilitators Administrative support Mental health staff as a resource Whole child approach Increased awareness of mental health issues Lack of connection with parents can prevent connecting youth with mental health services Facilitators Value their role in supporting youth and families Having professional networks enhances individuals’ ability to connect youth with mental health services Youths’ individual needs are used to determine appropriate services Stakeholders believe and external referral pathway would be helpful Note. P = Principal; CSES = Community School Engagement Specialist; SW = School Social Worker; SC = School Counselor; AP = Agency Personnel. Outer Context One theme was identified as a barrier for the Sociopolitical Context, and no facilitators were identified. One facilitator was identified for Funding, and no barriers were identified. No themes were identified as barriers or facilitators for Client Advocacy. Eight themes and three subthemes were included in Inter-organizational Networks. Four themes and three subthemes were identified as barriers, and the remaining four themes were facilitators. 30 Sociopolitical context. One theme was identified as a barrier for this category, which was referring to outside providers depends on insurance. No facilitators were identified for this category. Barriers. Referring to outside providers depends on insurance. Both school personnel and agency personnel identified insurance as a barrier that could hinder their ability to assist youth and families with accessing and utilizing mental health services. For example, school personnel stressed the frustration parents may feel when they are connected to a service that does not take their child’s insurance. One principal stated, “…it could create a nightmare if the community school engagement specialist refers somebody…and they don’t have somebody who takes our students’ insurance, well now we’ve wasted time and frustrated the parents.” Agency personnel also described the impact of working with students who have an insurance plan their agency is not paneled with. Agency personnel reported the challenges of working with students’ who have an insurance plan their agency does not accept. One agency clinician reported: What is hard is when we get a referral and we don’t accept their insurance. We have to refer them out. And we have some insurance companies that refuse to let people panel with them, so they are very limited to anything or we have some insurance companies that don’t even cover mental health services. Furthermore, in these situations, the agency personnel described whether to decide to work with the child for free, or to not see the child and know they may not receive any mental health treatment. One participant from a mental health agency reported, “We know going into it we can’t get payment for this case…I can make a decision on, okay, do we just again cut our loss and say, ‘Hey, we'll do it pro bono,’ and eat the cost of it.” 31 Funding. One facilitator was identified for this category. No barriers were identified for this category. Barriers. Grant funding has allowed for more resources. Participants emphasized how federal grant funding had provided opportunities to increase community awareness of the effects that trauma could have on youth. For example, one agency staff member noted, “We were able to use the Project AWARE grant funds to train local clinicians on trauma informed practices.” Participants described how the federal funding allowed capacity to be built in the intermediate school district and community by teaching adults how to recognize mental health symptoms in youth, how to best respond to identified youth, and how to treat youth in mental health settings. They also described how grant funding assisted with capacity building by providing funding used towards hiring staff that can focus specifically on student mental health and connecting students with resources. One community school engagement specialist reported: …they are part of the percentage that makes up the salary of my current job. Some of it [salary] is made up through Project AWARE, and some of it is made up through the State of Michigan Mental Health Block Grant. Participants also acknowledged the grant funding will be ending soon, and the uncertainty of how that will impact services provided to youth moving forward. One mental health agency staff member stated: Project AWARE has provided amazing funding for us to do that what we can do, but that is ending at the end of the year. Money and grants and more providers and more people willing to kinda like to bite the bullet and, get out there and get moving. Inter-organizational networks. Eight themes and three subthemes were identified for this category. The most prevalent themes noted the overall lack of capacity for mental health providers to serve the needs of youth. Other themes were related to logistics between agencies 32 that could serve as a barrier or facilitator for implementation of a systematic external mental health referral pathway. Barriers. Lack of provider capacity. Both school and agency personnel described the lack of provider capacity in the county, which was observed in multiple ways. School officials emphasized the need for more mental health services. For example, one school principal reported: I think the big thing we struggle with right now is being able to provide the community with the mental health services that they need…we have such a high priority here… and there’s not enough resources to be able to support all of the community for mental health. Personnel from mental health agencies reflected on the amount of referrals they had received for students, versus the amount of youth they were able to serve. In some cases, agency members reported receiving nearly twice as many referrals as youth that they could serve. One agency representative reported, “We had 150 referrals I think, and then, out of the 150 we're only able to see 75 ‘cause we're trying to get other providers to come to the table…we're just overloaded.” Participants reflected on the capacity challenges experienced in Erie county. One Community School Engagement Specialist spoke to larger system challenges contributing to capacity issues. The CSES noted: The agencies don’t have qualified staff to handle the need. A lot of the non-profit agencies here in Erie county, they don’t pay their workers very well, and it’s like- as I found out- very expensive to get your Master’s Degree in Social Work, and you have to find a job that supports you so you can pay your loans back. The “right type” of training. One common barrier described by participants was that many mental health providers did not have the “right type” of training. When referring youth to external providers, many school and agency personnel mentioned the importance of connecting youth to providers who were trained in evidence-based treatments to address symptoms of 33 trauma; specifically, trauma-focused cognitive behavioral therapy (TF-CBT). Although some community providers were trained in TF-CBT through different initiatives, there were still many community providers who have not had that training. Regarding TF-CBT, one agency mental health therapist noted, “…that is an important thing, because we try to provide trauma-focused CBT out in the schools, and not everybody has the training and background in that.” School staff agreed with agency personnel, stating, “… it’s difficult because there are limited providers here in [town] and there’s also limited providers who are providing services in schools, and limited providers who are trained right.” Long waiting lists. Another challenge commonly reported by participants was that providers or mental health agencies who do have the “right type” of training have long waiting lists, which delayed youths’ access to services. While describing the community services existing in Erie County, one community school engagement specialist stated, “I think that there are several agencies and several clinicians to see, but I know there’s a waiting list.” One school social worker described her dilemma with trying to refer students to mental health services in the community, stating, “People say we have a huge waiting list, or we can't see that type of client… that can be really off putting for families.” This was not only a challenge for school personnel, but for agency personnel as well. When describing the challenge with receiving a referral, and needing to refer that student to another provider, one agency therapist wished Erie County would “[have] the capacity to refer them to the right people that have openings.” The waiting lists were concerning to participants not only because it delayed youths’ access to care, but also because they were concerned families may end up giving up on accessing treatment due to long wait times. 34 Current network of providers could be expanded. As a solution to the lack of provider capacity, many participants suggested expanding the current network of mental health providers outside of the county. They believed this could be more helpful to some families, as they may be in closer proximity to other agencies or providers in a neighboring county. One agency staff member reported: One of the ways in which I see an external referral being more helpful in a county like Erie is- because there is a lot of capacity issues- I can see maybe building relationships with other agencies that are outside the county. School personnel offered a similar opinion, but focused more on what families may want or feel more comfortable with. When discussing implementing an external mental health referral pathway, one schools staff member said, “If it’s only for our county, that could be a challenge, because I do know some people really wanna go to [university in neighboring county] and in [city], and that’s not in Erie County.” Technology. Technology was another barrier to implementation of a systematic external referral process identified by participants. Participants reported concerns with confidentiality when potentially communicating with other agencies about youth. When attempting to identify a platform that agencies and schools could use to refer students, one agency representative reported, “Initially, we thought Google, we do use Google a lot but it wasn’t as… protected as we would hope… especially when we're looking at mental health needs of student, and demographic information being sent.” Participants were also concerned with how user-friendly a potential systematic external referral process would be, and how well the different agencies and schools would be able to communicate with it. For example, another agency representative stated: Yeah, we [agency] use Google…but I know there's a few other places that can't open the form or had issues with it. Whatever the system is gonna be, whatever technologies that 35 we're gonna utilize, I just think it needs to be a format that everyone can access, and that it's simple enough that it's not bogging down anybody. Another agency representative stated, “…just finding that format or the platform to use that everybody’s comfortable with using and sharing information across.” Overall, participants agreed that a universal, user-friendly system was needed. Different procedures between agencies. Participants expressed how there are different procedures amongst schools for how to refer students to mental health agencies, and different procedures amongst mental health agencies on how to accept referrals. Participants commented that mental health agencies had different criteria for which students they could accept, based on insurance and the presenting problem. One agency personnel said, “Agencies have their own policies, their limitations, community mental health authorities have their own policies and practices just as well.” Participants identified how these differing procedures could be challenging while attempting to streamline an external mental health referral pathway. One agency representative from the intermediate school district expanded on this challenging, by stating: We are currently in multiple school buildings. So the buildings that we are in, we have processes in place- how to identify…trauma or a mental health concern- and then each school building unfortunately has their own process and how they identify those students… student study team… attendance records, discipline referrals, grades…and then getting the student set up for services. Navigating confidentiality amongst partners. When thinking of an external referral process, participants reported challenges in how to notify parents of what information is shared between various parties, how that information is shared, and who will have access to that information. Some participants wondered who would be able to see the information in the referral process, and if there would be a point person at each agency or school, or multiple people that would be viewing the information. When thinking about how confidentiality would 36 be navigated, one school principal wondered, “Probably the same sort of confidentiality that it appears right now, that it's hard to – I mean, I think – then if the family knew ahead of time, what kind of information must be shared too.” Participants also wondered what other purposes the information shared during an external referral would be used for. Participants expressed the importance of ensuring the focus is on connecting youth with services rather than getting student data during the referral process. One agency representative stated, “I just think barely minimizing who has access to that information, for how long, and just making sure that it makes sense and that the focus is not so much on the data.” Participants worried about the long-term ramifications of sharing any student data that might be gathered during the external referral process, and how that might impact youths’ future attempts at getting mental health services, or when applying to jobs (i.e., the military) as an adult. A staff member from an agency reflected, “…parents not even being well enough informed to make that decision…not realizing if you open up this information for your child through other agencies, like what impacts will that have.” Facilitators. Meeting consistently amongst community partners. Consistent meetings among community partners was identified as a factor that could contribute to the potential success of a systematic external referral pathway. Participants reported currently attending a monthly meeting consisting of school personnel, local mental health agency staff, and other health and human service providers. According to one agency representative, “Before I joined the grant, the community specifically the Project AWARE Care Management Team was built to try to bring the key stakeholders at the table. Providers were there, educators were there, other local agencies.” Participants reported feeling as if consistent meetings allowed community partners to know what was happening in other agencies, with kids in the school system, and develop 37 common goals and next actions. One community school engagement specialist reflected, “My administration staff goes to the CMT meeting, which is great, because they know the resources to choose from… so it’s not just me.” Participants also described how consistent meetings validated the community members had shared values in supporting youth and family well-being. Agency collaboration. Similarly, agency collaboration during these Care Management Team meetings led to shared paperwork, reciprocal awareness of school and agency processes, as well as knowing the point person to contact at each school or agency. A school principal identified how agency collaboration has been helpful during crisis situations that have occurred at school: We get releases, so we can share, we can be the point person sometimes in communication and… our student that was just in crisis that we wanted to have that threat assessment on him, he [community school engagement specialist] made the appointment for the family for them to go get the evaluation done. Likewise, agency personnel emphasize the importance of the collaborative work, describing the process as, “We work closely with [name] at the ISD or at Project AWARE to determine what they need the schools to be doing, and then incorporating that into our processes.” When schools have agency paperwork it can create a more streamlined process for assisting parents to get services in place for their child. Agency personnel also noted the convenience of sharing paperwork between the agency and schools, stating: Some of the districts have our paperwork, like our consent for treatment, our release of information, our insurance, demographic form, and they've been getting that with the parents, so if we can't get a face-to-face with the parent the child doesn't suffer and we can still proceed with services and we just do the intake then over the phone with the parent, and then face-to-face with their child after they give us permission. While describing the collaborative efforts between the agency and schools, another agency member stated, “We talk about what are their needs, what are our needs, how do we 38 provide the best needs to them.” School and agency personnel report that sharing paperwork has been a way to remove barriers to mental health service access for families. Agency personnel providing services in schools. Participants reported how beneficial it has been to have clinicians from mental health agencies come into the schools to provide individual and group therapy services. One community school engagement specialist reported, “I’m lucky that my school has been open to the idea of bringing mental health professionals in our building to meet with kids… that’s very fortunate that that’s happened.” A school counselor echoed this finding, stating, “We have [agency] that comes into our school and does groups and individual counseling.” Furthermore, participants describe how this collaboration has addressed many barriers that often prevent families from utilizing and accessing mental health services, such as transportation issues or scheduling difficulties. Another community school engagement specialist reported: …there’s a lot of barriers in place for them [students] in getting to mental health resources within the community… that may have prevented them from being serviced if not for these on-site clinicians that are coming in and working with them. Agency personnel also described their agencies’ commitment to going into the school and providing services, as an additional way to best serve their families. Some schools and agencies already have an external referral pathway in place. When asked about implementing a systematic external referral process, some participants expressed they already had a well-functioning referral pathway. A staff member at the intermediate school district reported: [Agency] has pretty much taken the lead in doing this process in filling out all the forms in regards to what the school needs to do, what their role is, what the parents’ role is, so that we have forms created, so that the school completes it. It’s not electronic at this point…it’s just different forms… then there’s the back and forth between the school, the parent, and the provider. 39 School personnel report sharing forms between agencies and having a specific point person to contact has helped to streamline their own processes. Furthermore, participants report that having a specific point person has allowed for clear communication about whether students have been accepted for services at a specific agency; or, if there may be an issue where the agency cannot accept their referral. One example of why an agency may not be able to accept a referral would be due to challenges in working with the student’s insurance carrier. One school community engagement specialist stated: To some degree we've implemented that here… where teachers, families, coordinators, administration, students themselves can refer themselves to me and then I can take a look at what's going on, and when I can help externally refer them if more intensive services are needed at that time. Agency personnel also reflected on the external referral pathway their agency participates in. One staff member noted: Right now we have an email address that all of those electronic referrals come through from the school… our form and our paperwork, they'll – they scan it in and then they email it to that email address and that goes to – there's four of us on it. Another agency staff member stated: We have a straight referral form. Each school has access to that hard copy and PDF. We get the types up, we get them handwritten, and then they either e-mail it to an e-mail address that’s linked to my e-mail address, or they fax it over. Participants who reported already having an external mental health referral pathway in place expressed satisfaction with their method and the systems they had already put in place. Inner Context The inner context consists of organizational characteristics and individual adopter characteristics. Three barriers and five facilitators were identified for Organizational Characteristics. One barrier and four facilitators were identified for Individual Adopter Characteristics. 40 Organizational characteristics. Eight themes were identified in this category. Three themes were identified as barriers, and five themes were identified as facilitators. Barriers. Changing a system. Participants explained how implementing a systematic external referral pathway requires a shift in the thought process of all youth and family resource providers in the county. Participants expressed that in order to successfully communicate, work together, and support children and families, resource providers would need to share a common goal and have transparent processes. An agency staff member reported: It's a whole system's changed process, it's looking at things differently and sometimes we can get resistance at the local district levels. They have a lot on their plate and doing that one more thing to try to implement to can cause some push back, to try to work with the school and to know how to fit in to their process seamlessly and in the long run makes their job easier. School staff reflected on changes happening in their individual school, and how those changes may impact the processes of other schools and organizations they collaborate with. One principal noted, “We started studying last year in how we’re gonna be able to implement PBIS [Positive Behavior Intervention and Supports], but we’ve moved that out this year as well… so our systems are now falling in alignment”. A school social worker also described the challenges when adopting a systems change process, stating: Agencies will have new programs, or will have different grants, or will close… it’s really hard to stay current… having a central way to make a referral with someone- or some entity- that was current with what is available, with the way agencies evolve [would be helpful]. Lack of understanding about mental health issues. Despite many participants discussing how there has been an increase in awareness of mental health issues among students due to the grant funding received by the intermediate school district, participants expressed there are still many people who do not have an understanding about mental health issues. School personnel 41 discussed their concerns that parents may not understand mental health symptoms their child is experiencing, which may interfere with using an external referral process to access mental health services. For example, one principal explained: Parents…they are at times very resistant to any type of referral process and that’s one thing I think with time and education, parents can learn and understand. But that is a big barrier to any referral process, because if a parent says no, a parent says no. Agency personnel, on the other hand, expressed concerns that school personnel may not adequately understand students’ mental health needs, thus misinterpret issues to parents. One agency mental health therapist stated: … with an external process… people who are uneducated about the mental health problems, they are the ones who are explaining what they are doing to parents, you know what I mean, like I feel like there could be just diversion in communication between who is sending the referral and who the referral is for. Agency staff expressed concerns that communication amongst parents, agency staff, and school staff may be misconstrued. Participants expanded on school personnel’s understanding of mental health symptoms, particularly those related to traumatic experiences. When describing reactions to collaborating with local mental health agencies, one school community engagement specialist described: Some of the concerns… were ‘Oh, do we want those clinicians to come in, because it is trauma focused, bringing up all those trauma [into the classroom]… but we haven’t encountered any real issues…we’ve seen huge growth in those students that have participated in the program. Lack of resources. Participants identified a lack of resources as a barrier to implementing an external mental health referral pathway. Resources included staff, time, and money. One principal stated, “Time and money… there's never enough of either one of them.” School personnel described the limited funding options that schools have available to hire staff that can specifically focus on the mental health needs of students. For the staff that are available to 42 address students’ needs, participants report that school personnel are already overburdened with their job duties, and other tasks, and have limited time available. A school counselor reflected on the amount of time it takes to stay updated with the services that are available to students and families in the community, reporting, “It takes so much time- all the time to update this and that, and they move, their numbers change…time is of the essence and there just isn’t enough of it.” Participants identified challenges with other resources as well. One example is the physical room available at their locations. “Just the space within a building. I know that’s a concern of other buildings. They don’t have the space to meet with the parents of have a seminar hall… that seems to be an issue,” described one community school engagement specialist. Facilitators. Interdisciplinary meetings to review students. Some school personnel reported meeting consistently to review students. They stated the purpose of reviewing students was to recognize which students would benefit from an internal referral, or external referral for mental health services. One community school engagement specialist reported: We do it here every Monday… the teacher fills out the form of what they’re seeing in the classroom and what their concerns are. They contact the parent…the child could be facing [an issue] or any tragedies in the past or trauma… all that is collected and it’s brought to the team Monday. Additionally, they described how meeting consistently allowed student progress to be monitored, and school social workers or community school engagement specialists could identify whether youth would benefit form more intensive services than they may already be receiving at the school. One community school engagement specialist reported, “every Friday… we also have team meetings where we are able to…talk about issues with students…if we have talked about an external referral, we're able to do that as well during that time.” Participants expressed how having consistent meetings allowed for multiple professionals to be present. A school social 43 worker described attendance at the meetings, stating, “We have a team process…ISD social workers, speech pathologist, teacher consultants, two psychologists, along with their supervisor, our on-site social worker, the gen-ed [general education] social worker, and then the two co- principals, myself, and the classroom teacher.” School personnel reported having specific time set aside with an interdisciplinary team to discuss students was needed to focus on how to support the students and families’ needs. Administrative support. Participants described the importance of having administrative support in both schools and agencies to foster the implementation of an external referral pathway. One school social worker described the benefit of having administrative support for the mental health needs of students, “The support in my district is huge, our superintendent, and principal and both social workers… we've been invested since the beginning in the district.” Participants reported that having administrative support allowed them to feel as if they “had a voice” for the families they work with. They also described how having administrative support allowed for open discussion about ideas to help families, supports the intermediate school district could provide, and community resources that could be utilized. “We work closely with the ISD teach so that we can get ideas, and a variety of other supports,” stated one school counselor. Therefore, the support participants received from administrative was both tangle and intangible in nature. Mental health staff as a resource. Agency personnel reported having specific staff is helpful to managing the amount of external referrals received from schools. One agency representative stated, “We have a front-line staff which I think is really important because that just helps with some of the work load.” School personnel also reported that having specific staff dedicated to the mental health needs of youth and their families is helpful. A principal noted: 44 She [community school engagement specialist] has like a dual head. She is a social worker but she's also the Project AWARE worker- that is her responsibility. She definitely can do mental health screens and all, and then make referral out for families. School personnel reported having these staff members to specifically address youths’ mental health needs have been possible due to federal grant funding. While describing his role, a community school engagement specialist stated: Word is still starting to get out that that I’m here. I’m not as busy over at the high school as I am at the K3-8 building. The administration here at the K3-8 building really utilize me and they know my role… I get a lot of referrals from them to screening assessed kids and referring out. Participants described how these staff members are able to meet with students and parents to connect youth and their families with mental health services and other needed resources. Increased awareness of mental health issues. Participants believed a systematic external referral pathway would be useful because more school staff are aware of how to identify mental health needs in students. Participants described how training opportunities were able to be provided from grant funding. One school social worker reported, ““Teachers are getting more training and then they are better able to address some of the student needs.” An agency staff member believed that, regardless of training opportunities provided to teachers, school staff would become more aware of youths’ mental health needs due to various situations that arise. She stated: Whether there is a grant or not, staff is gonna become more knowledgeable, just kind of out of the situation- or if there is a mental health crisis at the school- I feel like teachers and admin…are becoming more aware of these issues. Whole child approach. Participants described how the school district, and community providers, had adopted a whole child approach to addressing student needs. One principal stated, “I realized after my first year of being a principal that not much was going to get done 45 academically if we did not address the needs, the mental health problems of our students.” Participants described a change in perspective of how to view student readiness and student academic achievement. An agency staff member reported: I think the mental health needs of the youth fall up under that umbrella as far as how educators are starting to view student readiness and student achievements and all of the factors that play a role into student achievement. Furthermore, school personnel reported focusing on students who had chronic absenteeism, and working to connect those students and their families to community resources with the belief that would improve attendance. A principal noted, “It’s about supporting the families as well as the children, because healthy families can produce healthy kids.” Individual adopter characteristics. Five themes emerged in this category during analysis. One theme was identified as a barrier, and four themes were identified as facilitators. Barriers. Lack of connection with parents can prevent connecting youth with mental health services. Participants described the challenges of connecting with parents as one potential barrier to utilizing an external mental health referral pathway. “I would say a lot of the families with the most neediest kids that need the help, they [parents] don’t follow through… and I’m more than recommending they do,” explained one community school engagement specialist. Participants also described needing parental consent to refer and connect youth with external mental health services. Some participants reported that sometimes children are living with other relatives, and parents may be in incarcerated or working long hours, and getting consent can be extremely difficult. School personnel reported that even when they are able to connect with parents, the parents may not want mental health services for their child. This dilemma was described by a community school engagement specialist, who stated: “The parents… they feel that their child 46 doesn’t need mental health or they’re worried about something coming out while they're talking.” School personnel believed that many parents did not want their child to speak with a mental health professional for fear that Child Protective Services may be called due to abuse, or parental substance use. School personnel also recognized that many families may want to connect their child with mental health services but have other struggles that may take priority of returning phone calls from the school. Another community school engagement specialist explained: …the parents just don’t answer the phone or don’t return phone calls, not because they don’t want services for their child, but just because they have so much other stuff going on. They’re worried about how they’re going to pay rent, how they’re going to get their kids to school…illness… this and that…substance abuse. Another community school engagement specialist used a metaphor to describe their experiences with some parents, stating: You can lead a horse to water, you can really set it up for him, offer to make the call from your office, offer to go to the intake appointment, offer to help them with their health care insurance, all these things…and they won’t follow through with those to connect their kids. Facilitators. Value their role in supporting youth and families. Despite the challenges of connecting with parents, school personnel reported how important their role is to connect with parents. Participants described the importance of building relationships with parents, so that parents can feel supported and feel comfortable asking for resources when needed. One community school engagement specialist described her process: Usually I will try to reach out to the parents and say, “Hey," – introduce myself and talk about my role and just say, "Has your child ever had mental health services, is that something you’re interested in," because it looks like maybe they could benefit from it. It’s actually very accepted by parents. I always worry about cold calling parents and say, “Hey, your child looks to be struggling,” but parents really appreciate that somebody is 47 taking the time to notice that and address those concerns and typically, they have those same concerns. School personnel describe this type of collaboration as a rewarding and fulfilling part of their job. Another community school engagement specialist described her role, stating, “I work not only for students, but I can work with parents to help get them into different kinds of counseling or anything that they might needs, as well as other people in the family.” While describing her role in collaborating with parents, a school counselor stated, “We work very closely with the parents to know what are they doing at home, what they feel they need, what works for them, what doesn’t….and what we feel like is the next step.” Participants described wanting parents’ input on how their child is doing, and what the parents are noticing and wanting for their child, so that school personnel know how to best intervene. Having professional networks enhances individuals’ ability to connect youth with mental health services. Having a broad social network, both inside and outside of the school system, was identified as crucial to connecting youth and families with mental health services as well as other community resources. One community school engagement specialist described how building rapport with teachers in the building created more opportunities to discuss students’ needs, noting, “Teachers are making referrals to myself in the form of either coming and talking to me because I am based just in the school…[they] are able to come to me with any concerns, and then I can reach out to those families.” School personnel also described how having connections at different agencies was an advantage to connecting youth and families with services quickly. Another community school engagement specialist reported: I know a lot of the resources in the community. Before my school job, I was a home based worker… one of my biggest roles was connecting families to services that had open CPS cases, so when I came on to this role at the school as a Community Engagement Specialist, it really fit me well because I was already tied to the community. I already know a lot of the key players in Erie County so it makes it easy for me to refer 48 out because I can call and help them with their process. I know the key players of the different agencies in town. Another community school engagement specialist agreed, stating, “With my role, it’s neat, because I know a lot of the key players of these agencies so I might be able to get them [families] in quicker than if they’re just calling the agency themselves.” Participants also described using their connections to know what other resources that are available for youth and families. “I do have a list of people that I’ve worked with in the past, that I can say, “Who do you suggest?” said a school counselor regarding finding a best fit of services for a family. Youths’ individual needs are used to determine appropriate services. School and agency personnel describe how they assess the needs of each individual youth to connect that student with the appropriate services. School social workers and community school engagement specialists report using screening tools to determine the severity of youths’ mental health needs, and differentiating between symptoms of trauma and other mental health symptoms to connect that youth with the most appropriate services. When describing internal and external mental health services for youth, one school counselor reported, “I mainly do group therapy, but if I see that that’s not helping or they need more one-on-one, I can’t provide that here so then I will talk with the parents and say I’m really feeling like they could benefit for some more.” In this situation, more services would include referring the student to an external provider. While describing students’ mental health needs, a school social worker reported, “With this initiative, they fit within a certain individual’s needs, and so there’s a lot of consideration, especially in tier two or tier three situations”. Participants described using screening tools to determine which services may be more appropriate for students- particularly since the schools were using tiered systems of support. “The big thing that we are implementing now, it is the mental health screening.. I do this with the student. That’s the first process in any type of mental health concern 49 or any type of referral process. Another community school engagement specialist described using a screening tool to look not only at the student’s individual needs, but the needs of the family as well, reporting, “It’s a social determinant of health screen. It makes you go through…with the families and you ask them if there’s a need for transportation assistance… finding a job, or food assistance, or getting health insurance…electricity, housing…” Stakeholders believe an external referral pathway would be helpful. School and agency personnel report believing that an external mental health referral pathway would be beneficial in connecting youth with mental health services. One agency representative stated, “It would be great if we can get something that streamlines the process to makes it consistent, and that all the mental health providers could utilize.” They also report believing it would alleviate parents’ stress by preventing situations, such as multiple phone screenings, where parents would have to repeat information to many providers (i.e., reason for referral, insurance information). One community school engagement specialist stated, “I think that that would just be a huge benefit and I think that we would see a lot more children being serviced.” When discussing the idea of an external referral pathway, a school counselor stated, “That would be fabulous. Many times families experience numerous road blocks and frustrations [to accessing services].” Participants expressed hope in what an external referral pathway would look like, and the potential possibilities for such a system. One agency representative reflected: Even being able to have the option- it’s like, I’m not just referring you, I’m gonna help you set an appointment and let’s find the release all at the same time. There isn’t like five steps that need to be done, but its ‘Hi ma’am, I’m gonna make the referral. You give me your insurance information, you give me whatever, I’m gonna call the places and I’m gonna get everything setup for you. 50 CHAPTER FIVE: DISCUSSION Despite its limitations, this study provides insight into barriers and facilitators of implementing an external mental health referral pathway during the exploration phase of EPIS. This section will discuss how to findings relate to the research questions. Factors that stakeholders find acceptable about implementing an external referral pathway will be discussed first. Next, barriers to implementation will be discussed. After discussing findings that are identified as barriers to implementation, findings that include facilitators to implementation will be identified. Finally, implications for the field of marriage and family therapy and limitations of the study will be outlined. RQ1: What do stakeholders find acceptable about implementing a systematic external referral pathway? The idea of a systematic external referral pathway is “fabulous”. Participants found the idea of a systematic external referral pathway “fabulous”. In fact, multiple participants used that exact word to describe it. They liked the idea of an external referral pathway for many reasons. First, it was viewed as a streamlined process for connecting youth with mental health services, eliminating time and stress from school and agency personnel. They also believed it would provide a feedback loop, for school and agency personnel to follow up with one another on the status of referrals. Participants hoped this feedback loop would prevent students from falling through the cracks. Participants also believed it would promote transparency among multiple school and agency staff. Staff members will know who made referrals, which students were and were not referred for services, and believed it would allow easier tracking for school and agency administration to follow up with staff. It was also perceived as a way to alleviate parents’ stress. There is an idea that once a referral is made, the agency will be contacting the 51 parents and setting services up. This way, there is less work on behalf of the parents to call multiple agencies and/or providers to access mental health services. There is limited literature examining mental health referral pathways. The one exception I could find was Hall and Wurf’s evaluation of the capacity of a school referral pathway (2018). The findings from my study align with those of Hall and Wurf. They found that using an external school referral pathway to connect youth with mental health services increased staff’s perception of their ability to manage at-risk students. School personnel also believed that it reduced their overall workload. This could be due to the level of transparency the external referral pathway offered, so that school personnel were not duplicating efforts with students. The results of the study highlight the acceptability of a systematic external referral pathway. The participants in this study were from a primarily rural area. Further exploratory pre- implementation studies should examine the acceptability of external school mental health referral pathways across further rural areas. Additionally, further exploratory studies should examine acceptability of external school mental health referral in urban and suburban schools as well. Some schools and agencies already use an external referral pathway. Some school personnel and agency personnel reported they already use an external mental health referral pathway that fits their current needs. Agency personnel report creating this referral system due to the need for a consistent process with the schools they partner with. They report giving school personnel their release of information and referral forms that school personnel can use when meeting with parents, and then fax the forms back to the agency. Agency personnel reported they communicate back to school personnel on whether they were able to follow up with the parents or not. School and agency personnel reports that using the agency’s documents at the school have also helped to establish clear roles for the school, the parent’s and the agency. 52 As previously mentioned, there is a lack of research on external school mental health referral pathways. The current study mentions the helpfulness of having defined roles in the referral process. In both this study and Hall and Wurf’s study (2018), there are staff members at schools who specifically refer youth to services as part of their defined role within the referral process. Parents who participated in Hall and Wurf’s evaluation reported feeling supported by the person who referred their child for services, reported that having someone in that position is beneficial, and that it was helpful to get connected to resources quickly. These findings provide an important consideration for practice in schools. Having a specific liaison stationed in the school setting that is trained in connecting youth and their families with mental health services and other community resources (i.e., housing assistance, food pantry, etc.) could be helpful to connect families with the resources they need. Given the importance of having a specific role that is working with parents to help youth access mental health services, it has been recommended that school personnel referring youth for services join with parents by participating in home-visits, receive training and outreach for how to work with parents, and formal training for school staff on parent engagement (Nadeem, Santiago, Katoaka, Chang, & Stein, 2016). RQ 2: What barriers do stakeholders anticipate to implementing a systematic external referral pathway? Changing a system. Participants described how implementing an external mental health referral pathway impacts not only the schools, but other community stakeholders as well. They described how this process requires a paradigm shift among those involved, and that it can take time and be a slow process. Participants also identified the logistical challenge of having different procedures among agencies, and the process of shifting procedures to be in more 53 alignment with one another. Otherwise, there may different referral processes for each school- agency partnership. Findings from Hall & Wurf’s (2018) study indicated that implementing the referral process did have some challenges. Participants reported concerns about how the referral process would work, and were confused about the number of parties involved and how adequate communication between schools, agencies, and parents would occur. In a multi-site qualitative study, Hodges and colleagues (2012) reviewed documents from six systems of care. They found shared values and goals are not enough to create systems change. Collaborative structures (e.g., co-location of services, financial decisions) are needed to sustain systems changes. Furthermore, they recommend examining the systems change process through continuous quality improvement. Implications for practice include stakeholders providing a commitment to long term planning, as well as a commitment to systems change after funding runs out. An important consideration will be to determine the capacity needs in order to continue addressing systems change, and examining processes, after funding runs out. Further implications include aligning forms between agencies, and ensuring shared goals and values continue (Aarons et al. 2012, Hodges et al., 2012). These factors are imperative to having sustainability of the referral pathways. Technology. Stakeholders wondered what type of platform would host the systematic external referral system. During a prior attempt at establishing a systematic external referral pathway, a few participants discussed attempting to use Google documents to hold and transmit information; however, there were challenges associated with using that system. For example, some participants were unfamiliar with how to use Google docs. Others stated they had difficulty 54 opening the documents. Participants identified the importance of having a system to host the external referral pathway that would be user friendly to all of those that would be utilizing it. These findings align with technological challenges identified in the research literature, including differences in organizations’ IT capability and usability; IT system differences; privacy, confidentiality, and security; and staff awareness of differences (Holen-Rabbersvik, Thygesen, Eikebrokk, Fensli, & Slettebo, 2018; Yang & Maxwell, 2011). Holen-Rabbersvik and colleagues (2018) found these factors led to a lack of electronic health record usability, inadequate workflow processes, technological systems incompatibility. Additionally, they described the need for stakeholders to understand the needs of technological different systems, and knowledge and practices regarding privacy and confidentiality to adequate communicate between the systems. Yang and Maxwell (2011) report that coordinating inter-organizational technological needs can be a complex task. To better understand technology and information sharing in the context of school mental health referral pathways, future research studies could examine facilitators to successful information sharing. The SAMHSA School Referral Pathway Toolkit (2015) reports that some schools have utilized innovative platforms when creating an external referral pathway. An exploratory study examining what has worked, what has not worked, and lessons learned would be beneficial. An implication for practice includes identifying an IT specialist who can be tasked with creating an inter-organizational platform to host the referral pathway, and how that individual (or team) is funded. Having up to date information about providers. Participants reflected on the challenges of local providers and agencies having waiting lists. They described the importance of knowing which providers were accepting new clients, in order to limit the amount of referrals 55 families received before finding an opening. They described how families may give up on accessing services if too many providers have waiting lists. Furthermore, they described the need to have up to date information on the type of insurance providers would work with, in order to prevent referring a family to mental health services that a provider cannot work with. As previously mentioned, the SAMHSA School Mental Health Referral Pathway Toolkit (2015) mentions that some innovative schools have platforms that keep provider information up to date. Unfortunately, they do not give examples. This resource does have a sample database to show how schools can create a list of providers, insurances they take, services provided, etc. This is a spreadsheet that can be kept in a paper or word document. However, it does not address the challenges of keeping information up to date. Resources—including time and personnel—would be needed to keep the database updated. As mentioned above, an exploratory study examining schools that have referral platforms would be beneficial. Such a study would allow other communities to understand how it has been done before, what has worked, what has not, what platforms they have used, and how to have referral information available in real time. There is an overall lack of providers in the county. The most common barrier brought up by participants was the lack of mental health provider capacity in the county. They reported many mental health providers had no openings to take new clients and had waiting lists. They also mentioned many providers did not have training in specific child-focused EBTs targeting symptoms of trauma. The shortage of mental health providers is a systemic issue in the U.S. that has largely been documented (Aaronson & Withy, 2017; Baldwin et al., 2006; Butryn, Bryant, Marchionni, & Sholevar, 2017; Mackie et al., 2016; Thomas et al., 2009; Walker et al., 2015). A similar 56 challenge in another Project Aware funded rural school district was identified by Ryst and colleagues (2016). In their study, they reported an overall increase in students trying to access services; however, with a high volume of student referrals there was not enough providers in the local community (Ryst, Rock, Albers, & Everheart, 2016). In a review of literature, Grant and colleagues (2018) identified non-traditional ways to address the shortage of mental health service providers. They found literature supporting the use of peer support workers (PSWs), mental health first aid (MHFA), and community advocacy organizations. This an interesting suggestion, as MHFA and community advocacy organization increase help seeking behaviors, but still do not address the shortage of mental health providers. However, they do suggest peer support workers can provide psychoeducation on mental illnesses that can lead to symptoms reduction. Finally, the findings of the current study align with Allison, Roger, and Abbott (2008). In a survey of elementary and high school principals (n=113) and school counselors (n=58), Allison and colleagues stated school personnel from both urban and rural areas reported frustrations trying to refer youth and their families to mental health services. They found mental health providers often had waiting lists, which they believed led to families giving up on accessing services. Additionally, they report that school personnel were more reluctant to refer families due to the amount of waiting lists in their respective communities. Further research expanding on the findings from this current study could be helpful to understand strategies used to expand provider capacity. For example, an exploratory study of how awardees of Project AWARE grant funding have addressed mental health provider capacity would be beneficial. The Health Resources & Services Administration (HRSA) has attempted to address mental health provider capacity through the National Health Service Corps, which assists with student loan repayment to health and mental health professionals when they work in 57 designated “shortage” areas. This finding also has policy implications. For example, policy requiring Medicaid to have a minimum threshold of payment for providers that is comparative to private insurances may reduce staff turnover. Such a policy may be an incentive for private providers to panel with Medicaid and provide outpatient counseling to students who may not qualify for services at Community Mental Health agencies, but still benefit from counseling services. Lack of understanding about mental health issues. Some participants reported there is a lack of understanding about mental health issues. Oftentimes, participants were referring specifically to parents’ misunderstanding about how mental health issues might be impacting their child. Some participants attributed this to true misunderstanding and lack of knowledge. Other participants attributed the lack of knowledge to parents’ denial about an issue. Other participants speculated that parents did not want other family information coming out as a result (e.g., risk of getting Child Protective Services involved) of become involved with mental health providers. These findings are somewhat aligned with the research literature on perceptions of parental understanding of youth mental health. In a survey of 104 mental health providers across Kansas and nearby states, nearly half of the providers reported that parents had no or little knowledge about children’s mental health disorders. When asked about the knowledge that parents do have, participants described it as “very inaccurate” or a mixture of accurate and inaccurate information (Frauenholtz, Conrad-Hiebner, & Mendenhall, 2015). It is important to note these findings only include mental health providers’ perceptions, and they’ve already come into contact with families and have been providing mental health services. The sample in this study does not include school personnel who are trying to connect youth and families with 58 mental health services. Most research in this area examines mental health providers experiences of working with parents (Baker-Ericzen et al., 2013), the therapist-parent alliance (Diamond, Diamond, & Liddle, 2000; Feder & Diamond, 2016; Fox, 2012; Lawson & Brossart, 2003) or parents’ perceptions of barriers to accessing mental health treatment (Baker-Ericzen et al., 2013; Reardon, Harvey, Baranowska, O’Brien, Smith, & Creswell, 2017; Reardon, Harvey, Young, O’Brien, & Creswell, 2017). Other researchers that have examined providers’ perceptions of parental understanding of youth mental health symptoms have focused on cultural factors (see Colucci, Minas, Szwarc, Guerra, & Paxton, 2015; Khanlou, Haque, Sheehan, Jones, 2015). Future studies should continue to explore school personnel’s perceptions and experiences when referring youth and families to mental health services. Additional research exploring families’ experiences during the referral process could inform school personnel’s referral practices. Information on strategies that have been helpful could be disseminated to inform other helping professionals. Navigating confidentiality. Participants expressed concerns about confidentiality if a systematic referral pathway were to be implemented. Participants wondered who will receive, and who will see, the information included in the referral form. Participants also wondered how the information would be used, aside from connecting youth with referrals (e.g., data collection). One participant described concerns over how the information shared during the referral process might impact the youth in the future (e.g., impact enrollment in the military). Participants also wondered how they could share information between agencies and schools in a way that complies with state and federal privacy regulations. The current study expands on previous research findings regarding confidentiality and implementation in school settings. When exploring implementation factors for a suicide 59 prevention program in schools, Stein and colleagues (2010) found that school personnel reported being unaware of which students were at-risk, and if any data on at-risk students existed at their schools. Participants attributed their uncertainty to confidentiality, and believed confidentiality prevented them from knowing more about mental health crises that have occurred at the school. Whereas Stein and colleagues (2010) identified how confidentiality could impede internal communication about at-risk students this study focuses on how to communicate externally to mental health providers. There are several implications for practice. Identifying an existing platform to host an external referral pathway that can meet state and federal regulations to uphold privacy regulations can be quite challenging. School and agency personnel could continue to use the referral pathway they currently have in place. Many participants expressed the referral methods they were currently using worked well for them. School and agency personnel could also continue to explore platforms that schools in other communities have used. However, this requires time and resources to adequately research, plan, and implement a new platform. As mentioned above, future research studies could expand in this area through the use of an exploratory study. The SAMHSA School Referral Pathway Toolkit mentioned that several schools had successfully used platforms to refer students. An exploratory study could identify which platforms those schools used, and compare the benefits and costs of each platform. Lack of resources. Both school and agency personnel cited a lack of resources (i.e., time, money, personnel) as a barrier to implementing a systematic external mental health referral pathway. Although schools have been identified as an ideal location to provide mental health services to youth (Fazel et al., 2014; Rones & Hoagwood, 2000) researchers have noted that many schools struggle to secure resources to provide these services (Atkins, Cappella, Shernoff, 60 Mehta, & Gustafson, 2017; Neufeld et al., 2017). Furthermore, some schools that do provide mental health services struggle to sustain provision of services (Foster et al., 2005; Leachman et al., 2016). A lack of resources in schools can be reflective of funding in the outer context of the EPIS model. Atkins and colleagues (2016) have identified the need for more researchers to use dissemination and implementation approaches to disseminate effective approaches to school mental health moving forward. As part of their strategic initiatives, federal funding agencies (e.g., NIMH) call for the use of dissemination and implementation science methods to address the practice-research gap. Implications for researchers include gaining additional knowledge and expertise in the field of dissemination and implementations science. One such training opportunity for researchers is the Implementation Research Institute (IRI). The IRI is meant advance the field of implementation science in mental health by providing professional development to researchers. Additional opportunities for gaining knowledge include applying for professional membership in the Society for Implementation Research (SIRC), attending SIRC’s biennial conference, or attending another D&I conference. Lack of connection with parents can get in the way of connecting youth with mental health services. Participants described connecting with parents can often impede connecting youth with mental health services. Participants described the difficulty making contact with parents (e.g., answering phone calls). Participants also described how when they are able to connect with parents, some parents do not believe their child needs- or would benefit- from services. There is limited research on school personnel’s perceptions of how parents impact referring youth to mental health services. Researchers have explored how connecting with 61 parents might impact mental health services; however, this research is usually mental health providers’ perspectives on parental involvement of children that are already involved in services (see Baker-Ericzen et al., 2013; Haine-Schlagel, Brookman-Frazee, Fettes, Baker-Ericzen, & Garland, 2012). Further exploratory research on school personnel’s experience connecting with parents to link youth to mental health services would be helpful. Strategies that have been successful could be explored, as well as understanding strategies that have not been successful. RQ3: What facilitators do stakeholders anticipate would aid in successful implementation of a systematic external referral pathway? Stakeholders believe an external mental health referral pathway would be helpful. Participants reported they have previously tried to implement a systematic external pathway via an online platform and were unsuccessful. Furthermore, participants reported they do intend to adopt a systematic referral platform if they find a feasible means to do so. Several participants believed implementing such a pathway would be “fabulous” and expressed excitement about the possibility of being able to do so. Overall, participants were receptive to the idea of an external systematic referral pathway because they believed it would help them to better serve the youth and families in their community. This finding relates to the research literature in two ways. First, the intent for adopting an external school mental health referral pathway expressed by participants parallels research literature identifying readiness for change as a necessary component for adoption of EBPs. In a survey of ninety-two nurses, Gale and Schaffer (2009) identified facilitators to organizational readiness. The top three facilitators to implementing an evidence-based practice included: 1) the application of the EBP is essential for job duties; 2) staff believe research literature and findings are applicable to everyday practice; and 3) adopting the EBP does not require a significant amount of time. This is similar to the findings in the 62 current study, when participants have reflected on valuing their work with families, and the need to connect these families with relevant resources. Implementing an external mental health referral pathway would help participants to carry out their work with families. Second, Hall and Wurf’s (2018) evaluation of an external school mental health referral pathway found many benefits. These benefits included a reduced workload for school personnel, an increased ability for school personnel to manage at-risk students, and that students felt more comfortable speaking with a counselor outside of school. It is possible that the participants in this study have hope for a similar outcome- and may experience a similar outcome- if an external mental health referral pathway is implemented. An important implication for both research and practice is to assess readiness for change. Many current tools exist to assess readiness for change in an organization (e.g., ORCA, OR4KT). Using such tools can identify important factors to consider (e.g., psychological readiness, availability of resources) and save personnel valuable time and resources before attempting to implement an EBT (Ritchie & Straus, 2019). Practitioners and researchers should utilize these tools into school-based projects to determine readiness for implement of EBTs. Meeting consistently among community partners. Participants reported consistently meeting with various community partners to review shared goals, troubleshoot challenges, and share successes. Participants reported these meetings were also helpful when conducting assessments of students’ needs. Additionally, they report finding the meetings helpful for understanding where there were gaps in services. This finding supports research literature on having an inter-organizational network to address systems change. Hodges and colleagues (2012) identified six lessons for systems change to be successful when utilizing an inter-organizational network. These lessons include: 1) 63 creating an early and consistent focus on values and beliefs; 2) translating shared beliefs into shared responsibility and shared actions among members; 3) recognizing that systems change is not linear; 4) knowing that being concrete does not mean being static; 5) structural change- without shared values and goals- rarely has the sustained positive impact that implementers seek; and 5) the systems change emerges from the individual choices and actions of stakeholders placed throughout the system. There are several practical implications for consistently meeting. Meeting consistently amongst community providers creates opportunity for collaboration. Furthermore, it also provides opportunity to enhance current services and identify services that are needed in the community. Another practical implication of meeting consistently amongst community partners is that when there are new resources in the community, it can be brought to the attention of many community stakeholders at once. New resources could also potentially create alternative options for an external mental health referral pathway. Interdisciplinary meetings to review students. School personnel reported they appreciated meeting consistently with other school staff to review students. They described the meetings as occurring consistently at the same time each week or month. Participants reported that it was helpful to have a specific time set aside to discuss students, in order to facilitate transparent communication about students’ needs. School personnel’s reports on the benefit of meeting consistently align with findings from Palinkas and colleagues (2014) on implementing EBPs in youth serving agencies. Specifically, they identified four characteristics leading to effective collaborative processes when meeting, including: focus, formality, frequency, and function. Focus refers to how broad or narrow the purpose of the meeting is. In the current study, participants identified the focus of the meetings 64 as reviewing students’ who may need more support. Formality is described as the method of communicating needs. Participants reported meeting in person. Frequency is how often the meetings or methods of communication occur. Participants reporting meeting consistently, however, the frequency varied by school (e.g., weekly v. bi-weekly). Identifying the purpose of the meetings is the function. Participants identified the function of the meeting as to identify action steps for connecting youth with supportive services. Next steps for research could include exploring how alignment of meeting criteria for collaborative processes impacts the function and focus of the meeting. For example, this could include examining how school personnel who consistently meet to review students impacts the number of student referrals, and to which resources those students are being referred to. Further exploratory studies could also examine school personnel’s experiences during these meetings. Such research could identify other components of meetings that are useful, or act as barriers to the function of the meetings. Grant funding has allowed for more resources. Participants described how grant funding has allowed the school districts prioritize youth mental health needs more fully. Some participants described how grant funding contributed to their yearly salary. Participants also reflected on how funding had allowed local clinicians to receive training on trauma informed practices. There appears to be limited research on how grant funding has been used to build capacity for mental health resources in schools. Research literature appears to be focus on the reach of specific grants or funding sources, or the outcomes of those funding sources (see Han, Luo, & Ku, 2017; Guegan, Dorling, Ollerhead, & Westmore, 2016; Proctor, McMillan, 65 Haywood, & Dore, 2008; Sprigg, Wolgin, Chubinski, & Keller, 2017). Furthermore, research literature appears to focus on larger systems in the community rather than school systems. These findings can be taken into consideration to inform future areas of research. Researchers could conduct a systematic literature review to understand what literature is available on funding in school mental health. Exploratory research studies could also be conducted to understand the limitations of grant funding in school mental health, and generate a research agenda for future areas of focus. Agency personnel providing services in schools. Another facilitator described by school personnel is having the support of agencies. A specific example of school personnel feeling supported is when agency’s offer the opportunity to have mental health clinicians provide counseling services at the school. Participants report that having agency personnel providing services in schools has been imperative to connecting students with mental health services; particularly students whose families may face too many barriers to access services at a location outside of the school. Schools have been identified as a solution to addressing mental health needs in youth (Hoover, et al., 2007). Agencies providing services in schools has been a response to addressing system changes to address youth mental health needs in a comprehensive manner (Regan et al., 2017). Researchers have also linked mental health provided to at-risk students in schools has been linked to reduced number of suspensions, improved attendance rates, and improved teacher reports for students’ peer relationships and externalizing behaviors (Ballard, Sander, & Klimes- Dougan, 2014). A practical implication is for communities to adopt a co-location model of mental health services for youth. A co-location model is when an agency places a provider in another setting, 66 like a school. This could be particularly beneficial in rural areas, where mental health agencies or private providers may be a considerable driving distance for families. Additionally, placing agency clinicians in schools can address barriers to mental health service access for low-income families. Many families may not have money for gas, or may be working long hours and unable to take their children to appointments. Having agency clinicians continue to provide services in school is a strategy to address this problem. Administrative support. Participants described having how having administrative support is a helpful factor when considering implementing an external mental health referral pathway. They described how having administrative support allowed them to feel as if they could advocate more for the families they serve, thereby performing their job duties to the best of their abilities. Administrative support can influence the implementation of EBPs (Aarons et al., 2011; ok et al., 2015; Damschroder et al., 2009; McIntosh et al., 2016; Sarno Owens et al., 2014; Schoenwald & Hoagwood, 2001). Administrative support, or lack thereof, can have a positive or negative influence on how EBP implementation is viewed (Sarno Owens et al., 2014). Additionally, higher implementation rates are likely to occur when different types of school personnel who are trained in youth mental health EBPs (Langley et al., 2010; Sarno Owens et al., 2014; Stein et al., 2010). Having administrative support can be beneficial to the adoption of an external mental health referral pathway. Other indicators- such as more teachers and other school personnel being trained to identify youth mental health symptoms- create a climate supportive of implementing an external youth mental health referral pathway. Forms of administrative support described by participants are reflective of an atmosphere promoting high levels of 67 implementation. Further implications for practice when using an external mental health referral pathway include identifying specific roles in the referral process. The EPIS model refers to this as “role specialization” (Aarons et al., 2011). Having specific roles can clarify expectations and job duties. Administration can also advocate for more mental health specific staff, or staff to aid mental health staff, in carrying out essential job duties. Mental health staff as a resource. School and agency personnel described how staff are an invaluable resource when managing the workload that comes with referring youth to mental health services. School personnel identified how their position exist due to federal grant funding, which has allowed them to spend their time assessing and referring youth to services, as many other school personnel do not have the time or knowledge to do so. Having these specific staff time to assess and refer youth to appropriate internal or external mental health services. Agency personnel also identified how having specific staff involved in receiving external referrals from schools has been imperative to more streamlined methods of functioning. Having specific staff to refer youth to appropriate internal and external mental health services is important when utilizing tiered systems of support. Specific school staff focusing on students’ mental health needs is an invaluable resource, as many schools lack resources to address youths’ mental health needs (Neufeld, et al., 2017). Implications for practice include examining schools’ absorptive capacity. Absorptive capacity is the school personnel’s knowledge and skills as they pertain to an external mental health referral pathway, and ability to incorporate the knowledge and skills into the school setting (Aarons et al., 2011). A higher level of absorptive capacity can indicate that school is ready to move to the adoption decision /preparation phase of the EPIS model, or determine if more staff supports need to be in place before moving forward with EBP implementation. One example would be to have a position that 68 specifically deals with paperwork or service coordination. This could be a master’s level intern who can coordinate referral processes for students and complete necessary paperwork for other agencies, or for the schools records. Increased awareness of mental health issues. Participants believed that more teachers are able to recognize mental health issues in students. They attributed the increase in awareness to two factors. First, that teachers had received more training in youth mental health symptoms and learned strategies to better address youths’ needs. Second, the increase in awareness was attributed to the current culture, and an increase of mental health situations arising at schools in general. School personnel can be among the first to notice mental health issues in youth and begin linking youth with mental health services (Sayal, 2006). Implementing mental health first aid training in community settings can help community members become more aware of mental health symptoms (Grant, Simmons, & Davey, 2018). Haggerty and colleagues (2018) conducted a pre-test/post-test survey of mental health workforce and non-mental health workforce learning outcomes after participating in YMHFA training. Non-mental health workforce participants included school personnel (i.e., teachers, administrators, support staff, nurse, student teachers). They found that non mental health workforce participants scored higher in mental health literacy from pre-test to post-test, with sustained knowledge at a 3 month follow up. Based on the current study, it appears that school personnel’s levels of increased awareness are recognized by other school personnel and agency personnel. This finding has implications for practice and research. When school personnel are able to identify mental health symptoms in youth, youth may have an increased advantage to being connected with internal or external mental health services. Youth who are referred for mental 69 health services from schools are likely to have decreased rates of truancy and suspensions, and mental health outcomes (Greif Green et al., 2016). Unfortunately, there is little research exploring youth outcomes after being referred to internal and external mental health services from schools. Researchers could continue to examine youth outcomes after participating in different forms of mental health treatment, inside and outside of the school context. School and agency personnel value their role in working with families. School and agency personnel identified the importance building relationships with families. Participants described how building relationships with families is imperative to connecting them with external services to benefit the child’s well-being. Additionally, participants described the collaborating with families as a rewarding part of their job. This finding aligns with research literature in two ways. First, implementation research shows how an individual’s values can influence implementation of an EBP (Schoenwald & Hoagwood, 2001). Second, mental health clinicians are more likely to support implementation if they believe it will help them to be better at their job (Lyon et al., 2013). In terms of practical implications, if school and agency personnel value their role in working with families, they may be more likely to find an external mental health referral pathway appropriate, acceptable, and adoptable if it can help them to be more effective at their job. One area this study did not examine was how parents may contact the school and ask what internal and external services are available for their child. Future areas of research could include examining how parents may initiate referral pathways, using schools as the hub. Having professional networks enhances individuals’ ability to connect youth with mental health services. Some participants described prior experience working in the social services field in the county. They describe how this experience was helpful to their current role 70 in the school, because they were familiar with most of the resources in the area. Not only were they familiar with resources, they reported knowing staff at the other agencies. Participants reported that having these connections allowed them to connect families quicker than if the families had gone through more of a traditional intake process. The current findings expand on the research literature by discussing how professionals’ social networks can connect youth more quickly to mental health services. In an evaluation of a school-based child and family team model, Gifford and colleagues (2015) found that the quicker professionals followed up with families about receiving mental health services, youth had greater odds of receiving services. For example, they found that for youth who were followed up with within fifteen days, were more than twenty times at greater odds that services would be received sooner than youth who were followed up with after fifteen days. This finding has implications for both practice and research. The social networks of school personnel could be a valuable resource when connecting youth to mental health services in a timely manner. School personnel dedicate time to enhancing their social networks with agency personnel they may be likely to refer to. Researchers could explore school personnel’s social networks, and which networks may quicken access to services for youth. Furthermore, characteristics of these social networks and interpersonal relationships could be explored to see how they may impact the referral process. Implications for the field of Marriage and Family Therapy Considerations from this study can be used to inform the field of marriage and family therapy (MFT) in practice and in research. This study utilized principles from dissemination and implementation science. Dissemination and implementation science is a growing field that marriage and family therapists are especially poised to address (Withers and colleagues, 2016). 71 Withers and colleagues suggest many reasons MFTs can- and should- contribute to dissemination and implementations science research. First, dissemination and implementation science research can help to bring the researchers-practitioner divide. Second, there are several funding streams that can expand opportunities for MFTs to conduct meaningful research, as well as enhance the impact of such research. Third, MFTs can reinforce the integration of systems theory within dissemination and implementation science research. Finally, MFTs can use dissemination and implementation science to examine the implementation of systemic interventions. In order to prepare future generations of MFT researchers to utilize dissemination and implementation science methods, more advanced training programs could begin to incorporate classes and/or coursework in this area. The findings of this study can also inform the clinical practice of MFTs, as well as other mental health providers. Participants in this study explained the importance of referring to providers that had the “right type” of training, primarily referring to providers that were trained in TF-CBT. Considering these findings, MFTs may benefit from receiving trainings on EBTs that are trauma-informed and child focused. Such trainings- and even certifications- can prepare MFTs to provide services that are in line with best practices for children who have experienced trauma. Participants in this study also expressed frustration that many providers had long waiting lists. One potential implication could be that practicing MFTs reserve a specific number of slots on their caseload specifically for youth referred by schools. Participants in this study also described how it was helpful to have agency personnel provide services within schools, which helped address many barriers preventing families from accessing mental health services outside of school hours. MFTs can take this finding into consideration by partnering with schools to 72 provide clinical services at the school building. Providing services at the school building could help address many barriers for families who may not be able to access services otherwise. Limitations The current study is not without its’ limitations. First, this study examined stakeholders from one county intermediate school district’s thoughts on implementing an external mental health referral pathway in the context of Project AWARE. This county was mostly rural. Other intermediate school districts that had received Project AWARE funding were not examined. Intermediate school districts in other counties that are more populated may have different results. Future exploratory research could compare the results of urban and rural schools who are examining external mental health referral pathways. Second, this study examined a small sample of stakeholders who were in some way connected with the implementation of an external mental health referral pathway. Personnel from only one mental health agency were interviewed, as that would be the agency an external mental health referral pathway would be piloted with. Interviews with stakeholders from other mental health agencies may have provided a different perspective than was included in the results of this study. Valuable insight into the experience of being referred from schools to mental health agencies could be provided by parents and youth. Their experiences could potentially informed processes that may be more comfortable for their needs. Parents were not interviewed for the current study, as it followed similar study protocol by Aarons and colleagues (2012) that also did not involve parents. Future research should examine parent and youth perspective in the referral process to better understand service level processes (Hamilton et al., 2014; Stein et al., 2010; Walker, Koroloff, & Bruns, 2010). 73 Third, the results of this study found limited data to inform most sections of the outer context of the EPIS model (i.e., sociopolitical context, funding, client advocacy). This could be due to many factors. One explanation could be that the interview questions did not elicit information from participants that would inform the outer context. Another explanation could be that many participants’ roles in the external mental health referral pathway were not relevant to certain factors in the outer context. Another possible explanation could be that stakeholders who may have more insight into the outer contextual factors of the EPIS model were not interviewed. Future research should focus on gathering data to inform the outer context of the EPIS model, and how those factors may impact the implementation of an external mental health referral pathway. Finally, this study examined the implementation of a systematic external mental health referral pathway. It did not examine how what students would benefit the most from such a pathway. For example, students who experience internalizing symptoms may not be as likely to enter into the referral process as their peers who exhibit externalizing behaviors. Future examinations should examine how such a referral pathway could benefit all students, and examine was gaps may potentially exist when addressing youth mental health. Conclusion This study examined the acceptability, adoptability, and appropriateness of a systematic external mental health referral pathway to be used between schools and a local mental health agency. In a previous systematic literature review examining EBP implementation in child and adolescent mental health, only one study was found to focus on the exploration phase of EPIS. The current study expands on the field of implementation science by focusing on factors related to pre-implementation of a practice in the school setting. School and agency personnel reflected 74 how an external mental health referral pathway was appropriate, believing it would fit their needs well. This was evidenced by how they participants described student mental health as a high priority throughout the schools in the intermediate school district. Participants expressed mixed thoughts on the adoptability of an intervention. While some participants described previously attempted to implement a referral system via an online platform- and their hopes to succeed in doing so in the future- other participants reported they already had an external mental health referral process that worked well for their needs. The participants who reported already having an external mental health referral pathway used email or fax to send referrals and follow up with the referring party and with parents of youth. Participants also believed that implementing an external mental health referral pathway was acceptable. Many participants described it as “fabulous”. However, it is possible that participants may have idealized what an external mental health referral pathway may look like, potentially believing that it may have more features than it could actually have once implemented. For example, participants expected certain information (i.e., student demographic information, insurance information) and processes (i.e., feedback loops, agency contacting parents) to occur, when it may depend on the capabilities of an online platform that is used to host the external mental health referral pathway. Overall, school and agency personnel indicated the thought of using a systematic external mental health referral pathway was a good fit for their needs and met their schools’ goals, finding it acceptable, adoptable, and appropriate. Future areas of research could explore schools that have successfully implemented a systematic referral pathway and disseminate findings. Particularly, it would be helpful to note which online platforms were used to host the referral pathways, as well as how user-friendly the platforms are. Pre-implementation studies for school based EBTs are limited, therefore future areas of research could focus on this area of implementation science as well. 75 Research focused on the exploration phase of EPIS- or pre-implementation of an intervention- could be used to inform the appropriateness of future EBT implementation into schools. 76 APPENDICES 77 APPENDIX A Telephone Consent Hello, my name is Erica Nordquist. I am a graduate student from Michigan State University conducting a research study about mental health referral processes. Your participation in this research is completely voluntary. This means that you do not have to participate unless you want to. Today you will be participating in a individual phone interview, which should take approximately 30 to 60 minutes to complete. Your participation is voluntary. If you do not wish to participate, you may stop at any time. Responses will be transcribed and summarized with other responses. Responses are anonymous and names will not be used. However, the results will be grouped by roles (i.e., school personnel, agency personnel). There are minimal risks associated with this interview. Taking part in this interview is your agreement to participate. Would you be willing to answer some questions to help me determine if you are eligible for this study? (If yes, proceed; if no thank them for their time and end the call). Good. I will read off a list of questions. Please answer yes or no. Are you an employee of one of the JISD schools? Do you serve in a role where you refer youth to mental health services? Do you working in a partnering mental health agency that accepts referrals from schools? Do you speak English? Do you agree to participate in this study? The purpose of this research study survey is to look at school mental health referral processes. We estimate that approximately twenty people will enroll in this study. You will be asked to answer a series of demographic questions and a complete a phone interview. This should take about 30-60 minutes. There is a small chance that some of the questions may make you feel uncomfortable. You don’t have to answer those questions if you don’t want to. In fact, you don’t have to answer any question that you choose not to answer. We will just skip that question and go on to the next one. All the information I receive from you by phone, including your name and any other identifying information will be strictly confidential and will be kept under lock and key. I will not identify you or use any information that would make it possible for anyone to identify you in any presentation or written reports about this study. If it is okay with you, I might want to use direct quotes from you, but these would only be quoted as coming from “a person” or a person of a certain label or title, like “one woman said.” When I finish with all the phone surveys from everyone who has agreed to participate, I will group all the answers together in any report or presentation. There will be no way to identify individual participants, however, response will be grouped by job category (i.e., school personnel, agency personnel). 78 The only risk to you might be if your identity were ever revealed. But I will not even record your name with your responses, so this cannot occur. There are no other expected risks to you for helping me with this study. There are also no expected benefits for you either. If you have concerns or questions about this study, such as scientific issues, how to do any part of it, or to report an injury, please contact the researcher Marsha Carolan at 552 W Circle Dr, East Lansing, MI 48824, via email at carolan@msu.edu, or by phone at 517-432-3327. If you have questions or concerns about your role and rights as a research participant, would like to obtain information or offer input, or would like to register a complaint about this study, you may contact, anonymously if you wish, the Michigan State University’s Human Research Protection Program at 517-355-2180, Fax 517-432-4503, or e-mail irb@msu.edu or regular mail at 4000 Collins Rd, Suite 136, Lansing, MI 48910. Do you have any questions? Do I have your permission to begin asking you questions? 79 APPENDIX B Demographic Questionnaire Participant #: ________________ Date: ______________________ Directions: Please indicate the response that best describes you. 1. Date of Birth: _______________ 2. What is your race? a. Native American/Alaskan Native b. Native Hawaiian/Pacific Islander c. Asian d. African American e. Hispanic f. Caucasian g. Multiracial 3. What is your age? a. 22-29 b. 30-36 c. 37-44 d. 45-52 e. 53-60 f. 60 and older 4. What is your highest level of education? a. High school graduate b. Some college, no degree c. Associate’s degree d. Bachelor’s degree e. Master’s degree f. Doctorate degree g. Professional degree 5. What is your yearly gross income? a. $0-$19,000 b. $20,000-$40,000 c. $40,000-$60,000 d. $60,000-$80,000 e. $80,000-$100,000 f. $100,000 and above 80 APPENDIX C Agency Personnel Interview Guide RQ1: What do intermediate school district key informants find acceptable about implementing a systematic external referral pathway? 1. What are your thoughts about implementing an external referral process? 2. How do student mental health needs fit with your agency’s priorities? 3. Can you talk about how you make decisions about accepting referrals from schools? RQ2: What barriers do stakeholders anticipate to implementation of a systematic external referral pathway? 1. What might interfere with the implementation of an external referral process? What challenges do you expect/encounter? a. Prompts: resources? Technology? Policy? RQ3: What facilitators do stakeholders anticipate would aid in successful implementation of a systematic external referral pathway? 1. What processes are already in place to address youths’ mental health needs? 2. When partnering with a school, what supports are in your agency to make implementation of an external referral process successful? 3. What additional supports are needed to make implementation of an external referral process successful? a. Prompts: in your school? In a partnering agency? In the community 81 APPENDIX D School Personnel Interview Guide RQ1: What do intermediate school district key informants find acceptable about implementing a systematic external referral pathway? 4. What are your thoughts about implementing an external referral process? 5. How do student mental health needs fit with your school or school district’s priorities? 6. Can you talk about how you make decisions about referring to an agency or clinician? RQ2: What barriers do stakeholders anticipate to implementation of a systematic external referral pathway? 2. What might interfere with the implementation of an external referral process? What challenges do you expect/encounter? a. Prompts: resources? Technology? Policy? RQ3: What facilitators do stakeholders anticipate would aid in successful implementation of a systematic external referral pathway? 4. What processes are already in place to address youths’ mental health needs? 5. What supports are in your school or school district to make implementation of an external referral process successful? 6. What additional supports are needed to make implementation of an external referral process successful? a. Prompts: in your school? In a partnering agency? In the community? 82 APPENDIX E Audit Trail Record Date Task Action Result 83 REFERENCES 84 REFERENCES Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23. Aarons, G.A., Green, A.E., Palinkas, L.A., Self-Brown, S., Whitaker, D.J., Lutzker, J.R., …Chaffin, M.J. (2012). Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implementation Science, 7(32). Retrieved from: http://www.implementationscience.com/content/7/1/32. Aaronson, A., & Withy, K. (2017). Does Hawai’i have enough psychiatrists? Assess the mental health workforce versus demand in the Aloha state. Hawai’i Journal of Medicine & Public Health, 3(1), 15-17. Alegria, M., Lin, J.Y., Green, J.G., Sampson, N., Gruber, M.J., & Kessler, R. (2012). Role of referrals in mental health service disparities for racial and ethnic minority youth. Journal of the American Academy of Child & Adolescent Psychiatry, 51(7), 703-711. Allison, S., Roeger, L., & Abbot, D. (2008). Overcoming barriers in referral from schools to mental health services. Child and Adolescent Psychiatry, 16, 44-47. Atkins, M.S., Cappella, E., Shernoff, E.S., Mehta, T.G., & Gustafson, E.L. (2017). Schooling and children’s mental health: Realigning resources to reduce disparities and advance public health. Annual Review of Clinical Psychology, 13, 123-147. Atkins, M.S., Rusch, D., Mehta, T.G., & Lakind, D. (2016). Future directions for dissemination and implementation science: Aligning ecological theory and public health to close the research to practice gap. Journal of Clinical Child and Adolescent Psychology, 45(2), 215–226. Anda, R.F., Whitfield, C.L., Felitti, V.J., Chapman, D., Edwards, V.J., Dube, S.R., Williamson D.F. (2002). Adverse childhood experiences, alcoholic parents, and later risk of alcoholism and depression. Psychiatric Services, 53(8), 1001–1009. Baker-Ericzen, M.J., Jenkins, M.M., & Haine-Schlagel, R. (2013). Therapist, parent, and youth perspectives of treatment barriers to family-focused community outpatient mental health services. Journal of Child and Family Studies, 22, 854-868. Baldwin, L., Patanian, M.M., Larson, E.H., Lishner, D.M., Mauksch, L.B., Katon, W.J., Walker, E., Hart, L.G. (2006). Modeling the mental health workforce in Washington state: Using state licensing data to examine provider supply in rural and urban areas. The Journal of Rural Health, 22(1), 50-58. 85 Ballard, K.L., Sander, M.A., Klimes-Dougan, B. (2014). School-related and social-emotional outcomes of providing mental health services in schools. Community Mental Health Journal, 50, 145-149. Bazeley, P., & Richards, L. (2000). The NVivo qualitative project book. Thousand Oaks, CA: SAGE Publications. Brannlund, A., Strandh, M., & Nilsson, K. (2017). Mental health and educational achievement: The link between poor mental health and upper secondary completion and grades. Journal of Mental Health, 26(4), 318-325. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77-101. Braun, V., & Clarke, V. (2014). Guest editorial: What can “thematic analysis” offer health and wellbeing researchers? International Journal of Qualitative Studies on Health and Well- being, 9, 1-2. Buckingham, S.L., Brandt, N.E., Becker, K.D., Gordon, D., & Cammack, N. (2016). Collaboration, empowerment, and advocacy: Consumer perspectives about treatment engagement. Journal of Child and Families Studies, 25, 3702-3715. Burns, B.J., Phillips, S.D., Wagner, H.R., Barth, R.P., Kolko, D.J., Campbell, Y., & Landsverk, J. (2004). Mental health needs and access to mental health services by youths involved with child welfare: A national survey. Journal of American Academy of Child and Adolescent Psychiatry, 43(8), 960-970. Butryn, T., Bryant, L., Marchionni, C., & Sholevar, F. (2017). The shortage of psychiatrists and other mental health providers: Causes, current state, and potential solutions. International Journal of Academic Medicine, 3(1), 5. Carter, N., Bryant-Lukosius, D., DiCenso, A., Blythe, J., & Neville, A. J. (2014). The Use of Triangulation in Qualitative Research. Oncology Nursing Forum, 41(5), 545-547. Centers for Disease Control. (2017). Suicide among youth. [Fact sheet]. Retrieved from: https://www.cdc.gov/healthcommunications/toolstemplates/entertainmented/tips/Suicide Youth.html Chapman, D.P., Anda, R.F., Felitti, V.J., Dube, S.R., Edwards, V.J., Whitfield, C.L. (2004). Adverse childhood experiences and the risk of depressive disorders in adulthood. Journal of Affective Disorders, 82, 217–225. Colucci, E., Minas, H., Szwarc, J., Gurra, C., & Paxton, G. (2015). In or out? Barriers and facilitators to refugee-background young people accessing mental health services. Transcultural Psychiatry, 52(6), 766-790. 86 Cook, C.R., Lyon, A.R., Kubergovic, D., Wright, D.B., Zhang, Y. (2015). A supportive beliefs intervention to facilitate the implementation of evidence-based practices within a multi- tiered system of supports. School Mental Health, 7, 49-60. Cooper, J.L., Aratani, Y. (2014). Children’s mental health policies in the United States: Perspectives from advocates and state leaders. Health Expectations, 18, 2213-2222. Creswell, J. (2007). Qualitative inquiry & research design: Choosing among five approaches. (2nd ed.). Thousand Oaks, CA: Sage Publications. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(1), 50. Diamond, G.M., Diamond, G.S., & Liddle, H.A. (2000). The therapist-parent alliance in family- based therapy for adolescents. Journal of Clinical Psychology, 56(8), 1037-1050. Dube, S.R., Anda R.F., Felitti, V.J., Chapman, D., Williamson, D.F., Giles, W.H. (2001). Childhood abuse, household dysfunction and the risk of attempted suicide throughout the life span: Findings from Adverse Childhood Experiences Study. JAMA,286, 3089–3096. Eccles, M.P., & Mittman, B.S. (2006). Welcome to implementation science. Implementation Science, 1(1), 1-3. Fazel, M., Hoagwood, K., Stephan, S., Ford, T. (2014). Mental health interventions in schools in high-income countries. Lancet Psychiatry, 1(5), 377-387. Feder, M., & Diamond, G.M. (2016). Parent-therapist alliance and parent attachment-promoting behavior in attachment-based family therapy for suicidal and depressed adolescents. Journal of Family Therapy, 38(1), 82-101. Foster, S., Rollefson, M., Doksum, T., Noonan, D., Robinson, G., & Teich J. (2005). School mental health services in the United States 2002–2003. Washington, DC: U.S. Department of Health and Human Services. Fox, J.E. (2012). “Co-constructing” stigma and the therapist-parent alliance. Psychotherapy, 49(1), 38-45. Frauenholtz, S., Conrad-Hiebner, A., & Mendenhall, A.N. (2015). Children’s mental health providers’ perceptions of mental health literacy among parents and caregivers. Journal of Family Social Work, 18(1), 40-56. Gale, B., & Schaffer, M.A. (2009). Organizational readiness for evidence-based practice. The Journal of Nursing Administration, 39(2), 91-97. 87 Gifford, E.J., Wells., R.S., Bai, Y., & Malone, P. S. (2015). Is implementation fidelity associated with improved access to care in a school-based child and family team model? Evaluation and Program Planning, 49, 41-49. Grant, K.L., Simmons, M.B., & Davey, C.G. (2018). Three nontraditional approaches to improving the capacity, accessibility, and quality of mental health services: An overview. Psychiatric Services, 69, 508-516. Green, J.G., Xuan, Z., Kwong, L., Anderson, J., & Leaf, P.J. (2016). School referral of children with serious emotional disturbance to systems of care: Six-month clinical and educational outcomes. Journal of Child and Family Studies, 25, 3728-3738. Gryglewicz, K., Childs, K.K., & Soderstrom, M.F. (2018). An evaluation of youth mental health first aid training in school settings. School Mental Health, 10, 48-60. Guba, E.G., & Lincoln, Y. (1989). Fourth generation evaluation. Newberry Park, CA: Sage. Guegan, E.W., Dorling, H., Ollerhead, L., & Westmore, M. (2016). Mapping public health research across the national institute for health research 2006-2013. BMC Public Health, 16, 911-922. Guo, S., Kataoka, S.H., Bear, L., & Lau, A.S. (2014). Differences in school-based referrals for mental health care: Understanding racial/ethnic disparities between Asian American and Latino youth. School Mental Health, 6, 27-39. Haggerty, D., Carlson, J.S., McNall, M., Lee, K., & Williams, S. (2018). Exploring youth mental health first aider training outcomes by workforce affiliation: A survey of project AWARE participants. School Mental Health, advanced online publication. Hall, M.T., &Wurf, G. (2018). Strengthening school-family collaboration: An evaluation of the family referral service in four Australian schools. Australian Journal of Education, 62(1), 521-35 Hamilton, J., Begley, C., & Culler, R. (2014). Evaluation of partner collaboration to improve community-based mental health services for low-income minority children and their families. Evaluation and Program Planning, 45, 50-60. Han, S.S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33(6), 665-679. Han, X., Luo, Q., & Ku, L. (2017). Medicaid expansion and grant funding increases helped improve community health center capacity. Health Affairs, 36(1), 49-56. Hane-Schlagel, R., Brookman-Frazee, L., Fettes, D.L., Baker-Ericzen, M., & Garland, A.F. (2012). Therapist focus on parent involvement in community-based youth psychotherapy. Journal of Child and Family Studies, 21, 646-656. 88 Hoagwood, K. (2003). The policy context for child and adolescent mental health services: Implications for systems reform and basic science development. Annals of the New York Academy of Science, 1008, 140–148. Hodges, S., Ferreira, K., & Israel, N. (2012). “If we’re going to change things, it has to be systemic:” Systems change in children’s mental health. American Journal of Community Psychology, 49, 526-537. Hodges, S., Hernandez, M., Pinto, A., & Uzzell, C. (2007). The use of qualitative methods in systems of care research. The Journal of Behavioral Health and Services Research, 34(4), 361-368. Holen-Rabbersvik, E., Thygesen, E., Eikebrokk, T.R., Fensli, R.W., & Slettebo, A. (2018). Barriers to exchanging healthcare information in inter-municipal healthcare services: A qualitative case study. BMC Medical Informatics and Decision Making, 18(92), 1-14. Holtes, M., Bannink, R., Joosten-van Zwanenburg, van As, E., Raat, Broeren, S. (2015). Associations of truancy, perceived school performance, and mental health with alcohol consumption among adolescents. Journal of School Health, 85(12), 852-860. Hoover Stephan, S., Weist, M., Kataoka, S., Adelsheim, S., & Mills, C. (2007). Transformation of children’s mental health services: The role of school mental health. Psychiatric Services, 58(10), 1330-1338. Hoover, S.A. (2018). When we know better, we don’t always do better: Facilitating the research to practice and policy gap in school mental health. School Mental Health, 10, 190-198. Husky, M.M., Sheridan, M., McGuire, L., & Olfson, M (2011). Mental health screening and follow-up care in public schools. Journal of the American Academy of Child & Adolescent Psychiatry, 50(9), 881-891. Jackson County ISD: An Educational Service Agency. (2018). Retrieved from: https://www.jcisd.org Kazdin, A. E. (2008). Evidence-based treatments and delivery of psychological services: Shifting our emphases to increase impact. Psychological Services, 5(3), 201–215. Kern., L., Mathur, S.R., Albrecht, S.F., Poland, S., Rozalski, M., & Skiba, R.J. (2017). The need for school-based mental health services and recommendations for implementation. School Mental Health, 9, 205-217. Khanlou, N., Haque, N., Sheehan, S., & Jones, G. (2015). “It is an issue of not knowing where to go”: Service providers’ perspectives on challenges in accessing social support and services by immigrant mothers of children with disabilities. Journal of Immigrant Minority Health, 17, 1840-1847. 89 Kober, N., & Rentner, D. S. (2011). More to do, but less capacity to do it: States’ progress in implementing the Recovery Act education reforms. Washington, DC: Center on Education Policy. Koch, T. (1994). Establishing rigor in qualitative research: The decision trail. Journal of Advanced Nursing, 19, 976-986. Koelsch, L.E. (2013). Reconceptualizing the member check interview. International Journal of Qualitative Methods, 12, 168-179. Langley, A.K., Nadeem, E., Kataoka, S.H., Stein, B.D., & Jaycox, L.H. (2010). Evidence-based mental health programs in schools: Barriers and facilitators to successful implementation. School Mental Health, 2, 205-113. Larson, A., Chapman, S., Spetz, J., & Brindis, C.D. (2017). Chronic childhood trauma, mental health, academic achievement, and school-based health center mental health services. Journal of School Health, 87(9), 675-686. Lawson, D.M., & Brossart, D.F. (2003) Link Among Therapist and Parent Relationship, Working Alliance, and Therapy Outcome. Psychotherapy Research, 13(3), 383-394. Leachman, M., Albares, N., Masterson, K., & Wallace M. (2016). Most states have cut school funding, and some continue cutting. Washington, DC: Center Budget Policy Priorities. Retrieved from: http://www.cbpp.org/sites/default/files/atoms/files/12-10-15sfp.pdf Lee, R., & Gortmaker, S. (2012). Health dissemination and implementation within schools. In R.C. Brownson, G.A Colditz, & E.K. Proctor (Eds.), Dissemination and Implementation Research in Health: Translating Science into Practice (pp. 419-436). New York, New York: Oxford University Press. Lincoln, Y., & Guba, E.G. (1985). Naturalist inquiry. Newbury Park, CA: Sage. Livet, M., Yannayon, M., Sheppard, K., Kocher, K., Upright, J., & McMillan, J. (2018). Exploring provider use of a digital implementation support system for school mental health: A pilot study. Administration and Policy in Mental Health, 45, 362-380. Locke, J., Beidas, R.S., Marcus, S., Stahmer, A., Aarons, G.A., Lyon, A.R., …Mandell, D.S. (2016). A mixed methods study of individual and organizational factors that affect implementation of interventions for children with autism in public schools. Implementation Science, 11, 135-144. Lyon, A.R., Ludwig, K., Romano, E., Leonard, S., Stoep, A.V., McCauley, E. (2013). “If it’s worth my time, I will make the time”: School-based providers’ decision-making about participating in an evidence-based psychotherapy consultation program. Administration and Policy in Mental Health, 40, 467-481. 90 Mackie, P.F.E., Zammitt, K., & Alvarez M. (2016). Practicing Rural Social Work. Chicago, IL: Lyceum Books. MacQueen, K. M., McLellan, E., Kay, K., & Milstein, B. (1998). Codebook development for team-based qualitative analysis. Cultural Anthropology Methods Journal, 10(2), 31-36. Marshall, T., Rapp, C.A. Becker, D.R., & Bond, G.R. (2008). Key factors for implementing supported employment. Psychiatric Services, 59, 886-892. Maschi, T., Schwalbe, C.S., Morgen, K., Gibson, S., & Violette, N.M. (2009). Exploring the influence of gender on adolescents’ service needs and service pathways. Children and Youth Services Review, 31, 257-264. McCarthy, M., Schellinger, J., Smith, R.R., Behimer, G., Hargraves, D., & Scherra, K. (2017). Comparison of service effectiveness for youth with depressed versus non-depressed caregivers. Research on Social Work Practice, 27(4), 413-422. McIntosh, K., Kelm, J.L., & Delabra, A.C. (2016). In search of how principals change: A qualitative study of events that help and hinder school administrator support for school- wide PBIS. Journal of Positive Behavior Interventions, 18(2), 100-110. Meier, M.H., Hill, M.L., Small, P.J., & Luthar, S.S. (2015). Associations of adolescent cannabis use with academic performance and mental health: A longitudinal study of upper middle class youth. Drug and Alcohol Dependence, 156(1), 207-212. Merikangas, K.R. He, J., Burstein, M., Swanson, S.A., Avenevoli, S., Cui, L., Benjet, C., Georgiades, K., & Swendsen, J. (2010). Lifetime prevalence of mental disorders in US adolescents: Results from the national comorbidity study-adolescent supplement (NCS- A). Journal of American Academy of Child and Adolescent Psychiatry, 49(10), 980-989. Michigan Department of Education. (2018). Michigan Profile for Healthy Youth: 2016-2017 [data set]. Retrieved from: https://www.michigan.gov/mde/0,4615,7-140- 74638_74639_29233_44681---,00.html. Miles, M.B., & Huberman, A.M., (1994). Qualitative data analysis: An expanded sourcebook. (2nd ed). Thousand Oaks, CA: Sage. Moskos, M.A., Olson, L., Halbern, S.R., Gray, D. (2007). Utah youth suicide study: Barriers to mental health treatment for adolescents. Suicide and Life-Threatening Behavior, 37(2), 179-186. Mundy, L.K., Canterford, L., Tucker, D., Bayer, J., Romaniuk, H., Sawyer, S., Lietz, P., Redmond, G., Proimos, J., Allen, N., & Patton, G. (2017). Academic performance in primary school children with common emotional and behavioral problems. Journal of School Health, 87(8), 593-601. 91 Nadeem, E., Santiago, C.D., Katoaka, S.H, Chang, V.Y., Stein, B.D. (2016). School personnel experiences in notifying parents about their child’s risk for suicide: Lessons learned. Journal of School Health, 86(1), 3-10. National Institute of Mental Health. (2018). Division of Services and Intervention Research: An Overview. Retrieved from: https://www.nimh.nih.gov/about/organization/dsir/index.shtml Neufeld, S.A.S., Jones, P.B., & Goodyer, I.M. (2017). Child and adolescent mental health services: Longitudinal data sheds light on current policy for psychological interventions in the community. Journal of Public Mental Health, 16(3), 96-99. Now is the Time Technical Assistance Center. (2018). Project Advancing Wellness and Resilience Education. Substance Abuse and Mental Health Services Administration. Retrieved from https://www.samhsa.gov/nitt-ta/project-aware-grant-information Nowell, L.S., Norris, J.M., White, D.E., & Moules, N.J. (2017). Thematic analysis: Striving to meet the trustworthiness criteria. International Journal of Qualitative Methods, 16, 1-13. Palinkas, L.A., Fuentes, D., Finno, M., Garcia, A.R., Holloway, I.W., Chamberlain, P. (2014). Inter-organizational collaboration in implementation of evidence-based practices among public agencies serving abused and neglected youth. Administration and Policy in Mental Health, 41, 74-85. Palinkas, L.A., Horwitz, S.M., Green, C.A., Wisdom, J.P., Duan, N., & Hoagwood, K. (2015). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and Policy in Mental Health, 42, 533-544. Patton, M.Q. (2015). Qualitative research & evaluation methods (4th ed.). SAGE Publications, Inc.: Thousand Oaks, CA. Phillippo, K.L., & Kelly, M.S. (2014). On the fault line: A qualitative exploration of high school teachers’ involvement with student mental health issues. School Mental Health, 6(3), 184-200. Proctor, E. K., McMillan, C., Haywood, S., & Dore, P. (2008). Advancing mental health research: Washington University’s Center for Mental Health Services Research. Social Work Research, 32(4), 249-259. Proctor, E., Silmore, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A. et al., (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38, 65- 76. Provan, K. G., & Kenis, P. (2008). Modes of network governance: Structure, management, and effective ness. Journal of Public Administration Research and Theory, 18, 229–252. 92 Provan, K. G., & Milward, H. B. (1995). A preliminary theory of interorganizational network effectiveness: A comparative study of four community mental health systems. Administrative Science Quarterly, 40, 1–33. Reardon, T. Harvey, K., Young, B., O’Brien, D., & Creswell, C. (2017). Barriers and facilitators to parents seeking and accessing professional support for anxiety disorders in children: Qualitative interview study. European Journal of Child and Adolescent Psychiatry, 27(8), 1023-1031. Reardon, T., Harvey, K., Bananowska, M., O’Brien, D., Smith, L., & Creswell, C. (2017). What do parents perceive are the barriers and facilitators to accessing psychological treatment for mental health problems in children and adolescents? A systematic review of qualitative and quantitative studies. European Journal of Child and Adolescent Psychiatry, 26, 623-647. Regan, J., Lau, A.S., Barnett, M., Stadnick, N., Hamilton, A., Pesanti, K., Bando, L., & Brookman-Frazee, L. (2017). Agency responses to a system-driven implementation of multiple evidence-based practices in children’s mental health services. BMC Health Services Research, 17, 671-684. Remigio-Baker, R.A., Hayes, D.K., Reyes-Salvail, F. (2014). Adverse childhood events and current depressive symptoms among women in Hawaii: 2010 BRFSS, Hawaii. Journal of Maternal and Child Health, 18(10), 2300-2308. Richards, L. (2005). Handling qualitative data: A practical guide. Thousand Oaks, CA: Sage Publications. Ritchie, L.M., & Straus, S.E. (2019). Assessing organizational readiness for change: Comment on “Development and content validation of a transcultural instrument to assess organizational readiness for knowledge translation in healthcare organizations: The OR4KT”. International Journal of Health Policy and Management, 8(1), 55-57. Romer, N., Green, A.L., Cox, K.E. (2018). Educator perceptions of preparedness and professional development for implementation of evidence-based practices within multi- tiered systems of support. School Mental Health, 10, 122-133. Rones, M., & Hoagwood, K. (2000). School-based mental health services: A research review. Clinical Child and Family Psychology Review, 3(4), 223-241. Rosenblatt, A. (2010). If it walks like a duck and quacks like a duck then it must be a rabbit? Programs, systems and a cumulative science of children’s mental health services. Evaluation and Program Planning, 33, 14-17. Russell, J.L., Meredith, J., Childs, J., Stein, M.K., & Prine, D.W. (2015). Designing inter- organizational networks to implement education reform: An analysis of state race to the top applications. Educational Evaluation and Policy Analysis, 37(1), 92-112. 93 Ryst, E., Rock, S.L., Albers, E.C., Everheart, C.A. (2016). Implementation of Project AWARE (advancing wellness and resilience education) in three rural Nevada school districts to increase mental health awareness, early identification of mental health issues, and intervention with school-aged youth. Journal of the American Academy of Child & Adolescent Psychiatry, 55(10S), p. S180. Sarno Owens, J., Lyon, A.R., Brandt, N.E., Warner, C.M., Nadeem, E., Spiel, C., & Wagner, M. (2014). Implementation science in school mental health: Key constructs in a developing research agenda. School Mental Health, 6, 99-111. Sayal, K. (2006). Annotation: Pathways to care for children with mental health problems. Journal of Child Psychology and Psychiatry, 47, 649-659. Shernoff, E.S., Bearman, S.K., Kratchowill, T.R. (2017). Training the next generation of school psychologists to deliver evidence-based mental health practices: Current challenges and future directions. School Psychology Review, 46(2), 219-232. Shoenwald, S. K., & Hoagwood, K. (2001). Effectiveness, transportability, and dissemination of interventions: What matters when? Psychiatric Services, 52(9), 1190–1197. Sprigg, S. M., Wolgin, F., Chubinski, J., & Keller, K. (2017). School-based health centers: A funder's view of effective grant making. Health Affairs, 36(4), 768-772. Stake, R. (1995). The art of case study research. Thousand Oaks, CA: Sage. Stein, B.D., Kataoka, S.H., Hamilton, A.B., Schultz, D., Ryan, G., Vona, P., & Wong, M. (2010). School personnel perspectives on their school’s implementation of a school-based suicide prevention program. The Journal of Behavioral Health and Services Research, 37(3), 338-349. Substance Abuse and Mental Health Services Administration. (2015). School mental health referral pathways (SMHRP) toolkit. Now is the Time Technical Assistance Center. Tapia, M.I., Ocasio, M.A., Estrada, Y., Pantin, H., & Prado, G. (2017). Engaging school mental health professionals to deliver evidence-based interventions to Hispanic families. Health Promotion in Latino Families, 18(4), 526-533. Thomas, K.C., Ellis, A.R., Holzer, C.E., Morrissey, J.P. (2009). County-level estimates of mental health professional shortage in the United States. Psychiatric Services, 60, 1323-1328. Thomas, K.C., Ellis, A.R., Konrad, T.R., Holzer , C.E., & Morrissey, J.P.(2009). County-level estimates of professional shortage in the US. Psychiatric Services, 60(10), 1323-1325. Tobin, G.A., & Begley, C.M. (2004). Methodological rigor within a qualitative framework. Journal of Advanced Nursing, 48, 388-396. 94 U.S. Department of Education. (2018). 2018 Determination Letters on State Implementation of IDEA [fact sheet]. Retrieved from https://sites.ed.gov/idea/files/ideafactsheet- determinations-2018.pdf. United States Census Bureau. (2018). American Fact Finder. Retrieved from: https://factfinder.census.gov/faces/nav/jsf/pages/community_facts.xhtml Walker E.R., Berry, F.W. III, Citron, R., Fitzgerald, J., Rapaport, M.H., Stephens, B., & Druss, B.G. (2015). Psychiatric workforce needs and recommendations for the community mental health system: A state needs assessment. Psychiatric Services, 66(2), 115-117. Walker, J.S., Koroloff, N.M., & Bruns, E.J. (2010). Defining “necessary” services and supports: Why systems of care must take direction from service level processes. Evaluation and Program Planning, 33, 49-52. Weist, M.D., & Paternite, C.E. (2006). Building an interconnected policy-training-practice- research agenda to advance school mental health. Education and Treatment of Children, 29(2), 173-196. Withers, M.C., Reynolds, J.E., Reed, K, & Holtrop, K. (2016). Dissemination and implementation research in marriage and family therapy: An introduction and call to the field. Journal of Marital and Family Therapy, 43(2), 183-197. Wolfe, I., Lemer, C., & Cass, H. (2016). Integrated care: A solution for improving children’s health? Archives of Disease in Childhood, 101(11), 992-997. Yang, T., & Maxwell, T.A. (2011). Information-sharing in public organizations: A literature review of interpersonal, intra-organizational and inter-organizational success factors. Government Information Quarterly, 28, 164-175. 95