CORE THERAPIST SKILLS SUPPORTING IMPLEMENTATION OF EVIDENCE-BASED PRACTICES WITH SERIOUS EMOTIONALLY DISTURBED CHILDREN IN COMMUNITY MENTAL HEALTH SETTINGS: A MODIFIED MIXED METHODS DELPHI STUDY By Debra Lynn Miller A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Human Development and Family Studies – Doctor of Philosophy 2021 ABSTRACT CORE THERAPIST SKILLS SUPPORTING IMPLEMENTATION OF EVIDENCE-BASED PRACTICES WITH SERIOUS EMOTIONALLY DISTURBED CHILDREN IN COMMUNITY MENTAL HEALTH SETTINGS: A MODIFIED MIXED METHODS DELPHI STUDY By Debra Lynn Miller Although evidence-based practices (EBPs) exist to support emotional and behavioral health for children and their families, their adoption in community settings remains startlingly low. While therapist training is a promising method to promote the successful adoption of EBPs, there is often a lack of agreement on the foundational skills therapists need to be successful in learning and adopting specific EBP approaches. Unfortunately, this absence of agreement hinders the advancement of training interventions that can better support EBP implementations in community health settings and provide families with access to research-based care. To address this concern, this study produced consensus on the core therapist skills and corresponding training methods that support the adoption of family focused EBPs in community health settings. Using a mixed method modified Delphi study process, data were collected from a group of 51 panel experts representing 11 family focused EBP models commonly implemented in community mental health settings to treat SED symptoms in youth. This expert panel reached consensus on 175 skills considered most important for successful EBP adoption. Follow up semi- structured interviews with participants yielded agreement on the training methods most promising to support therapists in acquiring these foundational skills. Research findings establish a specific skillset not presently found in the literature that crosscuts multiple family focused EBPs. These findings have significant implications to inform future training interventions supportive of EBP adoption in community health settings. Results are applicable to community health providers and leaders making strategic decisions about investment in EBP training and workforce development. Moreover, a call to clinical training programs is established suggesting this core therapist skillset be used to guide future curriculum development and assessment of strengths and capacity of therapists in training. Copyright by DEBRA LYNN MILLER 2021 This work is dedicated with love to my husband, Fredrick Miller, and to my children Allison and Thomas. May this always be an example of what we achieved together. v ACKNOWLEDGMENTS In the hardest moments of this journey to pursue my PhD, I never failed to be held and given what I needed to continue. My first and most important acknowledgement is to the divine love embodied in my family, friends, colleagues, and mentors. Your light gave me vision, reassurance, and the determination to continue to do this privileged work. I am forever grateful for these connections felt so deeply in my heart throughout this process. It has been a tremendous honor to work with my co-advisors. I first want to thank Dr. Adrian Blow for the invitation to study with you. If not for your early encouragement, I may not be writing this today. Thank you for seeing something in me worth cultivating, for your deep understanding that academia and community practice are necessary partners, and for providing the encouragement that my gifts would be valued in an academic space. I would also like to thank Dr. Kendal Holtrop for choosing me to work with you and coming onboard officially as my advisor. I am incredibly grateful for your gentle and consistent guidance and for the research opportunities you trusted me with. Thank you for always shining a light on me - reminding me of my worth and patiently guiding my professional experiences into a meaningful program of research. To both of you as a team, your thoughtful guidance has not only challenged but sustained me. You ushered me through some hard moments and reminded me to celebrate. You are amazing and I am forever grateful for your mentoring. It has been an incredible honor to work with my doctoral committee as a whole and I would like to acknowledge both Dr. Heather McCauley and Dr. Miles McNall. Dr. McCauley, I learned so much about methodology and advocacy from you, particularly how to raise marginalized voices and work with communities toward important research aims. I thank you for vi believing in the kind of program of research I wanted to cultivate, even before I fully understood it myself, and for reminding me that the dissertation step is only a beginning to a lifetime of important community work. Dr. McNall, I thank you for your openness to meeting with me early on in my program and for being a constant support throughout the process. I am so grateful for the validation you offered in my pursuit of an academic community partnership, your attention to details, and for your humor. I may have learned a thing or two about reframing from you. Thank you for all the ways you both challenged me to grow. This study would not have been possible without the support of my community partner, Community Mental Health for Central Michigan. I would like to especially acknowledge Julie Bayardo for her constant support, vision, and cheerleading. To have your support and reflection throughout this process was not only grounding but so meaningful. Thank you for always showing up to talk and dream. With immense gratitude, I acknowledge and thank the participants in this study. Truly, the credit of this work is shared with you all. Your investment in this study and your willingness to share your experiences in implementation offered a richness to these findings I never dreamed possible. Thank you for giving yourself and your time when you likely did not have much to spare. I remain in awe of the work you do. None of this would have been possible without the love and support of my family. Fred, I fail to describe with words the kind of love and support you gave me throughout this journey. Thank you for always finding ways to hold our promise that my going back to school would not impact our family in any way but good. You gave more than you got these past four years and your loyalty, determination, and vision bring me to tears. Thank you for always reminding me what is important. Allie and Tommy, I simply thank you both for being yourselves and all the vii thoughtful ways you found to help along the way. I got to experience this program and dissertation while watching you grow, and I am so thankful for always having your perspective to ground me as I did my work. I hope this work continues to show you that when you put your heart in what you love, all things are possible. A special thank you to my parents, Alice (Donn), and Roy (Sally) for the unequivocal belief that I could do this. You have held pride for my work and been consistent examples of hard work and determination through adversity. Thank you for all the ways you have made me who I am. And to my brother Roy, thank you for always believing in me. I always hold a special place in my heart for the stability you gave me early on in my educational career that undoubtedly helped me get to this point. And last, I thank my parents in law, Miyoko and Tom. If you were here, I know you would be so happy to see this accomplishment. Your values of hard work and educational attainment are deeply embedded in our family culture. I also want to thank my friends and colleagues on this journey with me. Sara, some of my best getaways during this time are credited to you. Thank you for always checking in on me and being my travel partner. Rachel, your friendship and unconditional support kept me going in this process. Seriously, for every time you reminded me that I did know my own heart or encouraged me to be vulnerable, I thank you and hope you know how incredibly special you are. Olga, Silvia, and Kathy, your mentorship and guidance got me to this space. For all you did to nurture my love for systemic work and to be brave, I am so grateful. Finally, I acknowledge and give heartfelt thanks to the Department of Human Development and Family Studies at Michigan State University for offering such a safe place to grow and learn. You gave me an academic home and an extended family of fellow students and faculty. I am so grateful to work with you all. viii TABLE OF CONTENTS LIST OF TABLES ................................................................................................................... xii LIST OF FIGURES ................................................................................................................. xiii CHAPTER 1: INTRODUCTION ........................................................................................... 1 Background of the Problem .......................................................................................... 1 An EBP Implementation Research Model............................................................................. 3 Conceptual Framework: Dissemination and Implementation ....................................... 4 Adapted Systems-Contextual Approach ....................................................................... 5 Theoretical Orientation: Systems Theory ..................................................................... 6 Context: Academic-Community Partner Relationship ................................................. 7 Negative Effects of a Research to Practice Gap ................................................................... 8 Increasing Needs of SED Children and Their Families ................................................ 9 Gaps in Therapist Training and Development .............................................................. 10 Purpose of the Study and Aims .............................................................................................. 12 Method: Modified Delphi Process ................................................................................ 13 Research Questions ....................................................................................................... 14 Summary .................................................................................................................................. 15 CHAPTER 2: LITERATURE REVIEW............................................................................... 16 Common Factors Paradigm .................................................................................................... 17 Common Factors Defined ............................................................................................. 17 A Common Factors Conceptualization of Core Therapists Skills ................................ 17 Therapist Competency ............................................................................................................ 19 General competency ...................................................................................................... 19 Therapist Skill Development ........................................................................................ 20 Potential Areas for Therapist Skillset Development ..................................................... 21 Research on Learning Methods ..................................................................................... 30 Summary .................................................................................................................................. 31 CHAPTER 3: RESEARCH METHODOLOGY .................................................................. 32 Academic-Community Partner Relationship ................................................................ 32 Researcher Positionality ................................................................................................ 35 Purpose of the Study and Aims ..................................................................................... 36 Research Questions ....................................................................................................... 36 Research Design and Methodology .............................................................................. 37 Selection of Participants ............................................................................................... 38 Procedures and Data Collection: Delphi Process ................................................................. 43 Delphi Round 1 ............................................................................................................. 43 Delphi Round 2 ............................................................................................................. 50 Delphi Round 3 ............................................................................................................. 53 ix Summary .................................................................................................................................. 58 CHAPTER 4: ANALYSIS AND RESULTS ......................................................................... 60 Research Questions ....................................................................................................... 60 Data Collection ............................................................................................................. 61 Results ...................................................................................................................................... 61 Description of Delphi Expert Panel .............................................................................. 61 Description of Participant Expertise ............................................................................. 63 Recruitment and Retainment ......................................................................................... 65 Reasons for Participating .............................................................................................. 66 Results Round 1: Identifying a Therapist Skillset ....................................................... 67 Results Round 2: Consensus and Non-Consensus Statements ........................................... 73 Consensus Statement .................................................................................................... 73 Non-consensus Items .................................................................................................... 74 Summary of Round 2 Results ........................................................................................ 84 Results Round 3: Identified Training Methods .................................................................... 84 Preparing the Training Environment ............................................................................ 85 Specific Training Methods for Developing Skillsets .................................................... 89 Additional EBP Training Considerations ..................................................................... 98 Summary of Results ................................................................................................................. 100 CHAPTER 5: DISCUSSION AND IMPLICATIONS ......................................................... 102 Foundational Skillset for EBP Practice ................................................................................. 103 Assessment Skills........................................................................................................... 104 Family Engagement Skills ............................................................................................. 106 Intervention Skills .......................................................................................................... 108 Reflexivity...................................................................................................................... 112 Adaptability.................................................................................................................... 113 Trauma Informed Care................................................................................................... 114 Overall Skillset Discussion and Summary..................................................................... 115 Training Methods to Support Learning of Core Skillset ..................................................... 117 Preparing the Training Environment ............................................................................. 117 Specific Training Methods for Developing Skillsets ..................................................... 119 Additional Considerations Related to EBP Training ..................................................... 123 Limitations of the Study .......................................................................................................... 125 Study Strengths ........................................................................................................................ 126 Study Implications ................................................................................................................... 127 Implications for Research .............................................................................................. 127 Organizational and Training Program Implications ...................................................... 130 Clinical Practice, Supervision, and Training ................................................................. 132 Implications for Future Delphi Studies .......................................................................... 134 Final Summary/Conclusions ................................................................................................... 135 x APPENDICES .......................................................................................................................... 137 Appendix A: Family Focused EBP Models Utilized for Participant Recruitment ........... 138 Appendix B: Sample Recruitment Email ............................................................................. 141 Appendix C: Consent .............................................................................................................. 143 Appendix D: Delphi Process Flow with Community-Engaged Scholarship....................... 145 Appendix E: Round 1 Open-ended Questions ..................................................................... 146 Appendix F: Demographic Information (Qualtrics) ............................................................ 147 Appendix G: Directions and Sample Survey ........................................................................ 148 Appendix H: Full Interview Guide Round 3 ......................................................................... 151 Appendix I: Round 3 Informed Consent .............................................................................. 153 Appendix J: Consensus Categories and Sub-Headings (Therapist Skillset) ...................... 154 Appendix K: Core Therapist Skillset Combined All Categories ......................................... 155 REFERENCES ......................................................................................................................... 163 xi LIST OF TABLES Table 3.1 Data Analysis Steps and Description – Round 1 ....................................................... 45 Table 3.2 Data Analysis Steps and Description – Round 3 ....................................................... 55 Table 4.1 Descriptive Statistics for Participant Demographics ................................................. 62 Table 4.2 Participant Years of Clinical Experience .................................................................. 63 Table 4.3 Participant Age Categories ....................................................................................... 64 Table 4.4 Participant Representation by Evidence Based Practice ........................................... 64 Table 4.5 Area of the United States Represented by Participants ............................................ 65 Table 4.6 Reasons for Participating in Delphi Study ................................................................ 66 Table 4.7 Therapist Skillset Categories Identified for Round 2 Survey ................................... 71 Table 4.8 All Round 1 Statements and Median and IQR Ratings by Participants ................... 74 Table 4.9 Consensus Percentages by Main Category ................................................................ 84 Table 4.10 Suggested Training Methods for Each Skill Category ............................................ 96 Table A.1 Family Focused EBP Models Utilized for Participant Recruitment ......................... 138 Table J.1 Consensus Categories and Sub-Headings (Therapist Skillset) .................................. 154 Table K.1 Summarized Therapist Skillset ................................................................................. 155 xii LIST OF FIGURES Figure 1.1 Understanding Therapist Skill Development in System Context ............................ 6 Figure 3.1 Community Engaged Partnership Research Abacus ................................................ 34 Figure 4.1 Flow Chart for Round 1 Results ............................................................................... 68 Figure D.1 Delphi Process Flow with Community-Engaged Scholarship ................................ 145 xiii CHAPTER 1: INTRODUCTION “Solutions have a voice - the art is knowing how to listen” - Gino Norris This dissertation study encompasses the coming together of many voices aimed toward the goal of improving access to evidence-based practices for youth with serious emotional disturbances being treated in community health settings. Broadly, this study is centered on raising voices – those of community partners, academic researchers, practitioners, and evidence- based practice experts - on how we can find more promising solutions to bring research informed treatment to the families in communities that would most benefit from them. The logic for this study starts with hearing our families and knowing that youth with serious emotional and behavioral symptoms and their families suffer at alarming rates in many life domains (Centers for Disease Control, 2018). Advancing research toward ameliorating these symptoms is necessary. The goals of this study echo strategies recommended from the Substance Abuse and Mental Health Services Administration (2020) to improve services to youth and families experiencing SED symptoms by prioritizing practitioner training and support. This study raises the voice of a community partner who corroborated the need for research that can inform how to support therapist development and guide future investments in EBPs. And finally, these voices connect to my own growth as a researcher who can design a study that amplifies and organizes what EBP experts already know about therapist skillsets necessary to treat SED youth in community health settings. Chapter one continues with introducing the problem under inquiry and establishes the reasons for this specific study and methodology. Background of the Problem Evidence-based practices (EBPs) are methodical and replicable therapeutic interventions tested and found to be effective when studied in clinical trials (Bearman et al., 2013; Stratton et 1 al., 2015). Defined as the “integration of best available research with clinical expertise in the context of patient characteristics, culture, and preferences” (American Psychological Association [APA], 2008, p. 5), EBPs are important therapeutic approaches to treating family systems effectively (Sexton et al., 2011). EBPs are commonly enlisted by community health agencies to address the treatment of serious emotional disturbances (SED) in youth (Hodges, Xue, & Wotring, 2004). SED is defined by Substance Abuse and Mental Health Services Administration (SAMHSA) as the occurrence of diagnosable mental, behavioral, or emotional disorders that substantially limit or impact a child’s role in one or more life domains, e.g., school, home, or community (SAMHSA, 1993). The processes of integrating EBPs in community practice settings, referred to hereafter as implementation, is a complex and challenging process at best (Bearman et al., 2013; National Institutes of Health [NIH], 2018; Proctor et al., 2011). This is because implementations within community agency settings are often not successfully adopted or maintained in routine practice (Bond et al., 2014; Karlin & Cross, 2014). Unfortunately, failed EBP implementations leave families and children without access to the care demonstrated to be effective in addressing their specific mental health conditions (Kazak et al., 2010; McHugo et al., 2007; President’s New Freedom Commission, 2003). Successful EBP implementations are necessary to bridge research and community mental health care practice that can ultimately benefit children and their families (APA, 2008; Barnett, Jankowski, & Trepman, 2019). When EBP implementation efforts are successful they often yield significantly positive outcomes in children and families such as reductions in serious mental health symptoms, behavioral improvements, and improved quality of life (Hodges et al., 2004; McHugo et al., 2007). The benefit of having successful EBP implementations in community mental health 2 programs cannot be overstated as they are a path to facilitating positive outcomes for children and their families (APA, 2008; NIH, 2018, SAMHSA, 2020). Efforts to systematically understand the processes that contribute to successful EBP implementations are underscored in the literature as essential to improving the sustainability of EBPs in community settings (Beidas et al., 2014; Stirman et al., 2012). Yet, these processes remain substantially understudied (Proctor et al., 2011). Said another way, while billions of tax dollars are spent studying EBP outcomes, only a small percentage of those dollars are invested in implementation research that could lead to sustaining EBPs in community settings (Aarons, Hurlburt, & Horwitz, 2011; NIH, 2018). Enhanced implementation research efforts are necessary to pinpoint how adoption and sustainability of EBPs can improve (APA, 2006, APA, 2008; Bond et al., 2014; Sexton et al., 2011). A more nuanced understanding of the skills that can best support therapists in learning EBPs can play a significant role in closing this research to practice gap (APA, 2008; NIH, 2018). Similarly, effective therapist training is considered a key strategy for EBP utilization in the community (Frank, Becker-Haimes, & Kendall, 2020). In acknowledgement of these aims, SAMHSA names both SED treatment and strengthening practitioner training as two of their five key priority areas. Further, they name supporting the adoption of EBPs as one of their five core principles (2020). An EBP Implementation Research Model This need for this dissertation study is driven by significant gaps related to the adoption of family focused EBPs in community health settings. Within these gaps, the need to identify a foundational therapist skillset and training methods that can support EBP adoption and treatment for youth with SED symptoms is substantial. The research model for this study is situated in a 3 dissemination and implementation framework and informed by engagement with a community research partner. Conceptual Framework: Dissemination and Implementation Implementation research models are positioned within a larger framework of dissemination and implementation (D&I) science (Mendel, Meredith, Schoenbaum, Sherbourne, & Wells, 2008; Tabak, Khoong, Chambers, & Brownson, 2012) and emphasize the need to understand a broad array of factors that impact EBP adoption, EBP sustainability, and client outcomes within community settings (Aarons, Wells, Zagursky, Fettes, & Palinkas, 2009; NIH, 2018). These factors include individuals and processes within the implementing organization (e.g., agency culture and structure), therapist variables impacting EBP adoption (e.g., adherence to EBP, training), and the impact of the treatment on children and families served by the organization (Aarons et al., 2011; Sexton et al., 2011). Multi-factor implementation models are useful in examining the complexity of how EBPs are maintained in community practice (Aarons et al., 2011; Fixsen et al., 2009; Mendel et al., 2008). However, these multi-factor models also make the implementation research process a more complex undertaking (Aarons et al., 2009; Proctor et al., 2011; Sexton et al., 2011), and subsequently run the risk of failing to specifically define some key factors that impact implementation efforts (Aarons et al., 2011; Fixsen et al., 2009). This study is focused most prominently on identifying a foundational therapist skillset that can impact EBP adoption – one piece of this larger implementation research framework. Nonetheless, the model for this study also recognizes the interconnectedness of community mental health agencies supporting therapist skill development and the methods they use to ultimately target improved EBP adoption in their communities (Bearman et al., 2013; Beidas & Kendall, 2010). 4 Adapted Systems-Contextual Approach A systems-contextual approach to implementation research provides a structure to study therapist skills and training methods while considering the interconnection of the organizational context (Beidas & Kendall, 2010). The systems-contextual approach is wholistic and supports the idea that EBP adoption can be successful “…when therapists are trained appropriately and when their context supports that behavior change” (Beidas & Kendall, 2010; p. 2; Sanders & Turner, 2005). Adapting this approach, an underlying therapist skillset supporting EBP adoption is envisioned when considering the bi-directional relationship between the training methods (i.e., observations, feedback, competency building) and the therapist or EBP adopter (i.e., their skills and ability to adapt). Both are impacted by the implementing organization’s abilities to strategically plan, provide quality training opportunities, and in the end do what is needed to support therapist behavior change (i.e., adoption of EBPs). Organizational factors such as agency support and quality of training efforts are directly positioned to improve therapists’ adoption of EBPs (Beidas & Kendall, 2010; Sanders & Turner, 2005). As seen in Figure 1.1, core therapist skills can be understood within the relationship of the therapist (adopter), training methods, and the implementing organization. 5 Theoretical Orientation: Systems Theory A systems-contextual approach is supported in a systems theoretical framework (Bateson, 1979; Von Bertalanffy, 1968), described as the orientation of an individual or community in the context of a larger system (in Bowers & Bowers, 2017). From this lens, the interaction of the therapist and the EBP organization are interdependent in implementation efforts (Aarons et al., 2011; Bateson, 1979; Beidas & Kendall, 2010, 2014; Gutman, 1991) and bi-directional processes exist between the therapist and their training resources within the organization (Sanders & Turner, 2005). The systems-contextual perspective clearly articulates the importance of training therapists to basic skill proficiency prior to implementing EBPs by examining both the relationship between agency training resources (e.g., readiness for implementation) and therapist competencies (Beidas & Kendall, 2010). Studying essential core therapist skills from this lens offers promise in developing targeted interventions that can support greater adoption and 6 sustainability of EBPs in community practice settings (Nelson, Stanley, Funderburk & Bard, 2012; Roundfield & Lang, 2017). Context: Academic-Community Partner Relationship The need for increased implementation research and better EBP adoption in community settings is underscored in the current research literature (Karlin & Cross, 2014; NIH, 2018). Graaf and Snowden (2021) punctuate this point by urging scholars to develop interventions that can “successfully translate and sustain in real-world public health settings” (p. 196). At the same time, community service providers implementing EBPs acknowledge experiencing low adoption of practice and uncertainty about how best to train their therapists to improve it (J. Bayardo, personal communication, 30 October 2018). Agencies charged with supporting SED youth and families through Medicaid funding streams wrestle with how to increase the quality of and the access to-evidenced-based treatments (Graaf & Snowden, 2021). These expressed needs are the foundation of the academic-community partner relationship formed in 2017 between myself and Community Mental Health for Central Michigan (CMHCM) - a large community mental health service provider chartered to serve families with children experiencing serious emotional disturbance (State of Michigan, 2020). Academic-community partner relationships commonly include research activities that bridge scholarship and collaboration, seeking to generate mutually beneficial new scholarship and practice (Balazs & Morello-Frosch, 2013; Doberneck, Bargerstock, McNall, Van Egeren, & Zientek, 2017). The voice of the community partner is essential to affirming information on a research topic of interest and to validating the need for additional investigation (Brown & Lambert, 2013; Doberneck et al., 2017). 7 Discussions with CMHCM about their own experiences upheld what was available in the literature regarding the need to empirically examine a common therapist skillset and training methods that can support EBP adoption. For example, inadequate therapist training is confirmed in the research literature as a known obstacle to EBP adoption (see Beidas & Kendall, 2010; Karlin & Cross, 2014; Layne, Steinberg, & Steinberg, 2014) and a necessary driver to improve behavioral healthcare outcomes for individuals and communities (SAMHSA, 2020). Correspondingly, CMHCM reported that the penetration rates of multiple family focused EBPs were notably low, largely due to therapists not having gained the skills needed to deliver the EBP practice competently (J. Bayardo, personal communication, 30 October 2018; 17 May 2019). EBP fidelity data captured in the research literature underscores the challenges of implementation that leads to poor outcomes for children and families (Barnett et al., 2019; Horigian et al., 2016). At the same time, CMHCM fidelity data revealed therapist skill gaps that prevented positive fidelity findings, or maintenance of skills important to the EBP model (J. Bayardo, personal communication, 30 October 2018; 17 May 2019). Evidence from the literature and CMHCM provide a solid foundation for studying therapist skillsets related to EBP adoption. The voice of the community partner in research relationships serves a vital role in continuing to validate the need for targeted research efforts and advocates for research agendas that will ultimately impact their ability to deliver evidence-based treatment (Balazs & Morello-Frosch, 2013; Barnett et al., 2019; Davis, Kliewer, & Nicolaides, 2017; Doberneck et al., 2017). Negative Effects of a Research to Practice Gap Children with serious emotional and behavioral disorders continue to lack access to the best interventions available to treat their disorders (Barwick et al., 2008; SAMHSA 2020). This disparity is commonly referred to as research to practice gap (APA, 2008; NIH, 2018). 8 Ultimately, this gap undermines the delivery of effective treatments to address serious mental health needs in children (Barwick et al., 2008; SAMHSA, 2020), furthering the loss of confidence in implementation efforts that community mental health agencies undertake (Barwick et al., 2008; Saldana, 2014; Saldana & Chamberlain, 2012). Unfortunately, knowledge gaps around implementation strategies that can support therapist’s adoption of EBPs persist and keep EBPs from routinely moving into community mental health practice where they can benefit SED children and their families (Barnett et al., 2019; Barwick et al., 2008; Saldana & Chamberlain, 2012). Sadly, half of SED diagnosed youth go without mental health treatment (Merikangas et al., 2010). EBP implementation research efforts seek to close the research to practice gap by replicating in the community setting the positive client outcomes that are achieved in research contexts, e.g., controlled clinical trials (Barnett et al., 2019; Marty, Rapp, McHugo, & Whitley, 2008). Positive outcomes for SED children are demonstrated when efforts to support therapist adherence through training and consultation are adhered to (Barnett et al., 2019). Understanding the specific skillset therapists need to support the adoption of family focused EBPs offers hope that positive client outcomes can continue to be achieved in community practice (Horigian, Anderson, & Szapocznik, 2016; Onken, Carroll, Shoham, Cuthbert, & Riddle, 2014). Increasing Needs of SED Children and Their Families Current public health trends support the need for increased adoption of EBP implementations for SED children and families (Nelson et al., 2012). Child behavioral and emotional problems are among the most common reasons for referrals to health services, with about five million U.S. children diagnosed with a mental disorder each year (Centers for Disease Control [CDC], 2018). While one in ten youth are estimated to require treatment for SED 9 (Williams, Scott, & Aarons, 2018), less than half will receive treatment for this condition within one-year of the diagnosis (Merikangas, et al., 2010). Many youths identified with SED have serious enough symptoms to significantly impair their functioning at home, at school, and in the community, which puts them at a greater risk for adverse long-term social, behavioral, or health outcomes (Friedman, Katz-Leavy, Manderscheid, & Sondheimer, 1996; Hodges et al., 2004; Mitchell, Kern, & Conroy, 2018). SED embodies a complex set of mental health problems for youth, which are most likely to include comorbid psychiatric disorders and a higher risk of treatment drop-out or failure (Kessler et al., 2012; Williams et al., 2018). Lack of access to, or completion of, specialized treatment for SED symptoms has serious and long-term public health impacts for children and families (NIH, 2018; Loras, 2018; Substance Abuse and Mental Health Services Administration [SAMHSA], 2020). For instance, suicide is currently the second leading cause of death for children ages 12-17 (CDC, 2018; Davis, 2014; Perou et al., 2013). Moreover, children referred for mental health care have higher prevalence rates of exposure to many forms of trauma and loss which have adverse health effects over their lifespan (Briggs et al., 2012; Lupien, McEwen, Gunnar, & Heim, 2009). In addition, an estimated 16% of children exposed to trauma develop diagnostic criteria for post-traumatic stress disorder (Alisic et al., 2014). Implementation research efforts that support the increased adoption of EBPs by therapists are necessary to assure the delivery of research-based treatments that can relieve SED symptoms in children and families (NIH, 2018; SAMHSA, 2020). Gaps in Therapist Training and Development The need to support therapists toward aims of adopting and sustaining EBPs in community mental health practice has been a consistent call in the literature and from 10 community practitioners (APA, 2008; Barwick et al., 2008; Beidas & Kendall, 2010; J. Bayardo, personal communication, 17 May 2019; NIH, 2018). Yet despite this consistent call, the skills and strategies that are necessary to achieve this aim remain unclear (Beidas et al., 2014; Fixsen et al., 2005; Rapp et al., 2010). Therapist training and consultation is one known implementation driver supporting EBP adoption (NIH, 2018; SAMHSA, 2020; Weisz & Kazdin, 2010) and is critical since the implementation practices of therapists directly impact EBP delivery to SED children and their families (Barnett et al., 2019; Barwick et al., 2008). Therapists are often represented in implementation research literature as needing more training (e.g., lacking in leadership skills and basic clinical competence), particularly with skills important to understanding children’s mental health care needs within the context of their family system (Aarons et al., 2011; Loras, 2018; Proctor et al., 2011). Therapist skill deficits often show themselves in poor EBP fidelity ratings, which is one of the reasons for EBP implementation failures in community mental health settings (Becker & Stirman, 2011; Bond et al., 2014; Horigian et al., 2016). It is well understood that community mental health agencies play a critical role in supporting therapists’ success in adopting EBPs through the training opportunities they offer (Aarons et al., 2011; Beidas & Kendall, 2010; J. Bayardo, personal communication, 30 October 2018). Research literature validates that successful EBP adoption by therapists cannot be achieved through singular EBP trainings without ongoing follow-up trainings (Allan, Ungar, & Eatough, 2018; Beidas et al., 2014; Bond et al., 2014). Further, agencies that train therapists in core competencies common across multiple EBPs fare better with EBP implementation efforts overall, as the training promotes critical thinking skills necessary to translate learning into real- world practice (Manuel, Mullen, Fang, Bellamy, & Bledsoe, 2009). 11 While therapist training plays a significant role in EBP adoption and subsequent delivery of services to children with SED and their families, little agreement exists around specific training gaps for therapists that can support better adoption of EBPs (Onken et al., 2014; Weisz & Kazdin, 2010). This in turn provides an opportunity to further examine what therapist training needs really are. Therefore, defining a foundational therapist skillset that can support the adoption of EBPs in community practice is necessary to keep EBP implementations from failing (Bond et al., 2014; NIH, 2018; Weisz & Kazdin, 2010) and to keep the research to practice gap from widening (Onken et al., 2014). Once defined, foundational therapist skillsets needed to promote EBP adoption can be translated to interventions capable of sustaining implementation efforts in community practice (Beidas et al., 2014; SAMHSA, 2020; Stirman et al., 2012). Purpose of the Study and Aims The purpose of this mixed methods modified Delphi study (Linstone, 1975) was to determine a foundational therapist skillset necessary for therapists to effectively adopt EBPs for children with SED in community mental health agencies. This study had three aims. The first aim was to identify, through brainstorming, a breadth of foundational therapist skills that EBP experts identify as important to learning and adopting EBPs for SED children and their families. The second aim was to narrow the skills identified in the first aim and build consensus around the core foundational skillset therapists need to acquire to ultimately improve their adoption of family focused EBPs. The third aim was to use the expert feedback obtained through the Delphi process to summarize a set of foundational therapist skills and corresponding training methods that can inform implementations that support therapist adoption of EBPs in community mental health agencies with SED children and families. 12 Method: Modified Delphi Process This study was completed through a modified Delphi process (Keeney, Hasson, & McKenna, 2011; Linstone, 1975). The Delphi process is a targeted, effective, group survey procedure able to (1) extract expert input on specific research topics that can assist with developing or validating content, and (2) utilize expert feedback in such a way that group consensus can occur (Lechowicz & Gazda, 1975). Consensus is built in the Delphi method when participants provide importance rankings on research statements throughout each round (Linstone, 1975). Participants were recruited for the Delphi study based on their expertise with EBPs that treat children with SED symptoms within their family context. The Delphi process started with a round of brainstorming that offered participants an opportunity to respond to three open-ended questions about therapist skills necessary for treating SED children and families (Keeney et al., 2011). Content from Round 1 was analyzed to create statements that were sent electronically to the participant panel for agreement ranking in Round 2 (Burnard, 1991; Keeney et al., 2011). Round 3 brought together participants in small focus groups to recap the foundational therapist skillset and discuss corresponding training methods to target the skillset areas. This feedback was vital toward the goal of informing future training interventions that can ultimately support therapists in learning EBPs. Mixed-method study. This was a sequential exploratory mixed-methods study where the results of the first qualitative analysis (Round 1) informed the subsequent quantitative methods (Round 2), resulting in an emerging framework for research inquiry (Creswell, Plano Clark, Gutmann, & Hanson, 2003). Mixed-method studies are widely supported in implementation research due to the complexity of studying implementation phenomena (Palinkas et al., 2015). Further, mixed-methods studies are supported in Delphi processes and especially in areas of 13 health research as they can be helpful in refining findings and impacting clinical care (Kennedy, 2004). Community-Academic partnership. A community partner relationship established with CMHCM affirmed findings from the research literature about the need for this study. Community-university partnerships can provide support in validating the need for areas of research inquiry and lend additional expertise in study development activities, e.g., assisting with recruitment efforts, triangulating data (Balazs & Morello-Frosch, 2013). In addition to affirming the need, CMHCM provided feedback on Round 1 questions before they were administered, supported study recruitment, and served as an additional measure of triangulating Round 1 data on skill statements. Research Questions Agreement on the foundational therapist skillset most likely to support the adoption of EBPs is needed to ultimately provide SED children and families more access to research-based care. This dissertation study will contribute to the dissemination and implementation scholarship and the literature on children’s mental health by determining a foundational therapist skillset and training methods necessary to improve EBP adoption in community settings. To address this and the study aims stated, the following research questions were explored: 1. What are the foundational therapist skills experts identify as important to improving therapist adoption of family focused EBPs targeting SED symptoms in children? 2. What is the consensus among panel experts on the core foundational skills therapists need to acquire to improve adoption of family focused EBPs in community mental health agencies? 14 3. What are the corresponding training methods for each identified foundational therapist skill that can ultimately inform an intervention to improve therapist adoption of EBPs in community mental health agencies treating SED children and their families? Summary Chapter 1 provides the background to the importance of this study in establishing an agreed-upon, foundational therapist skillset that can improve family focused EBP adoption in community health settings. Studying EBP implementation, while a complex undertaking, is appropriately considered in an adapted systems-contextual framework. This framework recognizes a consensus skillset situated among the therapist (EBP adopter), training methods, and implementing organization. This study responds to logical next steps suggested by the implementation literature as it relates to therapist training and improved EBP adoption in community mental health settings (Graaf & Snowden, 2021). Further, the need for this study is affirmed in the context of an academic-community research partnership that outlines low EBP adoption and uncertainty about how to invest in continued EBP training. The Delphi process is introduced as the research design used to garner expert consensus on therapist skillset and training methods necessary to improve EBP adoption in community health settings. The next section summarizes a review of the literature on therapist skillsets and the rationale for the study of therapist skills that crosscut multiple family focused EBPs. 15 CHAPTER 2: LITERATURE REVIEW Core therapist skills that lead to effective EBP implementation efforts are worthy of investigation, yet are often understudied (Beidas et al., 2014; Proctor et al., 2009). Improving core therapist skills is a promising method supporting EBP adoption efforts in community mental health settings (NIH, 2018; Weisz & Kazdin, 2010). However, little agreement exists on the skills needed for such training efforts (Bond et al., 2014; Karlin & Cross, 2013). This chapter will continue to build the framework established in Chapter 1 needed to examine core therapist skills and substantive training methods likely to support EBP adoption in community mental health agency settings. Chapter 1 highlighted the importance of this study’s fit in the larger dissemination and implementation context and how its need is validated through the support of a community- academic relationship. An adapted systems-contextual framework (Beidas & Kendall, 2010) supported by systems theory (Bateson, 1979; Von Bertalanffy, 1968) solidifies the concept of interdependence by studying therapist skills and training methods, both of which are key pieces of this study’s research questions. In this chapter, I will first examine the common factors paradigm which provides a necessary structure to study therapist skills common to multiple EBPs. Next, a synthesis of the literature pertaining to therapist competencies and training methods will be reviewed. Finally, prospective areas for therapist skillset development will be identified as potentially supportive to EBP adoption. These will be reviewed for inclusion in Delphi (Round 2) processes. 16 Common Factors Paradigm Common Factors Defined Simply defined, a common factors paradigm identifies the components of therapeutic practice shared among various effective therapies (Davis, Lebow, & Sprenkle, 2012; Sprenkle & Blow, 2004). A common factors paradigm (Sprenkle & Blow, 2004) provides a rationale for the idea that therapist skills that support EBP adoption will be relevant to multiple EBPs that treat children with SED and their families. Common factors recognize the dimensions of effective treatment that operate through EBPs but are not unique to those specific treatment models or theories of practice (Sprenkle & Blow, 2004). In the narrowest view, a common factor of treatment may be considered akin to a specific deliverable therapeutic skill, e.g., reframing or externalizing a problem (Lambert, 1992). A broader view of common factors considers elements such as client engagement or the therapeutic relationship (Hubble, Duncan, & Miller, 1999; Sori & Sprenkle, 2004). Both common factor views are important to the design of this study as the aim is to build consensus around those core therapist skills most important to supporting EBP adoption. A Common Factors Conceptualization of Core Therapist Skills A common factors paradigm is critical to the conceptualization of this study. First, it recognizes that there are many commonalities among family therapy models (Sprenkle et al., 1999) and therapists who provide effective treatment (Blow, Sprenkle, & Davis, 2007). Subsequently, a common factors paradigm provides a rationale for the idea that consensus around important core therapist skills that support adoption of EBPs in community practice can be built through commonalities among models and components of effective practice. The necessity of seeing core therapist skills through this lens is underscored by this study’s 17 community partner, CMHCM, as a very practical need for continuing to provide quality treatment to SED children and their families. They emphasized that understanding a foundational skillset is not only critical to the sustainability of multiple EBPs in their agency, but an important part of how they will continue to strategize opportunities for ongoing therapist professional development (J. Bayardo, personal communication, 30 October 2018; 17 May 2019). Validating therapist skillsets across multiple EBPs is a promising method of implementation research inquiry. Second, while research literature identifies emerging themes of important core therapist skills that support EBP adoption, there lacks specificity in how to categorize these skills into a meaningful skillset that could have impact across many EBPs (see Karlin & Cross, 2014; Layne et al., 2014a; Merlo et al., 2010). To this point, Chorpita et al. (2011) assert that specific EBP training often favors learning manualized treatment protocols, which comes at the cost of distilling the common practice elements that can be taught to support therapists in delivering effective mental health treatment more broadly. Subsequent sections will provide a review of the research evidence from EBP implementations as they relate to deficits in therapist skills. It is important to have a framework with which to understand this literature so that a skillset appropriate for future training can be identified and able to support therapist learning beyond specific EBP model techniques (Karlin & Cross, 2014). Chapter 3 reviews the methodological approach for this study. This approach includes analyzing qualitative content regarding therapist skills necessary to work with SED children and their families. A common factors paradigm provides both the structure and a research base that could be helpful for analyzing and organizing the content of expert responses in the Delphi process as it relates to providing treatment for family systems (Sprenkle & Blow, 2004; 18 Sprenkle, Blow, & Dickey, 1999). The next section will review what the research literature has to say about therapist skillsets identified as potentially able to improve EBP adoption. Therapist Competency Therapists, while critical to outcomes for children and families, are rarely the focus of implementation research studies (Allan et al., 2018; Blow et al., 2007; Orlinsky & Ronnestad, 2005; Wampold & Brown, 2005). Yet, they are the direct link transporting EBPs to families and children with SED in community settings (Aarons, 2004; Beidas & Kendall, 2010; Wampold, 2006) and are a vital ingredient in therapy outcomes (Blow et al., 2007). While therapists are encouraged by their agencies to adopt EBPs to increase their practice skills and improve outcomes for families, there remains an opportunity to better understand the specific skills needed to support EBP adoption (Beidas & Kendall, 2010; NIH, 2018). This section identifies important themes in the implementation research literature related to therapist general competency, as measured through fidelity and skill assessment. General Competency A lack of therapist competency is commonly cited in the literature as a barrier to successful EBP implementations (Adily & Ward, 2004; Becker & Stirman, 2011; Karlin & Cross, 2014; Manuel et al., 2009). Competency is not a well-defined term in the literature (Carr, 2019), nevertheless remains an important therapist variable connected to training efforts that are thought to support EBP adoption (Karlin & Cross, 2014; Manuel et al., 2009; Rapp et al., 2010). One definition of therapist competency simply refers to the quality with which interventions are delivered (Hogue et al., 2008). Other times, competency is referred to as an overarching measure defining adherence or fidelity to a specific EBP model (Perepletchikova, Treat, & Kazdin, 2007). 19 Competency and fidelity, while not synonymous, do share some similarities in the context of EBP implementation. In some instances, competency refers to the performance of advanced skill sets or expected methods commonly taught in EBPs, reliably evaluated with a fidelity checklist (Becker & Stirman, 2011). In other instances, it relates to general therapist skills fundamental to therapeutic practice (Karlin & Cross; 2014; Sprenkle, Davis, & Lebow; 2009). Fidelity information, while not completely akin to assessing competency, does provide a method of understanding how certain components of EBP practice are delivered. Examining fidelity data can provide clues for training areas that may need improvement and support EBP adoption (Becker & Stirman, 2011; Karlin & Cross; 2014; Sprenkle, Davis, & Lebow; 2009). Therapist Skill Development Standards for meeting fidelity to EBPs tend to focus on the learning of specific model techniques which may or may not be a fair representation of overall therapist development (Fruggeri 2012; Sprenkle, Davis, & Lebow, 2009). Current “gold standard” training methods target therapist skill development through workshops, manuals, and follow up supervision specific to EBP techniques (Beidas & Kendall, 2010; Sholomskas, Syracuse-Siewert, Rounsaville, Ball, & Nuro, 2005). While sufficient for some, this standard fails to recognize the likelihood that therapists have not had prior exposure to basic EBP training through their graduate programs or formal coursework (Beidas & Kendal, 2010; Karekla, Lundgren, & Forsyth, 2004; Wittenborn, Blow, Holtrop, & Para-Cardona, 2019). Frequently, EBPs emphasize learning specific skills based on an assumption that therapists are already well-versed in other critical therapy skills, e.g., attunement, therapeutic relationship building, engagement (Allan et al., 2018; Norcross & Wampold, 2011). In truth, therapists may need skill development in multiple areas to be successful at EBP adoption (Allan et al., 2018). 20 Therapist skill deficits have been explored by looking at fidelity trends specific to EBP implementations (e.g., Horigian et al., 2016). Fidelity data offers evidence of therapist skills necessary for that specific model (Allen & Johnson, 2012; Horigian et al., 2016; Karlin & Cross, 2014). However, caution should be noted in automatically translating these skills to a foundational skillset for therapists to have, since fidelity data are more likely to reflect deficits in specific model techniques over more generalizable therapeutic skills (Karlin & Cross, 2014; Perepletchikova, Treat, & Kazdin, 2007; Sprenkle, Davis, & Lebow, 2009). This point is underscored when considering that even when therapist fidelity in EBPs is carefully monitored, variability in therapist competence remains (Blow et al., 2007; Wampold, 2006). Despite this caution, it is worthwhile to examine what has been evaluated to date regarding EBP implementations and assessments of fidelity for specific EBPs. Potential Areas for Therapist Skillset Development The good news is that the literature guides us to some promising considerations for building consensus around core therapist skills that can support adoption of EBPs for children with SED and their families. This section highlights key areas of skill development gleaned from fidelity monitoring and implementation research specific either to certain EBP models, or to specific community health settings. While not conclusive, these areas offer some evidence supporting a set of therapist skills promising to increase EBP adoption. The skills that follow are referenced in the method section as a measure of triangulation for Round 1 Delphi coding processes, further discussed in Chapter 3 (Burnard, 1991). Therapist adaptability. Therapist adaptability, including the attitudes they have about their practice, affect EBP training and adoption (Aarons et al., 2004; 2011; Rogers, 2003). Therapist adaptability includes how they perceive and process their experiences with EBPs 21 (Aarons et al., 2009). Research supports the idea that therapists who value innovation and novelty of learning new practices are more likely to successfully adopt EBPs (Aarons et al., 2011; Rogers, 2003). Conversely, therapists who are not open to innovation sometimes struggle with EBP adoption, particularly if there is not time for them to process the personal meaning of the practices they are learning (Allan et al., 2018; Aponte et al., 2009). Sadly, many therapists report EBP learning as one of their most difficult experiences in their career (Allan et al, 2018). There is little research exploring themes of adaptability to EBP adoption (e.g., Dowling, Cade, Breunlin, Frude, & Seligman, 1982; Green & Kirby-Turner, 1990). The upside is that in one recent study, therapists reported benefitting from having time, repetition, and safe space to process their EBP training (Allan et al, 2018). Additional research literature supports this claim, underscoring that helping therapists understand how they align with their EBP practice ultimately advances science and client care (Institute of Medicine, 2001; 2007). Providing training opportunities where therapists can examine their personal and professional experiences of EBP could be a helpful addition to core therapist skills supporting EBP adoption (Aponte et al., 2009; Nel, 2006). Additional research literature suggests that therapist training efforts should help them reflect on the reasons for changing their practice (when learning new skills) to increase the feelings of empowerment necessary for EBP adoption (Aarons, et al., 2004; 2006; Chambers, 2008). Overall, therapist attitudes toward EBP implementation and adoption in general are situated in a broader implementation research scope than is intended in this study. Therapist attitudes toward EBP are reflective of additional organizational and contextual factors (Aarons et al., 2011). However, therapist skill development toward how they perceive the benefits of EBPs, how they understand its congruence with their practice, and how they conceptualize a fit with the 22 community they serve are important factors to considering the aim of EBP adoption (Aarons, 2004; Aarons et al., 2011; Allan et al., 2018; Layne et al., 2014a; Loras, 2018). Therapist skill development around adaptability should not be overlooked even though it may be harder to operationalize than other therapeutic skills. Client and family engagement. Training therapists in skills to engage client family systems in treatment is suggested across several implementation studies (e.g., Becker & Stirman, 2011; Horigian et al., 2016; Karlin & Cross, 2014). Client engagement skills are defined as the ability to gain access to children and families as required to implement treatment (Becker & Stirman, 2011; Horigian et al., 2016). Additionally, engagement is conceptualized as cooperation in treatment, feelings of commitment to goals, and alliance formation (Cunningham & Henggeler, 1999; Prinz & Miller, 1994; Sori & Sprenkle, 2004). The need for therapists to train in engagement strategies is underscored by Karlin and Cross (2014) with an emphasis on training therapists in therapeutic relationship skills that can ultimately increase client commitment to EBPs. Weisz and Kazdin (2010) back this assertion stating that to address challenges in EBP adoption there needs to be improved research on how therapist behavior and the therapeutic relationship best connect treatment engagement, completion, and outcome. Further in support, Norcross and Wampold (2011) advise continued attention to supporting therapists in learning engagement efforts that can adapt treatment to specific client factors and help engage cultural communities in EBP treatment (Horigian et al., 2016). Client/family engagement is a critical skill common to EBP fidelity and without it, EBP implementations risk failure (Becker & Stirman, 2011; Karlin & Cross, 2014). For example, in a long-term evaluation of the Brief Strategic Family Therapy (BSFT) implementation (Szapocznik 23 & Hervis, 2020), less than one-third of BSFT therapists reached fidelity (Horigian et al., 2016). Common findings from fidelity data included therapist failure to engage family members in treatment (Rohrbaugh, 2014). Additionally, client/family engagement skills are critical to understanding how to adapt EBPs to diverse clients and ultimately improve their adoption (Horigian et al., 2016; Norcross & Wampold, 2011). Trauma-informed concepts. Trauma-informed concepts are defined as a core set of principles that help clinicians understand trauma response in children and families (Layne et al., 2014b). When taught, these concepts are intended to help guide therapists to competency around areas such as understanding the complex nature of trauma, working with trauma reactions and reminders, explaining basic neurobiology of trauma, and evaluating trauma response and recovery through a cultural lens (see National Childhood Traumatic Stress Network Task Force, 2012). The relevance of developing trauma competency outside of specific EBP learning rests in the idea that clients referred to mental health services often have higher rates of exposure to trauma (e.g., sexual abuse, complicated grief, intimate partner violence) and children exposed to trauma are at higher risk for developing post-traumatic stress symptomology (Alisic et al., 2014; Briggs et al., 2012). For these reasons, Layne and colleagues (2014b) contend trauma concepts should be standard training for therapists learning EBPs. Furthering this point, Layne and colleagues (2014b) and Loras (2018) found significant evidence of improvement in therapists’ attitudes toward EBP adoption when they trained in core competencies that taught therapeutic skills for engaging in trauma informed care. Therapists reported that training in these skills offered them an opportunity for empowerment that was congruent with their lens of conceptualizing treatment for youth and families who experienced significant trauma exposure (Layne et al., 2014b; Loras, 2018). 24 Foundational trauma competencies are suggested as a core training method for therapists that can support different phases of EBP delivery (e.g., client engagement, assessment, treatment) and be applicable to multiple EBPs (Courtois & Gold, 2009). Further, trauma competencies are thought to help EBP practitioners utilize good professional judgement within EBP frameworks for interventions that may not be sensitive to a client’s experiences (Layne, et al., 2014a). In a separate study, Layne and colleagues (2014b) expressed concern that basic trauma principles have yet to be distilled in many manualized EBP interventions. Therefore, training on these competencies can help therapists be trauma-sensitive within the confines of their EBP practice (Chorpita et al., 2011). Systemic case conceptualization. Systemic case conceptualization is a critical therapy skill that considers client engagement, therapeutic alliance, or other specific treatment strategies based on factors that are beyond a client diagnosis or identified EBP treatment path (Coulter 2011; Sprenkle, Blow & Dickey, 1999; Staller 2006). Case conceptualization is key to avoiding the therapy trap of having a one size fits all approach (Nelson, Steele, & Mize, 2006). Training in case conceptualization as a skill is thought to be congruent with multiple EBP implementation efforts (Karlin & Cross, 2014). For example, a review of multiple, concurrent EBP implementations initiated through the Veterans Health Administration (VHA) system yielded findings that therapists lacked core competency in client engagement in treatment, case conceptualization, and trauma-informed care, which greatly impacted their ability to use the EBP approaches in which they were trained (Becker & Stirman, 2011; Fixsen et al., 2005; Karlin & Cross, 2014). Of these deficits, improvements in case conceptualization skills were found to benefit many of the EBPs practiced within the organization’s health system. Case conceptualization helped therapists better 25 understand the needs of the veteran population who often experience complex health and psychosocial concerns, further assisting them in their work with veterans in an agency setting with multiple EBP implementations. This skill development was found to ultimately improve therapist adoption of EBPs in their setting (Karlin & Cross, 2014). An emphasis on therapists learning case conceptualization skills is seconded in a qualitative study by Loras (2018). This study outlined the need for community practice agencies to support therapists in learning skills for conceptualizing SED child treatment within the family system in order to provide specialized child mental health care. These examples support the inclusion of case conceptualization as a key therapist skill worthy of further investigation in supporting EBP adoption. Reframing or changing the view. Based on data from multi-year EBP effectiveness studies, Horigian and colleagues (2016) assert the importance of the therapists having strong reframing skills. Reframing is a therapeutic skill that refers to changing the meaning that a person assigns to another person or situation (Szapocznik & Hervis, 2020), or simply changing the view (Sprenkle & Blow, 2004). This skill is critical for therapists to have to decrease negativity in family interactions and encourage retention in the therapeutic process (Fernandez & Eyberg, 2009; Robbins, Alexander, & Turner, 2000; Szapocznik & Hervis, 2020). Robbins and colleagues (2000) also emphasize that a therapist’s ability to learn reframing is an important strategy that can protect the therapeutic alliance with the client and family. Reframing is a common skill inherent to many systemic family therapy models and an established basis for developing advanced therapeutic skills (Weeks & D’Aniello, 2017). An example of its importance as a basic skill -- and what happens when it is not present -- is highlighted in the Trauma Focused Cognitive Behavioral Therapy (TFCBT; Cohen, & 26 Mannarino, & Deblinger, 2006) literature and effectiveness studies. First, skill in reframing is significant to building caregiver involvement, which significantly impacts the treatment outcome for youth. Therapists need reframing skills to support the child and the parent or caregiver’s ability to participate in treatment and be able to achieve mastery in reframing the meaning of the trauma (Brown, Cohen, & Mannarino, 2020). Second, in TF-CBT fidelity studies, reframing skill is found to be a protective factor for alliance building with youth – offering necessary reflections and validation that can continue to promote cognitive restructuring necessary to reduce trauma symptoms (Ovenstad, Ormhaug, Shirk, & Jensen, 2020). Last, in a utilization review of a large TF-CBT community implementation site, Allen & Johnson (2012) share findings therapists often do not complete the cognitive restructuring component, a skill based heavily in the ability to reframe maladaptive thoughts related to the trauma (Cohen, Mannarino, Deblinger, & Berliner, 2009). Following this implementation study, Allen and Johnson (2012) concluded there is a need for increased therapist education supporting their ability to challenge or change the view of unhelpful cognitions around trauma experiences in the family system. Behavioral strategies that support caregivers. Child behavioral problems are a significant symptom in a SED constellation and one that, unattended, can lead to significant safety and additional externalizing problems for youth and families, including hospitalizations and out of home placements (Graaf & Snowden, 2020; Hill, 2017; Hodges et al., 2004). In a TF- CBT implementation study, Allen and Johnson (2012) affirm the need for therapists to have skill in coaching parents on behavioral approaches that can support caregivers with implementing child behavior management strategies. According to Cohen and colleagues (2009), training therapists in behavioral approaches is necessary to assure that caregivers who manage difficult child behavior can be supported and continue to maintain engagement in EBP treatments. 27 There is a growing body of research finding that supporting parents with child behavior is critical to managing several diagnoses falling within the SED spectrum. For example, success with parent training in behavioral strategies is noted for childhood onset disorders such as autism spectrum (Iadarola et al., 2017) and attention deficit hyperactivity disorder (van der Oord & Tripp, 2020). Additional studies on specific parent management strategies targeting SED symptoms find promising outcomes for youth in the child welfare and foster care systems (e.g., see Akin & McDonald, 2018), commonly diagnosed with SED symptoms (Hodges et al., 2004). Self-reflection. Self-reflection induces therapists’ reflective understanding of their experiences in therapeutic practice, guided by personal opportunities to explore meaning and implications inherent to working as a therapist (Scott et al., 2021). As it relates to EBP implementation, this skill acknowledges that therapists’ experiences with EBP impact the way in which they respond to any kind of EBP training (Aarons et al., 2011; Allan et al., 2018; Beidas & Kendall, 2010). There is small but compelling evidence that continued focus on teaching manualized skills common to EBPs may come at the cost of promoting skills in self-reflection that help therapists integrate their experiences in providing treatment (Allan et al., 2018). Self-reflective work encompasses ways in which therapists know themselves and their personal challenges, experience their emotions, and manage their own responses in treatment (Aponte et al., 2009). Integrating this kind of training is a known, consistent challenge (Watson, 1993), particularly as it involves time, emotional capacity, and safety to operationalize (Aponte et al., 2009). However, when considering evidence from Allan and colleagues (2018), continuing to neglect therapists’ emotional experiences and capacity in learning EBP treatment further assures lack of overall adoption. Moreover, a therapist’s need for time and space to practice difficult emotions, understand emotional triggers in treatment, and maintain presence in session 28 is well supported in the literature (Aponte et al., 2009, Hubble, Duncan, & Miller, 1999; Owen, Duncan, Reese, Anker, & Sparks, 2014; Orlinsky & Ronnestad, 2005; Rousmaniere, 2016). While perhaps a bit more challenging to operationalize, self-reflective skill development is critical to therapeutic practice. Cultural humility and attention to diversity. Essential to building empathic connections, attending to diversity and taking a stance of cultural humility emerges as a key therapist skill. Generally, this skill builds on two important functions: 1) to uniquely validate a person’s experience in their multiple identities and, 2) to elicit important discourse about cultural differences and power in relationships that can facilitate positive change (Crenshaw, 1991; Mock, 2008; Sue & Sue, 2015). Simply put, this skill is a process of self-reflection that emphasizes honest and trustworthy relationships, critical in understanding clinical research efforts (Yeager & Bauer-Wu, 2013). Interestingly, most of the research literature on therapist skill development related to EBPs did not identify cultural humility specifically, but rather included it as a part of other skills, e.g., trauma informed care (see Layne et al., 2014b), self-reflective practice, (see Aponte et al., 2009), or systemic conceptualization (see Karlin & Cross, 2014). While not related to specific EBPs implementations, the ability for a therapist to bring a culturally sensitive, intersectional focus into the forefront of treatment relationship is well supported in the literature (PettyJohn, Tseng, & Blow, 2020). Cultural humility is also included as a vital foundation for clinical research and ever-present in community health delivery, therefore is being included here as a potential piece of the therapist skillset (Yeager & Bauer-Wu, 2013). 29 Research on Learning Methods Effective training processes are important to supporting successful EBP implementations (Metz, 2016). The review of literature on therapists’ skillsets yielded considerable evidence of therapist engagement in active learning strategies as a key to learning EBP skills. For example, the Veterans Health Administration found therapist behavior toward EBP adoption changed positively after therapists were engaged in training exercises that employed the same active strategies they were being taught to use to engage their clients in treatment (Karlin & Cross, 2014). Other EBP models, e.g., Parent Management Training Oregon model (GenerationPMTO) regularly use active learning strategies for therapist training to support behavior change and success in fidelity (e.g., Akin et al., 2014; Bearman et al., 2013; Forgatch & Patterson, 2010). Adult learning theory is an appropriate framework for considering these strategies as important training components since EBP training processes are aimed solely at adult learners (Kenner & Weinerman, 2011). Adult learning theory posits the importance of active engagement in learning, a therapist factor highly referenced in the implementation literature (Allan et al., 2018; Bearman et al., 2013; Kenner & Weinerman, 2011). Key principles of adult learning include providing a rationale for the content of material to be learned, supporting learners’ motivation to solve problems, respecting (and building on) learners’ prior experiences, matching approaches to a learner’s background and diversity, and engaging the learner in active processes, such as working with peers (Bryan, Kreuter, & Brownson, 2009). Further, supervision modeling and therapist-enacted role-plays, both active learning strategies, predict greater use of EBPs (Bearman et al., 2013). In addition to considering therapist skillset, it is important to be concerned about training methods capable of supporting therapist acquisition of skills and ongoing development. 30 Summary While implementation research is complex (Beidas et al., 2014), the need for effective treatment for children with SED and their families demands we understand how to better support therapists’ adoption of EBPs (Nelson et al., 2012). Through exploration of a common factor paradigm (Sprenkle & Blow, 2004), this chapter finalizes the framework needed to study therapist skillsets in implementation research across multiple EBP models. Further, by examining implementation and fidelity studies, therapist skill gaps related to specific EBPs (see Allen & Johnson, 2012; Horigian et al., 2016) and across specific treatment settings (see Karlin & Cross, 2014) are illuminated. Unfortunately, therapist skillset information gleaned from the literature so far remains either focused on specific EBP models (e.g., Allen & Johnson, 2012), or without clear methodology on providing training for certain skills (Allan et al., 2018). Either way, the literature falls short toward the aim of an agreed-upon foundational skillset therapists should have to improve their adoption of EBPs that treat SED symptoms in youth using a family systems approach. More work is needed to not only define a foundational therapist skillset, but to understand the methods that would be helpful toward training therapists in the future. This study will utilize the Delphi method to confirm a foundational therapist skillset that is able to support therapist adoption of family focused EBPs with SED children in community mental health settings. It will also pair this consensus skillset with training methods able to achieve therapist skill development, providing a pathway for future intervention development that can impact EBP adoption. 31 CHAPTER 3: RESEARCH METHODOLOGY A three-round exploratory, sequential, mixed methods modified Delphi study (Creswell & Plano Clark, 2011; Jorm, 2015) was used to determine a consensus therapist skillset supporting the adoption of EBPs in community health settings and a set of accompanying training methods. A mixed-methods design was chosen for its ability to demonstrate a complete picture of the phenomenon under study (Leedy & Ormrod, 2010). Mixed method designs are supported in the literature when studying EBP implementation outcomes as they can potentially address the complexities of the process (Palinkas et al., 2015; Saldana, 2014). Round 1 study results were analyzed using qualitative content analysis methods and coding triangulation with the research literature (Burnard, 1991), a coding team, and my community partner. Round 2 results include quantitative analysis and descriptive statistics for determining agreement among the Delphi participant panel. Specifically, median and interquartile range statistics were analyzed (Treveylan et al., 2015). Round 3 results were analyzed using qualitative content analysis (Burnard, 1991). Mixed methods research designs are favorable in Delphi studies for adding impact and versatility (Sandelowski, 2000) and are well- regarded as a best practice for investigating the complexity of health research (Jorm, 2015; Keeney et al., 2011; Kennedy, 2004). I start this methods section further outlining the academic- community partnership that informed this study by highlighting the role of my community partner in the study conception and design. Academic-Community Partner Relationship As reviewed in Chapter 1, an academic-community partner relationship with CMHCM was key to confirming the need for this study. Academic-community partnerships are often described as beneficial in supporting research designs, particularly when the research inquiry 32 seeks to support practical interventions that can be translated into community practice (Balazs & Morello-Frosch, 2013; Dari, Daux, Lieu, & Reynolds, 2019; Doberneck et al., 2017). Moreover, community providers are essential stakeholders in understanding implementation barriers common for youth treated in community settings (Goodcase, Brewe, White, & Jones, 2021). Although the research literature provides robust support for studying core therapist skills important to the adoption of EBPs, working with CMHCM further illuminated the need to identify specific therapist training components and methods. This kind of research data is necessary to inform future training interventions and make strategic decisions about investing in therapist training and program development (J. Bayardo, personal communication, 30 October 2018; 17 May 2019). Community partnerships formed to study EBPs are regarded favorably for their ability to examine study outcomes from more than one perspective and to use the knowledge gained to directly sustain EBP practice in communities (Miao, Umemoto, Gonda, & Hishinuma, 2011). Sharing of power and responsibility. Community-engaged research methods facilitate collaborative approaches that offer sharing of power among partners (Balazs & Morello-Frosch, 2013; Leavy, 2017). However, levels of collaboration vary in academic community partnerships (Balazs & Morello-Frosch, 2013; Doberneck & Dann, 2019). Figure 3.1 demonstrates through the community engaged research abacas, the degree of responsibility and proportion of voice of the community partner and investigator reflected in this research partnership (Doberneck & Dann, 2019). Center beads highlight the balance of voice regarding the research process steps. CMHCM had some level of voice in all steps of the research process through critical analysis of the research study. However, equal voice is most demonstrated in the abacas for the steps of 33 identification of the research study need, implementation challenges, and how to disseminate findings in the future. Also noted in the abacas is the responsibility of each partner, exhibited by the beads that fall to the left and right sides. As demonstrated in Figure 3.1, CMHCM held responsibility for identifying implementation issues, informing the study need, reflecting on data, and eventually, disseminating findings. In addition, they assisted the primary investigator with initial outreach contacts to potential study participants and supported data reflection by triangulating content themes analyzed from Delphi Round 1. However, I held equal responsibility for understanding implementation issues and informing the need for the study from the literature. Full responsibility was assigned to this investigator for decisions on research design and carrying out the study. This is congruent with the nature of this study being a dissertation project fulfilling degree requirements. Figure 3.1 Community Engaged Partnership Research Abacus Steps in Research Process Voice and Responsibility CMHCM University 1. Identify EBP implementation issues and strengths 2. Inform need for study 3. Decide on research questions 4. Select research design 5. Develop research instruments 6. Recruit participants 7. Collect data 8. Analyze data 34 Figure 3.1 (cont’d) 9. Critically reflect on data 10. Disseminate findings Note: Adapted from Doberneck, D. M., & Dann, S. L. (2019). The degree of collaboration abacus tool. Journal of Higher Education Outreach and Engagement, 23(2), 93-107. Researcher Positionality Researcher positionality includes acknowledging my identity as both a researcher and a community partner in this study. While there is theoretical support for the researcher sharing subculture with research participants from an ecological perspective (Bronfenbrenner, 1979), acknowledging my location and positionality is an ethical research practice. I come into the role of primary investigator as a licensed master’s level social worker with over 20 years of experience working in community healthcare settings, having provided family treatment, supervision, and training in several models of evidenced-based treatment with fidelity. I identify with many of the same categories as the expert participants in this study, e.g., fidelity monitor, EBP consultant, master trainer, and author. In addition, I acknowledge that I maintain an active working relationship with CMHCM, the study’s community partner. I have kept these points of position open for discussion in an attempt to mitigate bias by building in opportunities for critical reflection with my community partner, my academic advisors, and my community-engaged scholarship mentor. Community-engaged research strategies are inclusive of partners acknowledging their positions throughout partnerships and reflective discussions are encouraged (Balazs & Morello-Frosch, 2013). This study is intended to fulfill requirements for my doctoral program. My specific interests in this scholarship area are undoubtedly rooted in experiences implementing EBPs in community mental health care settings. Further, I acknowledge my commitment as a SAMHSA 35 Minority Fellow (SAMHSA, 2020) that intentionally supports research activity seeking to improve the adoption of EBPs for SED youth in community settings. Purpose of the Study and Aims The purpose of this mixed methods modified Delphi study (Linstone, 1975) was to determine a foundational therapist skillset that is necessary for therapists to effectively adopt EBPs that treat children with SED in community mental health agencies. This study has three aims. The first aim is to identify, through brainstorming, a breadth of foundational therapist skills that EBP experts identify as important to learning and adopting EBPs for SED children and their families. The second aim is to narrow the skills identified in the first aim and build consensus around the core foundational skillset therapists need to acquire to ultimately improve their adoption of family focused EBPs. The third aim is to use the expert feedback obtained through the Delphi process to summarize a set of foundational therapist skills and corresponding training methods that can ultimately inform an intervention supporting therapist adoption of EBPs in community mental health agencies with SED children and families. Research Questions Agreement on the foundational therapist skillset most likely to support the adoption of EBPs is needed to ultimately provide SED children and families in communities more access to research-based care. This dissertation study determined a foundational therapist skillset necessary to improve EBP adoption in community mental health settings. The following research questions were investigated: 1. What are the foundational therapist skills experts identify as important to improving therapist adoption of family focused EBPs targeting SED symptoms in children? 36 2. What is the consensus among panel experts on the core foundational skills therapists need to acquire to improve adoption of family focused EBPs in community mental health agencies? 3. What are the corresponding training methods for each identified foundational therapist skill that can ultimately inform an intervention to improve therapist adoption of EBPs in community mental health agencies treating SED children and their families? Research Design and Methodology History of Delphi method. A modified Delphi method was utilized to build group consensus on core therapist skills related to successful EBP adoption. The Delphi method was first developed in the 1950s for a United States military project by Norman Dalkey of the RAND corporation (Dalkey & Helmer, 1969). Garson (2014) defines the Delphi method as a targeted, effective, group survey process able to (1) extract expert input on specific research topics that can assist with developing or validating content, and (2) utilize expert feedback in such a way that group consensus can occur (Lechowicz & Gazda, 1975). This process can be helpful for building consensus if there is a lack of agreement on a specific research topic or disagreement on existing research related to a specific topic (Bisson et al., 2010). Delphi features. Delphi methods have the following key features: (1) anonymity of participants, which allows for free expression of opinions without group pressure, (2) iteration, where views can be refined from round to round based upon the group’s work product, (3) controlled feedback, where participants can change or clarify their views, and (4) statistical aggregation of group response, which allows for quantitative analysis (Rowe & Wright, 1999). Typical designs for Delphi studies include three rounds of consensus building and then 37 discontinuing the process (Skulmoski, Hartman, & Krahn, 2007). Modified Delphi processes often incorporate data from literature reviews into analysis of qualitative survey rounds and offer participants additional opportunities to discuss consensus findings (Keeney et al., 2011). Triangulation of data with the literature review was utilized in this study (Burnard, 1991). Rationale for Delphi method. The Delphi method is well regarded in mental health research, particularly related to training initiatives (Dari, 2017; Skulmoski et al., 2007). A Delphi process has been used to improve professional training (deMello et al., 2013), to improve mental health systems of care, e.g., developing a set of performance indicators in public mental health care (Lauriks, de Wit, Buster, Arah, & Klazinga, 2014), and to develop the content of an intervention (Ross, Kelley, & Jorm., 2014). Similarly, Delphi methods have been linked to research leading to improvements in professional practice (Goodyear et al., 2015) and are highly congruent with community-engaged research aims (Lapsey et al., 2017). Delphi methods are also popular for prioritizing issues and conceptualizing or developing frameworks (Okoli & Pawlowski, 2004). Delphi studies are frequently used for consensus building in research where there is a lack of clear consensus on a topic of interest and are conducive to implementation studies (Bishop, Snowling, Thompson, & Greenhalgh, 2016). Research efforts to build consensus can help inform future interventions and guidelines for practice (Manuel et al., 2009) which are congruent with the aims of this study. Selection of Participants Procedure. Selecting participants is a key strategy of the Delphi method and one that involves requirements for being considered an expert, for example, having knowledge of the research issue and willingness to participate in the sharing of that expertise (Adler & Ziglio, 1996). Strong strategies for recruitment are emphasized as critical to ensuring the success of the 38 Delphi process (Keeney et al., 2011). The sample for this study was purposive as is common to Delphi studies (Skulmoski, Hartman, & Krahn, 2007). I recruited experts who could provide rich information that would allow for consensus building (Patton, 2015) and experts who were well informed and experienced with EBP implementation and therapist training (Etikan, Musa, & Alkassim, 2016). McKenna (1994) suggested approaches to recruitment and retainment that are built on personal touches and relationship forming. Communication processes with participants encompassed ways to engage them in their interests around EBP adoption and presented the study’s aims as supportive to their work. For example, I tailored each recruitment email to potential participants specifying that I was reaching out to them as an expert representative specifically for the EBP they represented. Participants received an average of three emails from this investigator prior to sending the link for the Round 1 survey. This was to assure the study aims and process were understood. Phone calls or Zoom meetings were offered when participants had additional questions about the study, their potential participation, or wanted to understand the research literature on this topic better. Prior to involving any participants in research, approval to conduct human subject research was sought through the Michigan State University Institutional Review Board process. This study received an exempt determination. To select experts for participation, I systematically generated a list of family focused EBPs. First, I completed searches through the California Evidence-Based Clearinghouse for Child Welfare (CEBC, 2020), Blueprints (2020), and the National Institute on Drug Abuse (NIDA, 2020) searching for models that provided treatment for SED symptoms within the family context for children ages 3-17. This search yielded over 100 matches. Next, I narrowed the list by only including those programs with the highest rankings. This corresponded to rankings of 39 “model plus” or “model” for Blueprints and ratings of “1” or “2” for CEBC. This process narrowed the list considerably. Then, I narrowed the list further by excluding those programs that did not provide both home and community-based services, as both are common treatment settings for community mental health service agencies (State of Michigan, 2020). I further narrowed the list by excluding programs that were primarily group treatment or that targeted mild symptomatology. Appendix A identifies 12 specific EBP models that met criteria for treating children with SED and their family systems. Included in this Appendix are descriptions of the models (CEBC, 2020). Two models listed separately on the clearinghouse searches, MST and MST-PSB, were collapsed into one during the participant recruitment phase as it became clear MST-PSB was a specialized branch of MST and not an entirely different model. This resulted in 11 EBP models in the final pool. Participant recruitment. EBP experts representing each model in Appendix A were recruited for participation in the study. I oversampled to avoid having too small a pool of participants. This meant recruiting with a goal of up to five experts representing each of the 11 models, or 55 total, with hopes to have at least 20-25 throughout the Delphi study. Recruitment efforts sought to include experts from all models. This was intentional to help assure findings could be generalized across many family focused EBP models. Expert criteria. Participants were required to meet both of the following two criteria: 1) hold an advanced degree in a mental health-related field, e.g., MA, MS, MSW, PhD, PsyD, MD, or DO, and 2) have trained or supervised therapists in the EBP model they represent within the past five years. In addition, they had to meet one of the following criteria to be considered an expert: (a) have developed the EBP model or been responsible for implementing ongoing 40 research or adaptations to the model, (b) have been responsible for ongoing fidelity monitoring or supervision of therapists performing the model, or (c) been employed as a master trainer or consultant to an EBP implementation. These criteria were established to assure experts could speak to therapist skills necessary for model adoption as well as have understanding in methods that support their skill acquisition. Criteria to confirm expert capacity was reflected in the invitation to join the Delphi study and validated through demographic reporting in Qualtrics. Recruitment efforts. I invited participants to take part in the study through email contact, with an initial ask for them to confirm their wish to participate via an email reply before sending the Qualtrics link for the Round 1 survey. To gain access to experts in models for which I did not already have contacts established, I utilized my research advisors and community partner to facilitate reaching out to additional experts as needed. This meant, for example, that my community partner facilitated an introduction to an EBP model contact, or an advisor informed a model developer of the existence of this study. Once interest in the study was expressed, all study communication came directly through me. Appendix B provides an example of a recruitment request. A desire to have a diverse group of representatives from each EBP model was also specified in the recruitment email. A participant tracking spreadsheet was used to track recruitment efforts. Each EBP was listed on the spreadsheet along with main contact information as stated in the CEBC reference page for each model. Sometimes these contacts were the model developer, in which a direct email requesting participation in the study was sent (see Appendix B). In other cases, contact information was limited to a general mailbox contact. For these, a more general email was sent to inquire about a contact person for study recruitment. Contact was made for each model to assure there was an equal opportunity to be informed of the study. Often the main contact for an EBP 41 was willing to share this study information with their colleagues after learning about the study’s aims, even if they could not participate themselves. Last, if contact information was findable, an informational email went out to the model developers for each EBP that did not have a model developer listed on the CEBC or already recruited for the study. This outreach was helpful as a few model developers returned communication and expressed an interest in the study, suggesting colleagues in their organization as good fits for expert criteria. Participant Reimbursement. Each participant was offered a $50 gift card for each round of study completion. Gift cards were sent via email through Amazon.com. A separate consent procedure for sharing email addresses with Amazon was completed as this was not included in the original study consent. Participants were offered a choice to share a postal address for a gift card to be mailed instead of electronic delivery. Sample size and power. There is a lack of agreement in the literature around expert sample sizes for Delphi study panels. Review of literature indicates that studies with fewer than 10 experts are rare, and many studies vary in the 10-100 range (Akins, Tolson, & Cole, 2005). A sample of 15 experts is suggested as adequate (Akins et al., 2005). Eleven family focused EBPs were identified for this Delphi study. As earlier stated, I intended to recruit up to five experts representing each model, yielding a sample size of up to 55. This was to account for not all experts recruited being willing to participate and the possibility for some attrition between rounds (Hardy et al., 2004). Therefore, my goal was to have up to 55 participants ensuring that at least 20-25 could be engaged throughout the course of the study. A total of 51 participants were recruited for Round 1. Informed consent. Informed consent procedures were completed electronically via Qualtrics (2005). Appendix C demonstrates the informed consent document embedded in the 42 Qualtrics survey. Participation was confidential except for voluntary participation in the third Delphi round conducted in a focus group format via Zoom. In addition, I assured participants of the voluntary nature of the study, their right to discontinue participation at any time, and how to appeal any concerns they may have with the integrity of the research process. This study was determined exempt through the Michigan State University Institutional Review Board. Consent processes and follow up emails to participants outlined the anticipated time commitment to each round of study and the plan for compensation. Procedures and Data Collection: Delphi Process In Appendix D I provide an overview of the entire Delphi process. This study consisted of three rounds of inquiry. An additional quantitative round for consensus building was anticipated, however was not needed due to significant consensus being in place after Round 2. I will explain each Delphi round in the subsequent sections, organized by procedures, measures, data collection, and data analysis. Delphi Round 1 Procedures. Round 1 of Delphi studies typically starts with open-ended questions to allow panel members an opportunity to respond to the topic of inquiry in a non-leading manner (Keeney et al., 2011). Survey questions in this round were broad and descriptive in nature and were intended to generate a brainstorming process (Dalkey & Helmer, 1969; Sekayi & Kennedy, 2017). In this round I administered an online questionnaire that consisted of three open-ended questions directly related to this study’s research questions. I asked participants to list and briefly describe their responses to each of the questions. Appendix E references the entire Round 1 survey and directions. 43 Measures: Trevelyan and Robinson (2015) suggest developing questions for Round 1 that are related to the study’s research questions and that offer flexibility on obtaining data that can inform subsequent consensus statements. I included two questions specific to therapist skills and competencies, and one question related to general therapist adoption challenges. The following questions were asked to Round 1 participants: Question 1: Considering your own EBP model and other EBPs you are familiar with, what are the key skills you believe therapists need to learn to be successful working with families with children experiencing serious emotional disturbances? Question 2: Finish this sentence: When I think about therapists who were successful learning my EBP, I recall that they came into my training with the following foundational skills or competencies. Question 3: Separate from the skills and competencies listed above, what are the challenges you see to therapist adoption of EBPs? Participants were asked to list and briefly describe their responses in an open-ended fashion, consistent with this type of qualitative inquiry (Holey et al., 2007; Skulmoski et al., 2007). Data Collection. I sent study participants an email link to Qualtrics, a secure online survey platform (Qualtrics, 2005) to complete the consent form, basic demographic information, and responses to survey questions. Demographic indicators included age, gender identification, race/ethnicity, mental health discipline, years of clinical experience, years of working with specific EBP, identified professional title (e.g., model developer, trainer, consultant, supervisor, etc.) and geographic area of EBP implementation (nationally). See Appendix F for specific demographic prompts. Participants were given a timeframe in which to complete Round 1. Email 44 reminders with prompts such as resending the survey link were utilized to remind participants of outstanding surveys. All responses were tracked when returned through Qualtrics. Data Analysis. Round 1 is intended to produce large amounts of information available for qualitative analysis (Keeney et al., 2011). Participant responses to the three questions were downloaded from Qualtrics (2005) to MAXQDA 2020 (VERBI Software, 2019). These responses were analyzed for content following the 14-step process described by Burnard (1991), which supports groupings of themes that inform subsequent round development. The coding team consisted of this investigator, two study advisors, and feedback from the community partner. Table 3.1 highlights each stage of the data analysis process, along with the description of the activity that took place and persons who completed the activity. Table 3.1 Data Analysis Steps and Description – Round 1 Stage Description of Activity Persons Responsible 1 Reviewed research literature pertinent to therapist skillsets Primary Investigator Read all participant responses Downloaded de-identified transcripts into MAXQDA 2 Re-read participant responses to immerse in data Primary Investigator Captured notes on general observations Highlighted general themes 3 Captured all possible therapist skill statements (open coding) Primary Investigator Generated initial headings Generated initial categories 4 Exported statements to Excel for ease in grouping Primary Investigator Grouped statements by categories Added notes next to statement for sub-category consideration Reviewed for duplicate statements 5 Finalized spreadsheet with initial coding scheme to prepare for Primary Investigator team review 6 Invited two independent coders to review methods and Primary Investigator organization of potential skill headings and categories Coding Team Facilitated discussion and agreement on list of skill statements Generated possible sub-categories 7 Reviewed skillset statements to determine fit with categories Primary Investigator Went back to transcripts as needed to assure fit for category 45 Table 3.1 (cont’d) 8 Invited independent coders to review skill statements by Primary Investigator category Coding Team Facilitated discussion on categories and revised them as agreed on by consensus 9 Organized statements by primary category Primary Investigator Invited coding team to review suggested sub-categories Coding Team Facilitated discussion on sub-categories and revised them as agreed upon by consensus 10 Re-checked skill statements against categories and sub- Primary Investigator categories 11 Prepared final skill statements for Round 2 survey Primary Investigator Triangulation with research literature completed Community Partner 12 Sent draft survey out to community partner and two graduate Primary Investigator students for feedback on logic and flow of survey Community Partner Made changes to survey based on feedback Graduate Students 13 Sent final list of statements and documentation of coding Primary Investigator process steps to coding team for final agreement and to assure Coding Team accuracy of steps taken in this process 14 Sent Round 2 survey Primary Investigator Securely stored original transcripts Coding Team Prepared findings for results section of dissertation study Detailed description of thematic coding processes. Stage 1 includes review of the literature. As reviewed in Chapter 2, there were eight therapist skills referenced in the literature. These include: 1) therapist adaptability, 2) client and family engagement, 3) trauma-informed concepts, 4) systemic case conceptualization, 5) reframing or changing the view, 6) behavioral strategies that support caregivers, 7) self-reflection, and 8) cultural competence. In addition, this stage provided the first review of participant responses to assure completeness. As needed, any information on a survey that was not understood was resolved by contacting the participant directly. This step supported the trustworthiness of data. In Stage 2, participant responses were re-read with the goal of immersing the researcher in the data. This provided an opportunity to thoroughly read the data and capture notes on general observations and themes. Similarly, Stage 3 included the opportunity to read through 46 participant responses a second time, this time with the intention to capture all statements related to skills. Any statements related to skills were highlighted and maintained directly in participant’s own words. No duplicate statements were removed at this time. A count of statements to date was captured to include in the Results. Then, a frequency by key word was run in MAXQDA (VERBI Software, 2019) to utilize in subsequent stages for triangulation of data with skills captured in the literature (Burnard, 1991). In Stage 4, statements were exported to Excel for ease in organizing and grouping. Statements with similar headings were grouped together. As groupings were organized, notes were added next to the statements that reflected possible sub-categories. In addition, data were reviewed for duplicate statements. Stage 5 included finalizing the spreadsheet with the initial coding scheme. This included organizing statements in the initial coding scheme with suggested sub-categories. This was prepared for review with the coding team. Up until this point, data analysis was my primary responsibility. Stage 6 introduced the coding team to the analysis process. As a team, data analysis methods, potential skill headings and sub-headings were reviewed. The coding team processed language differences noted in describing skillsets, likely due to the diversity in participants’ clinical training and experience. For example, were “reframing” and “engaging families in new perspectives” similar? The coding team also processed the large number of statements generated in open coding. The team noticed that some skill statements were reflective of interpersonal therapist attributes more so than a specific skill, e.g., “being caring” or “a creative thinker.” Group consensus was utilized to make adjustments to the headings and categories. In Stage 7, I reviewed the skillset statements to determine fit with agreed upon headings and categories, going back to the transcripts as needed for context to assure the fit with the 47 heading or category. In Stage 8, the coding team rejoined the process to review statement and category fit. The coding team worked toward group consensus on any statements that lacked agreement on a category. The coding team continued to review skill statements through Stage 9, moving statements within broad categories into defined categories and sub-headings. Stage 10 offered the first opportunity for the skill statements to be organized formally by primary heading, categories, and sub-categories. The final categorical scheme will be discussed in Chapter 4. This stage was important toward beginning to conceptualize an organization of the statements that would work well for the Round 2 survey, identify a framework for the skillset, and continue to engage participants in the Delphi study process. In Stage 11, skill statements were re-checked against the coding scheme in preparation for the Round 2 survey development. Additionally in this stage, the final categories and coding scheme generated were reviewed with the skillsets found in the Stage 1 literature review. As noted earlier, one additional method of triangulating the coding scheme was to review coding generated from participant feedback with the research literature, allowing the opportunity to include any items in the Round 2 survey that did not emerge from participants but were present in the literature (Burnard, 1991). This process was completed using the frequencies generated back in Stage 3. Final skill statements were prepared and organized in Stage 12. In this stage a draft survey was sent to my community partner and two graduate students for feedback on categories, process, and flow. I also asked them to provide feedback on time for survey completion since there were many statements to rank. This feedback provided an opportunity for improvement before sending out the Round 2 survey. Final survey development took place in Stage 13, with the final survey sent to the coding team for one last review. The team reached agreement that no 48 additional skills from the literature needed to be added to the survey. In addition, the documentation of the coding process steps was sent to the team for final agreement and accuracy. In Step 14, the Round 2 survey was sent. I stored the original transcripts and started preparing findings for the results section. Trustworthiness. Trustworthiness is supported by findings that are the result of rigorous methods and describe participants’ experiences with accuracy (Leavy, 2017). Trustworthiness is further demonstrated by the criteria of credibility, dependability, confirmability, and transferability (Lincoln & Guba, 1985). First, to ensure trustworthy findings, we followed research-based steps that utilized a coding team for consensus and reflective process (Burnard, 1991). Second, expert feedback from Round 1 was triangulated with the review of the literature. Skills represented in the research literature as important to improving EBP adoption, i.e., therapist adaptability, family engagement, trauma-informed concepts, systemic case conceptualization, reframing (changing the view), behavioral strategies to support caregivers, self-reflection, and cultural competence were all present in the Round 1 statements generated. However, we let the coding scheme emerge through content analysis steps which led us to a different organization of skillsets than suggested in the research literature, i.e., many of the skills named in the literature became sub-categories of the coding scheme. Third, Round 1 categories and headings were critically reflected on with the community research partner to assure the coding scheme aligned with perceptions they had from experiences in community practice and therapist training needs. Critical reflection is a process commonly utilized in community engaged research practices (Owen, 2016) and is defined as a process of 49 analyzing or questioning one’s own experiences with another source that can offer context or additional knowledge (Jacoby, 2014). Finally, the process and content of the content analysis was documented in the MAXQDA (VERBI Software, 2019) software program. This, along with de-identified Excel spreadsheets, serve as an audit trail of the thematic analysis and subsequent statement development. Keeping original transcripts, coded data, and recorded process steps is supported by Burnard (1991) for reliable content analysis. Round 1 quantitative analysis. Participant surveys were downloaded into SPSS (Pallant, 2011) to summarize demographic information. Descriptive statistics were run and included participant means for categories such as age and years of experience, and frequencies for categories such as discipline and professional role. Ranges and percentages were utilized in reporting demographic data to further protect confidentiality of participants. Delphi Round 2 Procedures. The goal of Delphi Round 2 was to determine consensus on the core foundational skills therapists need to acquire to improve adoption of family focused EBPs in community mental health agencies. To answer this research question, participants were asked to rank statements in the Round 2 survey, which were derived from the content analysis process explained in Round 1 (Burnard, 1991; Keeney et al., 2011). Recommendations for Round 2 included finding a balance between accurate coding schemes that do not collapse categories prematurely and managing a survey length that does not discourage participants (Keeney et al., 2011; Trevelyan & Robinson, 2015). Measures. The Round 2 questionnaire consisted of 222 statements over six categories (32 total sub-categories). Participants were asked to rate the level of importance for each 50 questionnaire item on a 7-point Likert scale ranging from “strongly disagree” to “strongly agree.” The rationale for a 7-point scale is supported by Finstad (2010) for scale sensitivity, due to this study having less than 100 participants. Further, a range of Likert response categories are recommended to be between four and seven for Delphi studies (Trevelyan & Robinson, 2015). In addition, a text box was included within each sub-category so participants could include any comments they had about the statement or the ranking they assigned. Appendix G includes directions for this round and a sample of the survey. Data collection. Similar to Round 1, I emailed participants a secure link to Qualtrics to complete this survey. Directions specified that Round 2 looks different than Round 1 and that the responses requested this time are for rankings of importance on specific statements (Keeney et al., 2011). To reduce the risk of missing data, I prompted the surveys to not allow advancement to the next question without an answer. Participants were given an option to enter a neutral entry if there were statements that they were uncertain about endorsing. They also had an option to comment on statements throughout the survey in each of the sub-sections. Due to the large number of statements generated for the Round 2 survey, the coding team decided on some additional safeguards to assure consistency in data collection and a lack of attrition through this round of inquiry. First, participants were made aware of the potential additional time that may be needed to complete Round 2 through instructions with the survey link. To estimate the survey time, I gave a pilot survey to my community partner and two fellow graduate students to gain an average time for completion, which was around 45 minutes. Second, the questionnaire was structured to break down categories into sections to decrease long blocks of text. Statements were ranked at the sub-category and individual statement level. For example, if the section was “Observing Family Behavior,” participants would rank the statement 51 “Therapists need to be keen observers of the behavior and communication of their clients” as well as, “In general, skills in observing family behavior are important.” Each categorical area provided participants with the number of statements in that section in order to help them see their progress. The use of headings in lengthy surveys can help organize consensus statements (Keeney et al., 2011). Lastly, to mitigate the risk of ordering effects, I randomized the categories of responses through Qualtrics so that the same skill categories were not always first or last. Data analysis. I analyzed data in Round 2 quantitatively using descriptive statistics, i.e., central tendency (median) and interquartile ranges (IQR) (Linstone & Turoff, 1975). First, each statement and its corresponding Likert scale responses were uploaded to SPSS (Pallant, 2011). There were no missing data. Next, descriptive statistics were run for each statement. Data were reviewed to find those skill statements that had a median of six or seven, and an IQR of less than two. The median is a widely used statistic in Delphi studies as a group response measure (Linstone & Turoff, 1975). IQR is a second measure of consensus which provides information about the variability of responses without being affected by extreme scores, therefore supporting where group consensus is occurring (Fish & Busby, 1996; Nachimas & Nachimas, 1981). IQR range is suggested to be less than 2 to demonstrate consensus strength (Kittell-Limerick, 2005). Last, the median and standard deviation of each of the statements in Round 2 were reviewed for spread of response, along with comments entered on the survey that provided reasons for ranking. This summary was prepared in an Excel spreadsheet for analysis by the coding team and reviewed for accuracy. Research literature suggests the decision to have an additional round of consensus to further converge data be planned for at the outset of the study and guided by the responses from prior rounds at the researcher’s discretion (Keeney et al., 2011). The goal of complete consensus 52 is not always achievable or even necessary. In fact, most often if the majority of indicators have consensus additional rounds of consensus building are not indicated (Keeney et al., 2011). Round 3 was prepared for in the study design, however it was decided by the research team to not be utilized for three main reasons. First, in the review of the central tendency and spread for the statements (n=47) that did not have consensus, a majority represented a range of participant rankings of “1” or “2” out of “7.” This indicated a likelihood that an additional round was not likely to converge further consensus and that there were participants who felt strongly that these statements did not fit with core therapist skills. That is, the data showed strong support for these as non-consensus items. Second, thematic analysis of comments added to these statements did not yield any patterns, such as misunderstanding a statement, that would support an additional round (Braun & Clarke, 2012). Finally, the benefit of another round of inquiry was weighed against the risk for attrition, given Round 2 was longer to complete than originally expected (Hardy et al., 2004; Keeney et al., 2011). Since there was only one quantitative round, calculations for stability between rounds was not necessary (Holey et al., 2007; Keeney et al., 2011). Delphi Round 3 Procedure. Follow up qualitative rounds of inquiry are common in modified Delphi studies and suitable for Delphi studies with a smaller number of participants (Keeney et al., 2011). In Round 3, I initiated a series of small focus groups that aimed to 1) review consensus statements, and 2) open discussion among experts on the training methods that can best support adoption of the core therapist skillset identified, to improve adoption of EBPs for SED children. This Delphi round primarily answered the third research questions being investigated in this study, i.e., What are the corresponding training methods for each identified foundational 53 therapist skill that can ultimately inform an intervention to improve therapist adoption of EBPs in community mental health agencies treating SED children and their families? Measure. A semi-structured interview guide was utilized for the focus group discussions. The guide was developed to accompany a visual table of the consensus skill areas shared on screen during the groups. Questions from the guide included asking participants to share about their interest in the study, and to expand on methods or strategies they would use to help therapists achieve the identified skillsets. The interview guide also included a prompt for discussing items that did not reach consensus, time permitting. Each prompt for discussion included a reminder of the study’s aims and research questions. The full interview guide is referenced in Appendix H. Data collection. Participants were offered eight different options to sign up for participation in Round 3. Focus groups were scheduled for 60 minutes. Schedule confirmations were sent via a Zoom scheduling link. Informed consent specific for this round was included in the scheduling link and each member was asked to affirm their consent upon joining the Zoom link and participating in the focus group. While we anticipated perhaps less than half of the participants would choose to continue with the focus groups, we had significant interest in this round with 40 participants committing to a focus group. See Appendix I for Round 3 consent. Discussion was facilitated by sharing a screen of the consensus items by category. See Appendix J for consensus categories shared with participants. Participants were acknowledged for their time and commitment to the Delphi process and asked to share briefly about their reasons for wanting to participate. Next, a semi-structured interview process was utilized to facilitate discussion around the training methods that can support the therapist skillset identified. Open ended questions served as prompts to start discussion among group members. Groups were 54 co-facilitated by me and an advisor to help assure presence in the process and to support managing the discussion within the timeframe allotted. Groups ranged in size from 3 to 6 participants (M = 4.5). Data analysis. The focus groups were conducted via video conference (Zoom), recorded, and then transcribed to assist with accuracy for data analysis. Content was analyzed thematically following the steps identified from Burnard (1991). Table 3.2 highlights each stage of the coding process. Additional details on stages are included below. Table 3.2 Data Analysis Steps and Description – Round 3 Stage Description of Activity Persons Responsible 1 Facilitators debriefed after each focus group, making Primary Investigator brief notes on the interviews and emerging themes Group Co-Facilitator Prepared group transcripts. Downloaded full interviews from recorded Zoom session into MS Word Read all participant responses 2 Re-read participant responses to immerse in data. Primary Investigator Memo function was used for notes. General themes highlighted 3 Imported MS Word transcripts to MAXQDA for open Primary Investigator coding 4 Statements exported to Excel for organizing content Primary Investigator 5 Finalized spreadsheet with initial coding scheme to Primary Investigator prepare for team review 6 Invited two independent coders to review methods and Primary Investigator organization of potential skill headings and categories. Coding Team Coding team discussed findings to date Adjustments were made based on group consensus 7 Reviewed training method statements to determine Primary Investigator primary categories Coding Team 8 Removed duplicate statements that held the same Primary Investigator meaning, e.g., “Role-play” and “engage group in role- play” 9 Reviewed sub-headings as a coding team. Results table Primary Investigator envisioned to match skill categories with training Coding Team methods identified in this round 10 Re-checked training methods against coding scheme Primary Investigator 55 Table 3.2 (cont’d) 11 Final method statements were paired with skillset Primary Investigator statements in preparation for results and discussion 12 Triangulation of findings with research literature on Primary Investigator Adult Learning Principles Coding Team Critical reflection with community partner Community Partner 13 Sent final coding scheme and documentation of coding Primary Investigator process steps to coding team for final agreement and to Coding Team assure accuracy of steps taken in this process 14 Securely stored original transcripts Primary Investigator Prepared findings for Results section of dissertation Coding Team study In Stage 1, facilitators spent time after each group debriefing with each other. For the first group, this time was utilized to adjust the interview guide and prioritize timing after seeing how the group responded to the questions. For example, the visual aid for consensus statements was modified to make it easier for participants to scan on the screen share. Facilitators took brief notes on emerging themes. In addition, focus group transcripts were downloaded from Zoom and read through to assure all data was recorded. Stage 2 included re-reading the group transcripts to immerse myself in the data. I highlighted general themes and utilized the memo function for notes on general themes. Next, in Stage 3, transcripts were uploaded to MAXQDA (VERBI Software, 2019) to initiate open coding and look for all statements that reflected training methods. These statements were organized under three headings. Then, in Stage 4, statements were transferred to Excel for ease in organization and to prepare for higher order grouping. To facilitate this, I added a column next to the statements to summarize the content and review the context of statements individually. This was an important process step as statements were complex and multi-dimensional. Breaking them down was critical to assuring a reliable coding scheme (Burnard, 1991). An additional re- read was then completed to identify possible sub-categories. Transcripts were reviewed to assure 56 accuracy of capturing the context of statements. In Stage 5, the coding spreadsheet was prepared for review with the coding team, organized by heading and suggested categories. In Stage 6, the coding team convened to review methods and coding organization. We started with a review of the research questions and interview guide to assure statements were being captured with the lens of the research question. This was important since participants had abundant knowledge about implementation processes and training in general, therefore statements generated were rich in context. The coding team decided on adjustments to the organization of the main headings based on group consensus. In Stage 7, the coding team continued to work with statements to clarify the training methods and to come to agreement on a higher-order coding for statements. Then, in Stage 8, we agreed to remove duplicate statements that held the same meaning, e.g., “Role-play” and “engage group in role-play.” In Stage 9, we reviewed sub-categories as a team and came to consensus on coding structure. We brainstormed ways to present the data and findings that would be most accessible for reference, since we were tying together the skillset findings from Round 2 with the training methods identified in Round 3. Next, in Stage 10, I re-checked the training methods against the coding scheme and original transcripts. This step was important to assure that methods were captured and that they aligned with the corresponding skills that participants intended. This led to Stage 11, in which the final coding sheet was prepared. Then in Stage 12, the coding team discussed that many of the training methods suggested overlapped multiple skill categories. For example, “role-play” is used to teach therapists intervention strategies as well as engagement approaches. It is also a tool helpful for self- reflection. Therefore, we came to agreement that there was high overlap in training methods and 57 that certain methods with high saturation would be repeated in the results section covering several different therapist skills. To check this and incorporate a way to triangulate our findings, we did two things. First, I reviewed the extant literature on adult learning theory and adult learning principles (Bryan et al., 2009), finding that many of the training strategies enlisted by participants lined up theoretically with this framework. Next, I engaged my community partner in reflection around how these training methods might translate to practice. This was important to consider additional ways to best organize these results so that they were accessible. Reflective discussions are encouraged in academic-community partnerships to understand potential pathways to translate research findings (Balazs & Morello-Frosch, 2013). In Stage 13, I sent the final coding scheme and documentation of coding process steps to the coding team for final agreement and to assure accuracy of steps taken in this process. Finally, in Stage 14, I securely stored the original transcripts and finished preparing the findings for the Results section. Summary Mixed methods Delphi studies are highly regarded for their flexibility and potential to impact practical, timely research inquiries (Jorm, 2015; Keeney, 2011; Kennedy, 2004; Sandelowski, 2000). In this chapter I acknowledged my positionality as a researcher with shared culture as an EBP practitioner, supervisor, researcher, and trainer. I then described a robust Delphi process, used along with an active academic community partnership, to provide a methodologically rich pathway to determine a foundational therapist skillset and the corresponding training methods that can impact EBP adoption in community health settings. Trustworthiness of qualitative rounds of inquiry in the Delphi process were supported by adherence to established coding stages (Burnard, 1991), methods of triangulation with the 58 research literature and community partner feedback (Balazs & Morello-Frosch, 2013; Burnard, 1991), and the use of a coding team for agreement on coding schemes. The next chapter will move into the results of the Delphi process, which includes understanding the participants and their reasons for participating, bridging Round 1 findings to a Round 2 survey tool, consensus skillsets, and finally, a match of skillset and training methods. 59 CHAPTER 4: ANALYSIS AND RESULTS In this chapter I review the data that support the discovery of a therapist skillset and training methods identified through a three round Delphi process. Improving therapist training is an important aim toward supporting EBP adoptions for SED youth (SAMHSA, 2020). Delphi studies are methodologically effective for finding agreement on research topics where there is a lack of agreement or existing research (Bisson et al, 2010). While research has revealed some promising clues for important therapist skills (e.g., Horigian et al., 2016; Karlin & Cross, 2014), a consensus skillset remains unidentified. Findings from this Delphi process offer consensus on a foundational therapist skillset and corresponding training methods that hold promise for promoting EBP adoption in community health settings (Frank, Becker-Haimes, & Kendall, 2020). Research Questions As reviewed in Chapter 3, the research questions guiding this mixed methods modified Delphi study included: 1. What are the foundational therapist skills experts identify as important to improving therapist adoption of family focused EBPs targeting SED symptoms in children? 2. What is the consensus among panel experts on the core foundational skills therapists need to acquire to improve adoption of family focused EBPs in community mental health agencies? 3. What are the corresponding training methods for each identified foundational therapist skill that can ultimately inform an intervention to improve therapist adoption of EBPs in community mental health agencies treating SED children and their families? 60 Data Collection This Delphi study consisted of three rounds of inquiry. Recruitment and retention of participants, along with descriptive statistics of the Delphi expert panel are included in the findings as they are critical to the Delphi method (Keeney et al., 2011). Round 1 was a qualitative inquiry that asked participants to describe the therapist skillset necessary to work with families and children with SED symptomatology. Findings were summarized into statements that became the Round 2 survey. Round 2 was a quantitative round in which participants ranked their level of agreement with each therapist skillset statement. For example, a sample statement was “Therapists should have the ability to identify critical relational processes in the moment during sessions.” Participants ranked agreement on a 7-point Likert scale, with a ranking of 1 = strongly disagree through 7 = strongly agree. Quantitative analysis in Round 2 utilized median and interquartile range (IQR) statistics for analysis. Items that met consensus were indicated by a median > 6 and an IQR of < 2 (Keeney et al., 2011; Linstone & Turoff, 1975). Finally, Round 3 was a qualitative round facilitated in a focus group format that led expert participants to discuss training methods able to support the consensus skillset areas. Rounds 1 and 3 were coded thematically utilizing Burnard’s (1991)14 step process, as detailed in Chapter 3. Results Description of Delphi Expert Panel A total of 51 panelists were recruited for the Delphi study. The panel included 40 women and 11 men, with no other gender identifications reported. Participants mostly identified as Caucasian/White (62.5%, females, 54.5% males), followed by Hispanic (20% females, 18.2% males) and Black or African American (12.5% females, 9.1% males). One participant identified as Middle Eastern North African (MENA) and another as Irish/German/English. All the panelists 61 had advanced degrees, which was an inclusion criterion for this study. The majority of female participants had a master’s degree (75%), while most of the male participants held a doctorate degree (81.8%). Social work was the predominant mental health discipline for females (55%) and psychology for male participants (72.7%). There were four licensed Marriage and Family Therapy participants who were all females. Three participants identified no longer holding a clinical license. Six participants were model developers (12%), equally represented by males and females. Master trainers were the second largest group, representing 30% of female participants and 18.2% of male participants. Six participants identified as researchers, all of which were female. Table 4.1 identifies demographic information for participants, sorted by gender. Table 4.1 Descriptive Statistics for Participant Demographics Female (N=40) Male (N=11) % % Race/Ethnicity African American 12.5 9.1 Asian 2.5 9.1 Hispanic 20.0 18.2 White 62.5 54.5 Other 2.5 9.1 Education Masters 75.0 18.2 Doctorate 25.0 81.8 Clinical Licensure Counseling 5.0 9.1 Psychology 25.0 72.7 Social Work 55.0 9.1 Marriage and Family 10.0 0.0 Therapy Other (no longer 5.0 9.1 licensed) EBP Role Identification Model Developer 7.5 27.3 Master Trainer 30.0 18.2 62 Table 4.1 (cont’d) Consultant 15.0 18.2 Implementation Coord. 12.5 9.1 Clinician/Supervisor 20.0 27.3 Researcher/Other 15.0 0.0 Description of Participant Expertise Participants’ expertise is important in a Delphi study (Keeney et al., 2011). Although years of practice was not a specific expert criterion, this study did seek participants with experience implementing their particular EBP in community settings, as stated in the recruitment emails. Participants had approximately 22 years of clinical experience overall (SD = 9.96) and an average of almost 17 years of experience (SD = 9.71) with their specific EBP. Table 4.2 provides the means and ranges for years of experience. Table 4.2 Participant Years of Clinical Experience Mean Min Max Standard Deviation Years of clinical 21.86 5 49 9.962 experience (total) Years of experience 16.63 5 47 9.710 with their EBP Participant age. Participant age was collected in ranges. Most participants identified as between 35 and 64 years old (82%). Ten percent of participants were age 65 or older. Table 4.3 shows the complete participant breakdown for age. 63 Table 4.3 Participant Age Categories Age Category Frequency Percent Valid Percent Cumulative Percent 25 - 34 4 7.8 7.8 7.8 35 - 44 16 31.4 31.4 39.2 45 - 54 15 29.4 29.4 68.6 55 - 64 11 21.6 21.6 90.2 65 - 74 3 5.9 5.9 96.1 75 - 84 2 3.9 3.9 100.0 Total 51 100.0 100.0 EBP representation. There was a total of 51 panel members representing 11 different EBPs who participated in this study. Part of the recruitment design was to recruit up to five representatives from each of the selected family therapy models. This was purposeful to ensure that findings could be generalized as appropriate to more than one specific EBP model. Table 4.4 includes the percentage of representatives that were recruited for this study. One EBP, Trauma-Focused Cognitive Behavioral Therapy, had 6 participants or 120% participation, due to two final commitments for participation being made at the same time. Oversampling was intentional to help account for any attrition that might have occurred during the process (Keeney et al., 2011). Table 4.4 Participant Representation by Evidence Based Practice % of Evidence-Based Practice Name and Abbreviation N Sample Adolescent-Focused Family Behavior Therapy (AFBT) 4 7.8 Attachment-Based Family Therapy (ABFT) 4 7.8 Brief Strategic Family Therapy (BSFT) 4 7.8 Child-Parent Psychotherapy (CPP) 3 5.9 64 Table 4.4 (cont’d) Functional Family Therapy (FFT) 5 9.8 GenerationPMTO (GenPMTO) 5 9.8 Multidimensional Family Therapy (MDFT) 5 9.8 Multisystemic Therapy (MST and MST-PSB) 5 9.8 Parent-Child Interaction Therapy (PCIT) 5 9.8 Trauma-Focused Cognitive Behavioral Therapy (TF-CBT) 6 11.8 Triple P Positive Parenting Program 5 9.8 Total 51 100 Implementation locations. Experts participating in the study represented different implementation sites geographically throughout the United States. Table 4.5 highlights the areas where participants have supported EBP implementation in the United States. Table 4.5 Area of the United States Represented by Participants Implementations completed by Geographic Area participant in the of United States region (n=) Northeast 30 Midwest 32 Southeast 26 West Coast 26 Southwest 20 Recruitment and Retainment The importance of recruiting and maintaining participants throughout the Delphi study is critical to the process of consensus building (Keeney et al., 2011). To punctuate the importance, Green et al., (1999) refers to the formation of the panel as the “lynchpin of the [Delphi] method” (p. 200). This study anticipated up to 55 participants. I was able to recruit and maintain 51 (93%) through Round 1. Of the 51 that moved on to Round 2, 47 (92%) were retained through that 65 Round. Round 3 was a voluntary process in which we estimated 20 participants would complete. Instead, 40 participants scheduled themselves to participate in this Round. Thirty-six of the 40 (90%) completed this focus group round. Reasons for Participating The interview guide for Round 3 included a prompt that explored reasons for participation. Participant responses included the following categories: 1) advancing EBP implementation, 2) improving therapist training, 3) having interest or voice in the research topic, 4) opportunities to give back/share their experiences, and 5) supporting research in general. Table 4.6 provides example statements from participants: Table 4.6 Reasons for Participating in Delphi Study Reason for Representative Participant Quote Participating Category Advancing EBPs “What really attracted me to your study is just that it looks like it's something that can advance the field. Been involved with evidence-based practice for 20 years now. And it goes in fits and starts, but I'd love to see it jump ahead.” Improving Therapist “My motivation for participating is a lot of time [is spent] training Training therapists, reviewing video recordings of their sessions, preparing trainings that are really focused on clinical skills. And I find myself that it's not always effective, what I do? I'm curious to figure out what, you know or our kind of wanting to spend some time thinking about it.” Interest in Topic “And kind of I'm interested in finding out what you learn and having a voice at the table for what we think and see as helpful as kind of core skills for therapists to have. So happy to be a part of it.” Opportunity to Give “And I think what has really brought me to coming here today I Back or Share think is just being able to share what we've learned and being able Experience to support like you said [participant name omitted] clinicians coming into the workforce and being prepared to provide community mental health services. And really showing what we've learned to be really effective in helping support children and families.” 66 Table 4.6 (cont’d) Supporting Research “So partly, I wanted to participate to contribute to what's been a long process and really inspired process that you would take this up as part of your doctoral work is so cool.” Results Round 1: Identifying a Therapist Skillset The goal of Round 1 was to obtain qualitative feedback to answer the first research question: What are the foundational therapist skills experts identify as important to improving therapist adoption of family focused EBPs targeting SED symptoms in children? The first round of coding yielded 896 statements related to therapist skills and competencies with no duplicate statements removed. The next round of coding excluded therapist attributes as they were found to not answer the research question related to therapist skills. For example, participant statements such as “being relentless” or “team-oriented” were excluded because they were not descriptors that were able to be operationalized into skills. This yielded 371 statements. Group consensus process organized the 371 statements into eight higher order categories: 1) general therapeutic skills, 2) reflexivity, 3) assessment practices, 4) trauma informed care, 5) diversity and cultural competence, 6) therapeutic leadership, 7) adaptability, and 8) engagement. Some statements fit more than one category, e.g., “assessing family culture” fit in assessment practices and diversity/cultural competence. Statements were reviewed for duplication, which resulted in an additional 149 statements being excluded. This left 222 statements for the Round 2 survey. Triangulation of skill statements with literature and final review of the coding team resulted in no new skills added to the Round 2 survey. This process is depicted in a flow diagram in Figure 4.1. 67 Figure 4.1 Flow Chart for Round 1 Results 68 Round 1 yielded a total of 222 statements describing foundational therapist skills that experts identified as important to improving therapist adoption of family focused EBPs targeting SED symptoms in children treated in community health settings. These statements were grouped together into six categories: Assessment, Family Engagement, Intervention, Reflexivity, Adaptability, and Trauma Informed Care. The six categories are described as follows. Assessment. There were 41 statements in this category. Statements in this category described therapist skills related to appraisal and evaluation. Specifically, this included actions such as observing family interactions, history, behaviors, and culture. This category also included a therapists’ ability to conduct structured assessment processes for areas like safety and risk. It also included conceptualizing treatment from a systemic view. This category was organized in the following seven sub-categories in preparation for the Round 2 survey: 1) observe family behavior, 2) assess family interactions, 3) assess family culture, 4) assess family history, 5) conceptualize systemically, 6) complete behavioral assessment, and 7) conduct structured assessment. Family Engagement. There were 43 statements in this category. Statements in this category described therapist ability to connect with families for treatment through activities such as building a bond, facilitating goals, enhancing motivation to participate, and use of empathy. In addition, this category included the therapist’s ability to engage all family members in the treatment process. This category was organized in the following five sub-categories in preparation for the Round 2 survey: 1) establish a therapeutic bond, 2) build an empathic response, 3) engage families in treatment, 4) facilitate agreement on family goals, and 5) enhance client motivation for engaging in treatment. 69 Intervention. This was the largest category with 100 statements. Statements in this category described therapist skills related to either planning, conducting, or maintaining active treatment. In addition, specific treatment skills such as cognitive behavior therapy, family therapy, case management, and parenting support are included in this category. This category was organized in the following 14 sub-categories in preparation for the Round 2 survey: 1) skills in treatment planning, 2) process-focused treatment skills, 3) conflict management skills, 4) session management skills, 5) therapist organizational skills, 6) maintaining treatment alliance, 7) skills in reframing, 8) skills in facilitating change, 9) skills in working with emotion, 10) basic skills in cognitive behavioral approaches to treatment, 11) basic skills in family therapy, 12) parenting approaches, 13) active teaching skills, and 14) case management skills. Reflexivity. There were 23 statements in this category. Statements in this category described therapist skill in practicing reflectively and reflexively, i.e., their ability to be aware of how their work impacts them and how to manage their own reactions to their work. More specifically, this category includes skill in perspective taking and understanding of how their own social location and culture impacts practice. Also included in this category is therapist ability to understand the resistance they may have to supervision and to their own clinical growth. This category was organized in the following three sub-categories in preparation for the Round 2 survey: 1) skills in reflective practice, 2) awareness of self of the therapist issues, and 3) skills in managing own self-care. Therapist Adaptability. There were nine statements in this category. Statements in this category largely reflected adjusting or adapting therapeutic approaches in interest of engaging and maintaining families in treatment. More specifically, this included therapist actions such as using assessment of family culture to tailor evidence-based practices or their own approach. This 70 category also included a therapist’s ability to be flexible and open to incorporating new skills or supervisor feedback. This category was organized in the following two sub-categories in preparation for the Round 2 survey: 1) tailoring evidence-based practices to families, and 2) therapist ability to be flexible. Trauma Informed Care. There were six statements in this category. Statements in this category described therapist skills related to focus on family experiences with trauma. This included actions such as talking about complex issues that create trauma reactions, e.g., neglect, abuse, oppression and helping clients learn how to regulate emotions. This category also included a therapist’s ability to remain grounded when working with family trauma. This category included one sub-category that summarized the skill statements: 1) skill in conducting trauma-informed practice. Table 4.7 provides an overview of each category organized by sub-category. A sample statement from each category is provided for reference. All skill statements will be shown in subsequent tables. Table 4.7 Therapist Skillset Categories Identified for Round 2 Survey Skillset Category Sub-Categories (n=32) Sample Skill Statements (n=6) Assessment 1. Observe family behavior “Assess the family based on their 2. Assess family interactions interactions.” 3. Assess family culture 4. Assess family history “Demonstrate assessment skills that 5. Conceptualize systemically are based in family systems theory. 6. Complete behavioral assessment 7. Conduct structured assessment 71 Table 4.7 (cont’d) Family 1. Establish a therapeutic bond “Relate to the concerns of all family Engagement 2. Build an empathic response members.” 3. Engage families in treatment 4. Facilitate agreement on family “Join systemically and strategically.” goals 5. Enhance client motivation for engaging in treatment Intervention 1. Skills in treatment planning “Coach parents in the moment on 2. Process-focused treatment skills handling parenting situations.” 3. Conflict management skills 4. Session management skills “Facilitate family restructuring when 5. Therapist organizational skills necessary.” 6. Maintaining treatment alliance 7. Skills in reframing (changing the “Engage family members in basic view) cognitive-behavioral coping skills.” 8. Skills in facilitating change 9. Skills in working with emotion 10. Basic skills in cognitive behavioral approaches to treatment 11. Basic skills in family therapy 12. Parenting approaches 13. Active teaching skills 14. Case management skills Therapist 1. Skills in reflective practice “Be aware of their biases.” Reflexivity 2. Awareness of self of the therapist issues “Hear supervision feedback and try 3. Skills in managing own self-care suggestions given.” Therapist 1. Tailoring evidence-based “Understand the difference between Adaptability practices to families tailoring and modifying an evidence- 2. Therapist ability to be flexible based practice.” “Adjust their therapeutic approach when needed. 72 Table 4.7 (cont’d) Trauma 1. Conducting trauma-informed “Ground themselves when working Informed practice with family trauma.” Care “Demonstrate skill in recognizing, naming, and working with families around the family's trauma experiences.” The product of Round 1 is the summary of statements in the form of a survey utilized for consensus building in Round 2. Table 4.8 summarizes these 222 statements, organized into six main categories and 32 sub-categories. Each of the 32 sub-categories are in bold and italics for reference. Table 4.8 includes the median score of the respondents along with the IQR, which will be discussed in the Round 2 Results. Results Round 2: Consensus and Non-Consensus Statements Round 2 of the Delphi process sought to answer: What is the consensus among panel experts on the core foundational skills therapists need to acquire to improve adoption of family focused EBPs in community mental health agencies? Consensus Statements The results from Round 2 demonstrated consensus on 175 of the 222 (79%) items that went out for ranking. Table 4.8 highlights the skills that reached consensus, reflected in bold. Statements are organized by main category and included the following number of consensus statements in each: Assessment (n=29), Engagement (n=34), Intervention (n=76), Adaptability (n=22), Reflexivity (n=9), and Trauma Informed Care (n=5). At the end of each subsection, participants ranked their agreement with the overall sub- category. These subcategory statements are italicized and shaded for reference in Table 4.8 and 73 start with the prompt, “In general…” All sub-categorical statements (n=32) reached consensus. Median and IQR statistical measures used for consensus are included in the table. Non-consensus Items A total of 47 of the 222 items (21%) did not reach consensus. Table 4.8 shows the items that did not meet consensus as non-bold. By category, non-consensus statements include the following number in each: Assessment (n=12), Engagement (n=9), Intervention (n=24), Adaptability (n=1), Reflexivity (n=0), and Trauma Informed Care (n=1). Items in this category had an IQR of 2 or higher, indicating a more statistically significant spread of responses and therefore less agreement on the skillset statement. Table 4.8 All Round 1 Statements and Median and IQR Ratings by Participants Skillset Statements Median IQR Assessment Skills (n=41) Be keen observers of the behavior and communications of their 1 clients. 7 0 Have the ability to identify critical relational processes in the 2 moment during sessions. 7 1 Be able to observe how family members interact and use these observations to identify areas of strength and struggle within 3 the family structure. 7 0 Be able to identify family interactions that maintain problems 4 within the family system. 7 0 5 In general, skill in observing family behavior is important. 7 0 6 Assess the family based on their interactions. 7 1 See beyond the presenting issues and consider drivers of 7 symptoms rather than just the “tip of the iceberg.” 7 0 Recognize the positive aspects of the caregiver/child relationship (e.g., "Look at the big smile on her face when you 8 did that!). 7 0 Assess all ecological factors contributing to identified problems 9 in the family system. 6 1 Demonstrate assessment skills that are based in family systems 10 theory. 6 2 74 Table 4.8 (cont’d) Demonstrate assessment skills that are based in awareness of 11 epigenetics. 5 2 Demonstrate assessment skills that are based in awareness of social 12 determinants of health. 6 2 13 Assess for family dynamics related to resistance. 6 2 14 In general, assessing family interactions is important. 7 1 15 Identify gaps in cultural knowledge. 7 1 16 Assess and understand a family's unique cultural identity. 7 1 Apply a sociocultural lens to assessment practices and be skilled 17 in cultural humility. 7 1 18 In general, the skill of assessing family culture is important. 7 1 19 Assess for historical trauma in family systems. 6 1 Explore the complexities inherent in a family’s history and develop a working understanding of these events as they relate to the 20 presenting problem. 6 2 Understand a family’s history and recognize how it impacts the 21 family's current circumstances and perspectives. 7 1 22 In general, the skill of assessing family history is important. 6 1 23 Conceptualize behaviors, interactions, and interventions. 7 1 Comprehend and conceptualize underlying causes of problems 24 and facilitate solutions based on these. 7 1 Stay focused on family process and not the content in which 25 interactions are occurring. 6 2 26 Think abstractly. 6 2 27 Think systemically (not linearly). 7 1 Demonstrate assessment skills that include multiple 28 perspectives and multiple domains of the person's ecology. 7 1 29 In general, the skill of systemic conceptualization is important. 7 1 Reliably classify behavioral observations of parent-child 30 interactions. 6 2 31 Understand behaviors as an expression of unmet needs. 6 2 32 Understand how behaviors are reinforced or extinguished. 7 1 Demonstrate skill in functional behavior assessments (identify what 33 purpose the behavior serves to implement change effectively). 6 2 34 In general, assessment of behavior is important. 7 1 Administer, interpret, and provide feedback for standardized 35 assessments. 6 3 36 Conduct a basic assessment that includes a risk assessment. 7 1 75 Table 4.8 (cont’d) Administer empirically validated assessments normed on children 37 and adolescents to screen for a range of symptoms. 6 2 Assess (initially and ongoing) for safety with children with 38 severe emotional disturbances. 7 1 Assess for trauma symptoms and they should do this with each 39 family they work with. 6 1 Utilize assessment of clinical concerns to develop treatment 40 plans based on prioritization of needs. 7 1 41 In general, skill in conducting structured assessment is important. 6 1 Engagement Skills (n=43) 42 Establish rapport with difficult clients. 7 0 Talk to individuals with a non-biased approach and listen non- 43 judgmentally. 7 0 Be equitable in allocation of attention between family members, 44 e.g., eye contact, time, questions asked, responses. 6 2 45 Relate to the concerns of all family members. 7 1 46 Build strong relationships with people in a brief amount of time. 6 2 47 Support and encouraging families. 7 0 48 Demonstrate skill in the use of humor. 6 1 49 Join systemically and strategically. 6 1 Match to youth and to the adult members of their family in 50 ways that are authentic. 7 1 51 Use self-disclosure when appropriate for joining purposes. 5 1 Engage in conversations regarding race/racism, diversity, and 52 equity. 6 2 Match their tone, style, language and focus to connect with 53 families. 6 2 Demonstrate skill in both verbal and non-verbal 54 communication. 7 1 Move from an individual, deficit-based focus to a relational, 55 strength-based focus. 7 0 Relate to all ages, genders, family roles, sexual preferences, 56 belief-systems, and styles with acceptance and understanding. 7 1 57 In general, skill in establishing a therapeutic bond is important. 7 0 Embody the emotions the client is verbalizing or 58 communicating non-verbally. 6 1 59 Appreciate and sincerely reflect concern for others. 7 1 60 Be sensitive to each family’s unique context. 7 1 76 Table 4.8 (cont’d) Empathize with the difficulty of parenting a child with 61 significant behavioral challenges. 7 1 62 In general, the ability to build empathic response is important. 7 0 Build and maintain multiple alliances (e.g., with the child and 63 parents). 7 1 Demonstrate skill in empowering family members and not take 64 sides. 7 1 Engage new support persons when they are added to the 65 treatment team. 7 1 Engage families and youth with a history of failed therapy 66 attempts. 7 1 67 Engage multi-stressed families. 7 1 68 Engage younger kids. 6 1 Engage families from a diverse range of ethnic/racial, 69 geographic, and socioeconomic backgrounds. 7 1 Engage other family members needed for treatment that are 70 not present from the start. 7 1 Conduct relational reframes early in treatment to help engage the 71 whole family in the treatment process. 6 2 Establish multiple therapeutic or working alliances with the parent 72 and the youth separately. 5 2 73 In general, skill in engaging families in treatment is important. 7 0 74 Align families around treatment goals. 6 1 Challenge families to develop goals that will help them change and 75 grow. 6 2 Connect problems identified in the family and how the treatment will address those problems in clear, concise, and 76 easy to understand terms. 7 1 Motivate all family members to accomplish goals that positively 77 impact the entire system. 6 1 In general, facilitating agreement on therapeutic goals is an 78 important skill. 6 1 Demonstrate skill in basic motivational interviewing (open ended questions, meeting the clients where they are, reflections, 79 asking permission, etc.) to increase parent 7 1 Enhance motivation in adolescents who are not otherwise 80 motivated to attend therapy. 6 1 81 Enhance each family member’s own motivation for change. 6 1 Motivate all family members to accomplish goals that positively 82 impact the entire system. 6 1 83 In general, skill in enhancing client motivation is important. 7 1 77 Table 4.8 (cont’d) Intervention Skills (n=100) Understand family culture and weave it into their treatment 84 plan. 7 1 Demonstrate skill in developing a plan for treatment 85 interventions based on a systemic case conceptualization. 7 1 Demonstrate skill in planning treatment based on systemic 86 diagnoses. 6 2 Demonstrate skill in treatment selection and planning, based on 87 knowledge from standardized assessments. 6 2 Demonstrate skill in facilitating a family centered approach to 88 treatment planning. 7 1 Develop treatment plans based on assessment of clinical 89 concerns and prioritization of those concerns. 6 1 90 In general, skills in treatment planning are important. 7 1 Demonstrate skill in describing the impact of positive parent- child interactions in real time (e.g., “Look at the big smile on 91 her face when you did that!) 7 1 Think complexly and clearly communicate those complex ideas in 92 sessions. 6 2 Coach the family to engage in different ways of communicating 94 and behaving with one another. 7 1 Know when they are being pulled into the middle of the 95 family's conflict. 7 1 Demonstrate skill in noticing feelings in the moment and then 96 helping parents and children make sense of those feelings. 7 1 Facilitate conversations between family members about their 97 vulnerabilities. 6 2 Help family members communicate clearly without putting self 98 or others down. 7 1 Demonstrate skill in and have desire to intervene with families 99 directly. 7 1 Direct process to address each family member’s content and 100 emotions. 6 2 101 In general, process-focused treatment skills are important. 7 1 Demonstrate skill in being directive with both parents and children to contain situations that could potentially get out of 102 hand (e.g., a child becoming aggressive in session. 7 1 Demonstrate skill in conflict resolution and diplomacy to help 103 families deal with conflicts they experience. 7 1 78 Table 4.8 (cont’d) Demonstrate skill in conflict resolution and diplomacy to support families with systems such as probation, neighbors, 104 school staff, etc. 7 1 Mange conflict when it arises in session so that it can be 105 targeted for change. 7 1 106 In general, conflict management skills are important. 7 1 Not let distractions or crises of the week derail execution of 107 planned treatment. 6 2 Demonstrate preparedness to teach or implement their plan for 108 the session. 7 1 109 Demonstrate good time management skills. 6 1 Demonstrate the ability to follow and maintain a session 110 structure. 6 1 Stick to the evidence-based practice they are using despite other 111 issues that may be brought up in session. 5 3 112 Manage multiple people in session. 7 1 113 Set session agendas responsive to the family’s needs. 6 1 Take leadership in a session by asserting themselves when 114 necessary. 7 1 Demonstrate exemplary listening skills balanced with the 115 ability to supportively interrupt. 7 1 116 Juggle multiple roles in a family session. 6 2 117 Directly address session goals. 7 1 118 In general, session management skills are important. 7 1 Demonstrate organizational skills to balance multiple families with high needs, as well as multiple responsibilities such as sessions, 119 court hearings, additional treatment team appts. Etc. 6 2 Balance time for sessions, contact, and assuring documentation 120 is complete. 6 1 121 Manage urgent needs with families as they arise. 7 1 Demonstrate skill in structuring the therapeutic relationship by being reliable, i.e., showing up on time, contacting family with 122 any changes needed. 7 0 123 In general, therapist organizational skills are important. 7 1 Reconnect with family members who are challenged in a 124 session. 7 1 Demonstrate skill in acknowledging and working with 125 resistance when it occurs. 7 1 126 Balance alignment throughout a session. 7 1 127 In general, skill in maintaining treatment alliance is important. 7 0 79 Table 4.8 (cont’d) 128 Effectively manage negative affect expressed by parents. 7 1 129 Reframe negative thoughts to more positive thoughts. 6 2 130 Use metaphors to support creating new perspectives. 6 2 Deliver new perspectives to family members in a manner that influences their willingness to see themselves and each other in 131 new or more hopeful ways. 7 1 Demonstrate skill in systemic reframing, i.e., providing a more 132 adaptive version of the family’s view of the presenting problem. 6 1 Demonstrate skill in conducting relational reframes early in 133 treatment to help engage the whole family in treatment. 6 1 Identify underlying relational patterns within the family in a way that does not shame or judge and instead provides hope or 134 worthiness. 7 1 135 Demonstrate skill in normalizing family situations. 6 1 136 In general, skills in reframing are important. 7 1 Facilitate conversations between family members when 137 appropriate. 7 1 Balance amplification of distress and generation of hope to 138 increase motivation to change in the session. 6 1 Help youth and parents identify and work through the blocks 139 or barriers to adopting new behaviors. 6 1 140 Demonstrate skill in challenging the family’s reality. 6 2 Demonstrate skill in the use of enactments, i.e., guiding and directing family members to have new (more adaptive) 141 experiences of each other in session. 7 1 Decentralize themselves in order to let the family execute necessary 142 changes on their own. 6 2 Provide a rationale for family members to examine themselves and 143 the need for change. 6 2 Use a range of interpersonal (warmth) and directive (teaching) 144 strategies in sessions to achieve changes. 7 1 Motivate clients through questioning boundaries and setting 145 expectations. 6 1 Ask youth and family members to interact differently with each 146 other. 6 1 147 Facilitate family restructuring when necessary. 6 3 Leverage alliances within the family system to create change within 148 the family structure. 6 2 149 In general, skill in facilitating change is important. 7 1 150 Use emotions skillfully to prompt change. 6 1 80 Table 4.8 (cont’d) 151 Manage, evoke, and leverage emotional intensity for change. 6 1 Use emotions as motivators, i.e., realizing that helping the client feel their emotions will help them realize changes that are 152 necessary. 6 2 Demonstrate skill in emotional attunement that facilitates emotional 153 deepening in sessions. 7 2 Help clients connect to their previously avoided emotions through a 154 variety of techniques. 6 2 155 Elicit feelings of understanding, love, and connection. 7 1 Create or enhance a healthy emotional experience for a youth 156 and their family. 7 1 157 Create emotional healing experiences in session. 7 1 158 In general, skills in working with emotion are important. 7 1 Use descriptive praise - praising what exactly is liked about 159 actions or thought processes of family members. 6 1 Engage family members in basic cognitive-behavioral coping 160 skills. 6 1 161 Engage family members in basic problem-solving skills. 7 1 Engage clients in regulation skills, such as mindfulness and 162 anxiety reduction strategies. 6 1 Demonstrate skill in specific CBT approaches, e.g., Dialectical 163 Behavioral Therapy, Cognitive Restructuring. 6 1 164 Engage family members in family psychoeducation. 6 2 In general, skills associated with cognitive behavioral theoretical 165 approaches are important. 6 1 166 Conducting a family therapy session. 7 1 Promote the parent/caregiver as the agent of change for the 167 child. 7 1 Demonstrate specific skills in structural and strategic family 168 therapy theoretical approaches. 6 2 169 In general, skills in family therapy are important. 7 1 Demonstrate skill in basic parenting knowledge ready to use in sessions, e.g., how to help parents decide on rules, or incentives 170 and consequences that will work for their family. 7 1 171 Coach parents in the moment on handling parenting situations. 7 1 172 Share knowledge of positive parenting practices. 7 1 Implement relevant behavior-management skills, which are often needed for children presenting with severe emotional 173 disturbances. 7 1 81 Table 4.8 (cont’d) 174 In general, knowledge of parenting approaches is important. 7 1 Balance active teaching (i.e., role play, modeling) and verbal 175 teaching (i.e., explanation, questioning) approaches. 7 2 Demonstrate skill in active teaching that engages parents in the 176 learning process. 7 1 Demonstrate skill in effectively using active teaching strategies 177 such as role play, brainstorming, eliciting, movement in session. 7 1 Demonstrate skill in using role play with families to allow them an 178 opportunity to practice skills learned before implementing. 7 2 179 In general, active teaching skills are important. 7 1 Demonstrate general skills in case management, including an 180 ability to advocate for families. 6 1 Demonstrate skills in advocacy with regard to oppression, 181 racism, or inequities experienced by family system. 6 1 Engage stakeholders and community members across the youth's 182 ecology in treatment as needed. 6 2 183 Coordinate care and collaborate with other service systems. 6 1 184 In general, skills in case management are important. 6 1 Reflexivity Skills (n=23) 185 Be reflective about the meaning of their work. 6 1 Understand the impact they are having on the family 186 relationship in treatment. 7 1 Be reflective about relational and sociocultural influences on 187 families. 7 1 188 Take perspective with clients. 6 1 189 Evaluate whether their clinical strategies are effective. 7 0 190 In general, skills in reflective practice are important. 7 1 191 Be aware of their biases. 7 0 Understand their physiological, affective, and cognitive 192 responses to their clients. 7 1 193 Be reflective about how their work impacts them. 7 1 194 Be aware of their own emotional wounds and potential triggers. 7 1 195 Hear supervision feedback and try suggestions given. 7 0 Understand their resistance to supervision feedback or 196 opportunities for clinical growth. 7 1 Understand their own social location (i.e., their experiences with power, privilege, and subjugation as it relates to race, 197 class, gender, sexual orientation 7 1 82 Table 4.8 (cont’d) Acknowledge their limited understanding of interactions 198 between family members based on cultural and family norms. 6 1 Understand how their diversity (i.e., identity and differences) 199 impacts their work. 7 1 Understand how their own culture (i.e., language, customs, 200 beliefs, and values) impacts their practice. 7 1 201 In general, awareness of self of the therapist issues is important. 7 0 202 Manage their self-care while in session, at work, and at home. 6 1 Demonstrate knowledge of trauma responses and how to 203 ground themselves. 7 1 204 Maintain a calm presence in the presence of chaos or conflict. 7 1 205 Self-regulate their own emotions. 7 1 Sit in the chaos, crisis, or disruption in order to determine what is 206 happening in the family system. 6 2 207 In general, skills in managing own self-care are important. 7 1 Adaptability Skills (n=9) Tailor (not modify) evidence-based practices in order to best 208 engage families in treatment. 7 1 Assess family cultural identity and subsequently tailor (not 209 modify) the evidence-based practices 7 1 Creatively tailor intervention strategies to specific families and 210 situations. 7 1 Understand the difference between tailoring and modifying an 211 evidence-based practice. 7 1 Adapt treatment to the family based on their unique strengths 212 and difficulties. 7 1 In general, skill in tailoring evidence-based practices to unique 213 families is important. 7 1 214 Adjust their therapeutic approach when needed. 7 1 215 Incorporate new skills into their therapeutic skillsets. 7 1 216 In general, therapist ability to be flexible is important. 7 1 Trauma informed care practice (n=6) Talk about difficult and complex issues such as trauma, neglect, 217 abuse, oppression, and power with families. 7 1 Demonstrate skill in recognizing, naming, and working with 218 families around the family's trauma experiences. 7 1 Demonstrate working knowledge of trauma reactions and how 219 to help clients regulate their emotions. 7 1 220 Ground themselves when working with family trauma. 7 1 83 Table 4.8 (cont’d) 221 Thoughtfully bring up traumatic events. 7 2 In general, skill in conducting trauma-informed practice is 222 important. 7 1 Note. Consensus statements are in bold font. Non-consensus statements are in regular font. Summary of Round 2 Results The objective for Round 2 was to achieve consensus among panelists on therapist skill statements (n=222) needed in EBP adoption. This round yielded agreement on 175 therapist skill statements over six main skillset headings, i.e., Assessment, Engagement, Intervention, Reflexivity, Adaptability, and Trauma Informed Care. Items that did not meet consensus (n=46) were present in all skill categories, except Therapist Adaptability, in which all items met consensus. Table 4.9 highlights the percentages of consensus by category. Table 4.9 Consensus Percentages by Main Category % Consensus Non- % Non-Consensus Category Consensus by Category Consensus by Category Totals Assessment 29 71 12 29 41 Engagement 34 79 9 21 43 Intervention 76 76 24 24 100 Reflexivity 22 96 1 4 23 Adaptability 9 100 0 0 9 Trauma Informed Care 5 83 1 17 6 Totals 175 47 222 Results Round 3: Identified Training Methods The goal of Round 3 was to obtain qualitative feedback to address the third research question: What are the corresponding training methods for each identified foundational therapist skill that can ultimately inform an intervention to improve therapist adoption of EBPs in 84 community mental health agencies treating SED children and their families? A main thematic area, Training Methods, was identified that described methods and strategies to support the development of a foundational therapist skillset. From this main theme, three subthemes emerged: 1) Preparing the Training Environment, 2) Specific Training Methods for Developing Skillsets, and 3) Additional Considerations Related to EBP Training. Preparing the Training Environment Preparing the Training Environment emerged as a theme that embodied therapist trainers setting up the training environment for success. Participants described how good preparation must be in place for training efforts to be successful before specific training methods are considered. The following quote highlights some of the themes considered in this category: “I think for [EBP] what we do in this state is that we have an informed two-day, which teaches a few of the skills. So it's kind of like an intro to the model. And some people come and they're going to be [trainees] and you have to go through that before you enter training. So some people have come and said, "Oh, this is not a fit." And other people have come with the same group and say, "Well, I want to do it now." So it's kind of trying out the skills you're able to fit. We also have the trainers that are training that, watch them to see how they're doing in role-play, how they're doing in their questioning.” Essentially this theme encompassed what experts reported as the best strategies to set the stage for success in training. This included: a) communicating the demands of EBP training, b) providing a safe space for processing and supervision, and c) assessing therapist readiness for training. 85 1.a. Communicating the demands of the EBP. This first sub-theme encompassed having conversations about what the EBP and its training requirements entail. This was described by participants as a strategy to open conversations about demands of the EBPs, explore the theoretical orientation to the practice, and even discuss a therapist’s willingness to do the practice. The following participant quote exemplifies the importance of having conversations about EBP demands prior to training in an EBP: “…those conversations prior to even learning, like training, coaching, understanding a model, going through all those steps and stages, all of these preparation conversations, I've learned, are so important in terms of what will your experience even be like learning something completely new? What will we need to navigate around? How anxious are you going to be? How does it feel for you to be vulnerable not catching on quickly? What can we anticipate happening down the road so that we can just be accepting of that and learn how to kind of get around that quickly so it's not a real roadblock for us?” This theme was punctuated by a second participant as follows: “But I think having a lot of conversations in preparation of that is helpful. I mean, especially when I think about undertaking [EBP], which is this huge commitment and huge learning initiative, we've got to be really honest with that, with clinicians, in terms of what it will take for them, especially for those that we think will have to make a major shift in their systemic thinking, preparing them in advance for what to expect.” In addition, several participants commented on setting training up for success by assuring preparatory conversations about a therapist’s willingness to train in the EBP have happened. For example: 86 “One of the things in my evidence-based practice we really changed this year is we want you to be here and if you want to be here because a lot of people are pushed into things they don't want to do.” And seconding this comment: “So that's one of the questions that I asked therapists when I interview them before coming into [EBP] training is what makes you want to be in the [EBP]? And if the answer is something like, "I have no choice," then I tell them that, "You're better off doing something else...It just doesn't work." I mean, they got negative response to being forced or being given no choice.” 1.b. Providing a safe space for processing and supervision. The second sub-theme included strategies to assure spaces for emotionally safe learning opportunities. Many participant statements echoed creating this space was not only a preparatory step but an ongoing practice. The strategies participants discussed were intended to set the stage for training and ongoing support in learning EBPs. This theme encompassed an understanding that therapists need to feel emotionally safe and supported to learn: “But before we get there [training], one of the things for me personally in working with clinicians is that it's really important in my eyes that they have and they feel safe in that environment, whatever it is that we're trying to teach them and help them with. I think it's important, just like a client, that they feel safe enough to bring out all those issues, to be vulnerable, that the therapist or student or however we want to describe them in that moment…” Another participant spoke to setting a safe foundation for beginning therapists: 87 “But I think that preparing, preparing people to be clinicians and deal with trauma, our communities, there's a very high exposure to trauma. I think being able to be healthy as folks is something that we kind of push as ongoing. As a beginning therapist and I think setting the foundations early is preferred. This work can be very intensive.” 1.c. Assessing therapist readiness for training. The final subtheme of preparation for training includes the idea that therapist readiness to participate in the EBP is assessed. This theme was reflected in participant statements around therapists’ skills and whether they felt they (as a trainer, supervisor, or consultant) would see the trainee as able to be successful in the model. One participant shared: “And we've been able to engage in what they call I think…an implementation readiness kind of pre-work type of stage. And it's not to say you're in or you're out. It's to really kind of gauge like, "What are your foundational knowledge, your skills and knowledge and attitudes? And where do we need to tailor the training? So that way we know we are providing you with what you need in order to have the uptake of this practice," because our goal is really to spread the practice. And if we miss the mark as trainers in not providing some of those-- not really foundational, but those funds of knowledge that we need clinicians to have or supervisors, then we're kind of missing the mark. So there's a kind of a form that we are able to use to be able to help us kind of target what those things are. “So it's a matter of trying to find out what is their skill set that can match with a good evidence-based practice that they kind of want to do right at the outset.” Similarly, the following participant spoke to assessing for a certain skill level before starting EBP training: 88 “I think all of this is great for moving them into EBP, but if they don't come at a certain level, you can't train them in an EBP. You end up holding the group back, training somebody in [basic skills].” And finally, this subtheme encompassed participants who shared strategies of hiring therapists specifically for their EBP: “I think engagement is so important. I would say I always hire people that are that personality. They're very genuine and engaging for [EBP].” Specific Training Methods for Developing Skillsets The second theme that emerged focused on specific training methods that could be used to develop therapist skillsets. This theme, which was highly related to the research question, captured a number of training methods identified by participants as well as the corresponding skillset areas for which these methods could be applied. To best organize participant responses, I outlined the results according to categories of the consensus therapist skillset identified in Round 2. Table 4.10 summarizes the therapist skillset and corresponding methods suggested by participants. Assessment. Participants remarked on seven strategies supportive of learning assessment skills. There were seven methods identified specific for assessment. These included: 1) giving trainees the opportunity to observe experienced therapists, 2) practice having therapists give assessment questions and observe interactions for diagnosing, 3) reviewing and processing assessment procedures as a team, 4) role-playing assessment questions and interactions, 5) receiving mentoring as they were learning assessment processes, 6) utilizing case vignettes to think about case conceptualization, and 7) be paired with training partners while assessment 89 skills were being learned. The following example was shared from one participant highlighting the use of practice and processing assessment procedures in a team environment. “[Have them be] part of the process of creating the clinical formulation and the diagnosis and the determination of likely prognosis, length of treatment, all that heart of a good assessment in an interactional style. And then have the tables flipped. And have them provide an assessment observed by an experienced clinician. And then after the assessment process, assessment with the client, walk through all those steps in a collaborative kind of back-and-forth way…Within three weeks of doing that, say 20 hours a week, you've got somebody who has a sophisticated assessment and differential diagnosis skill set” Similarly, another participant echoed the importance of practicing assessment skills and procedures utilizing a team setting that includes therapists with experience and provides opportunities for discussion. “I think the example I gave on the assessment where there's a bullpen, a bullpen, of experienced therapists and everybody is talking about the same assessment. You could do that with a video and everybody taking their own notes on the video and then discussing the differential diagnosis and how to word the clinical formulation.” Family Engagement. Participants had six methods to share that were supportive of learning engagement skills. Largely these methods emphasized ways in which therapists could either practice or reflect on barriers to engaging family members in treatment. This included those in training or consulting positions actively challenging (or helping a therapist reframe) cognitions around family members they perceived as resistant. Methods included: 1) assessing and then helping therapists reframe their cognitions around resistance to engagement, 2) practice joining with family members through role-play, 3) building reflective practice to help therapist 90 understand their own resistance to engagement, 4) providing supervision or coaching specific to engagement strategies, 5) utilize paperwork from EBPs to track engagement of family members, which allows therapists to see it visually, and 6) utilize case vignettes of engagement scenarios to conceptualize strategies to join with family members. The following example was shared from one participant highlighting the use of reflective processing and supervision as a strategy to teach engagement skills. “…then when you're talking about engaging families, I can see that maybe potentially using a recording and reviewing the recordings to say, "How are you engaging families? What does that look like? Where do you think you did engage?" So incorporating some reflective, active learning opportunities.” Another participant echoed the importance of role-playing engagement situations for learning and assessing barriers to therapist being able to engage families in treatment: “And really, I make them do role plays because I'm trying to flesh out just their basic engagement skills. Because that's probably the hardest thing to teach, is how to be a really engaging clinician. You either have it or you might not have it at all.” Intervention. Participants shared 10 various methods that aided therapists in learning intervention skills. Largely these methods emphasized ways of experiential, in-vivo practice of (or supervision on) specific interventions or techniques. Methods included: 1) role playing interventions, 2) practicing or rehearsing interventions, 3) utilizing a co-therapist to teach intervention in-vivo, 4) use of peer support or peer consultation group to support planning and utilizing interventions, 5) supervision to reinforce family interactive perspective in treatment, 6) supervision that is live or based on video recording of sessions, 7) providing video examples for therapist to watch of well-preformed interventions, 8) active modeling of interventions (in peer 91 group or supervision), 9) reviewing key intervention sequences in session video for experiential learning and reflective processing, and 10) utilizing training partners while onboarding to an EBP. To utilize these methods, participants emphasized the need to observe therapists learning interventions, either through live supervision, audio or video recordings, role-play, or rehearsal. Methods were inclusive of therapists watching others do interventions as well as watching their own videos of themselves doing interventions. The following example was shared from one participant highlighting modeling as a main method of training intervention skills followed up by a chance to process the learning through feedback: “You model them. And I just got done with a series of trainings that were done on the Internet. And we were able to-- what typically happens is we do a presentation and then a modeling and then splitting up into groups and kind of walking around as they're split up and dyads or in small groups trying out the interventions, giving them feedback, and then taking a kind of a-- after that, having kind of a general consensus afterwards about some of the things that they enjoyed or learned from and so forth. And the behavioral observation part, I mean, it's so critically important to see that happen.” Participants also heavily noted role play as an effective training method for teaching intervention skills, again incorporating the feedback loop. “I think, in any of these areas practicing in role play is really critical. That's what I think the best training method is to have people in pairs or in small groups have to role play, whether that's assessment, whether that's engagement, or the intervention skills is to actually do it and practice it, and then have a feedback loop, and then build on the role plays so that there's a linkage to the next one and the next one and the next one. For adult learners, I think that's probably one of the most effective techniques that I found.” 92 Reflexivity. Participants highlighted eight strategies that were supportive of building reflexivity skills. Methods for training reflexivity highlighted the use of supervision as a primary tool with specific nuances in how to approach learning toward building a therapists’ reflective capacity and understanding of their own social location. These strategies included: 1) peer supervision (either group or pair) which focused on a non-hierarchical way to self-reflect on therapeutic work, 2) peer support groups to be able to provide group opportunities for support and self-reflection, 3) supervision aimed at building reflective capacity through relationship building and helping a therapist explore their response to their work, 4) video recording and then supervision from that video, bringing therapists back to key sequences to explore their reactions and learning, 5) emphasizing strengths and building skills from a focus on the positive, 6) practice exercises in a group to elicit feedback on their responses to the work, 7) teaching trainees to evaluate their own practice through fidelity tools or watching their own videos, and 8) offering opportunities for experiential learning through role play or video – stopping at segments to process thoughts and feelings in the moment. Participants reflected that supervision capacity and use of recording is important to be able to facilitate these methods. One participant indicated the significance of opportunities for self-reflection and connected this as important to addressing therapists’ risk for secondary trauma. They shared: “I think the self-reflection and self-care piece, again, is something that we are understanding more and more how we need to provide the space for therapists to not feel guilty about it and needing to do it. And again, with some of the new social worker code of ethics guidelines that's emphasized more here in this state, probably nationally also. And it's amazing how difficult that's been in working with therapists to get them to really do that, so. And given what we said about all the other stuff, all the challenges and 93 everything else, if you're confronted with and needing to know how to-- what the best way is to respond, you've got to really be very reflective and understand where you're coming from, how you can tend to that secondary trauma, vicarious trauma. It's just so much. And just don't want to lose sight of that. I thought that was-- I was glad to see that included in your findings.” Another participant provided a suggestion on supporting the therapist in supervision to learn reflexivity through understanding your own social location or positionality as a therapist and considering this as on ongoing experience necessary for therapists to be successful in EBP adoption. “But it's about knowing that when you do work with a different culture, to be able to identify who you are, where you situate, understand who they are, and then do your research and reach out to those and be an active consumer of different cultures and be open to conversation and open to different lenses and create the space for that conversation. So, I think we can never fully prepare somebody. I think it's an ongoing experience that people need to have” Adaptability. Participants shared four strategies for teaching adaptability. Adaptability was noted by participants as akin to being flexible, engaging, and promoting culturally sensitive or modified approaches to treatment. Suggestions also encompassed ways in which therapists can evaluate their own work and utilize supervision for reflective processing opportunities. Methods included: 1) providing opportunities for self-evaluation, 2) reflective supervision around cultural humility and resistance, 3) exposure to videos of their own treatment with feedback, and 4) motivational interviewing. One participant summed up the connection of 94 addressing engagement and resistance through teaching the skill of evaluating their own practice. “But teaching them to start like, ‘I want you to look at yourself as a clinician and understand how you may or may not have contributed to resistance or them not following up or make another appointment,’ and whatnot. And that's a new concept for a lot of new clinicians. Like, ‘What? It could have been me? But I think I did everything right.’ And so that skill, I think, initially, as I'm hiring people and getting them trained, it's about, ‘Let's learn how to evaluate your own practice,’ which is super important as they move to an evidence-based practice and they're improving and they're tweaking. But they've got to have that third eye and that just seems to take a lot of work [laughter] to train.” Another participant echoed the connection of adaptability to engagement, promoting the use of strategies such as teaching motivational interviewing. They summarize as follows: “We even abandoned the word resistance because that's a one way. It's all their fault. And so we don't even use that word. I mean, it's just semantics in some way, but there's theory behind this. And so I think, you know, we've learned about motivational interviewing over the years. We've learned about, we have to adapt our approach. At, as, as things roll out and we are responsible for the engagement.” Trauma informed care. Participants shared four key strategies to promote trauma informed care practice in their trainings. Methods discussed included strategies that target addressing trauma at both the interpersonal and organizational strategies levels. Reflective supervision was echoed as a main strategy for promoting trauma informed practice, as was helping create cultures where self-care can be encouraged. Methods included were: 1) promoting transparency in organizational processes, 2) reflective supervision around keeping trauma 95 primary in intervention and then processing the impact of this on the therapist, 3) creating plans and strategies to help sustain therapists in their practice, and 4) promoting self-care strategies. One participant shared their organizational methods for building this skill: “We hold essentially annual trainings. So for any staff who might have come in since the last time that we reviewed those or taught those and we have all separate from our clinical training program. Those six principles are addressed on a regular basis as an agency. So from a training standpoint, we teach them and then outside of that, it is something that management and non-training staff are addressing on a system wide level with our organization. So I guess if I had to whittle it down to how we do that, we address it at a, at a systemic level. Before we really incorporate it into the clinical care. So we try to make sure that those items are embedded into the agency first.” Another participant targeted this skill in terms of utilizing reflective practice and presence when working with sexual trauma within the family system: “So there isn't, there isn't a session where that's not foremost in our therapists minds is the last thing we wanna do is, is to cause trauma and it's hard to do. It's a kind of intrusive work. I mean, even if we weren't dealing with sexual acting out on youth’s behavior, to simply expose family members who were, have a long trauma history, to narratives, the scenery, to imagery. You always got the chance that you're going to introduce some kind of flashback or some kind of shut down. And a lot that we treat act out because of the trauma histories and nobody's link to that in the past.” Overlap of training methods. One salient finding from the focus groups is that similar methods of training map onto teaching different skillsets. For example, categories like video 96 supervision, role play, and practice emerge in multiple categories. For ease of visually seeing these results, Table 4.10 outlines each main skillset and suggested training methods. Table 4.10 Suggested Training Methods for Each Skill Category Therapist Skillset Suggested Training Methods Category Assessment Observe experienced therapists Practice assessment and diagnostics Process assessment procedure in team setting Role play assessment questions and interactions Mentoring Case vignettes Training partners Family Reframing and assessing resistant cognitions Engagement Practice joining through role-play Building reflective capacity Supervision or coaching targeting engagement strategies Paperwork and worksheets to track family engagement Case vignettes for engagement conceptualization Intervention Role play Practice or rehearsal Co-therapy to teach interventions in-vivo Peer support/consultation Supervision to reinforce family interactive perspective Live supervision or supervision by review of video Video examples (Greatest Hits) Modeling interventions (peer group or supervision) Reviewing key sequences in session video for experiential learning Training partners Reflexivity Peer supervision Peer support groups Reflective supervision Video recording and review of video in supervision Focus on strengths Group practice exercises Teaching trainees how to evaluate your own practice through video/fidelity monitoring Experiential learning through role play or video Adaptability Self-evaluation Reflective supervision around cultural humility and resistance Exposure to videos of their own treatment with feedback Motivational interviewing 97 Table 4.10 (cont’d) Trauma Informed Promoting transparency in processes Care Practice Reflective supervision processes Emphasis on helping therapists sustain practice Promoting self-care Additional EBP Training Considerations Participants also described a number of additional considerations that provide an important context for EBP training efforts; these combine to form the third theme resulting from this study. Salient themes around deficits in clinical training programs, agreement on therapist competencies, and organizational factors such as lack of time or resources to attend to training needs emerged in this category. Deficits in graduate training programs. The first additional training consideration is related to graduate training programs and preparing therapists to apply a systemic focus in their work. Participants shared statements related to the overlap of clinical training in graduate programs and their work with new therapists, often reflected in a lack of preparation for systemic therapy skills. One participant summarized as follows: “…that in your graduate training you should be having at least one course on family systems and family therapy ...but I think it's essential, regardless of if you're in psychology or social work or MFT-- I mean, MFT gets it, but why are they the only ones that are getting it. I think it should be a core piece. If you're going to work with children or adolescents, you've got to know how to work with a family, because your access to those children or adolescents are through the family. And it's impossible to engage youth in treatment if you haven't engaged the caregivers. And so I think that that's an essential piece of training, in my own personal opinion.” 98 A second participant echoed concern that without good graduate training experiences, community mental health settings are left with a significant training burden: “I mean, my personal belief is that every university is giving out mental health degrees should train a therapist and one EBP that they can be certified in by the time they graduate. Otherwise, they're just feed for the community mental health fodder, right?” Therapist competencies. Participants had many statements related to whether or not therapists have appropriate competencies for their work and whether or not systems are set up appropriately to gatekeep when needed. For example, one participant shared: “I think if we go from entry into graduate school to clinician who holds fidelity in an evidence-based practice, there's so many gatekeeping opportunities that get missed.” A second participant discussed competence-based learning questioning where the best spot is for this to happen: “I do agree that there has to be competence-based learning opportunities for therapists, and that takes recording, video recording, maybe the supervisor's sitting in in the session with them, right, being able to observe, I mean, whatever it takes. But I find that takes a little longer. I want to say can't-- can't necessarily be done in graduate school because they’re not being specified to learn a certain type of therapy, and it's a little harder to do versus, "Okay. I'm now in the field. I'm practicing this evidence-based model, and now I have the time to actually gain competency." So, I think competence-based learning is a great way of kind of training therapists” Organizational factors. The final thematic area emerged around organizational factors that impact EBP training. This area encompassed how agencies prioritized training and other 99 factors like time to learn and expectations of therapists. One participant shared concern on agency priorities as it relates to training and building skills: “Um, so I think we can, we can lay out all of these great things [training methods] that would put it there. But if there's not the funding and the agency priorities in the time and the reinforcement or our recognition of excellence versus yeah, excellence versus number of hours.” Similarly, another participant shared concern about workflow and decisions made at the leadership level relating to training efforts: “So figuring out how with senior leaders, they can ensure that the workflow is such that clinicians not only have time to see families every week but if there is a no-show or a cancellation that they can reschedule within the same week.” Summary of Results Delphi studies are well-regarded for their ability to produce significant data on complex areas of health research inquiry (Jorm, 2015; Keeney et al., 2011; Kennedy, 2004). The results in this chapter outline both qualitative and quantitative findings that sustain each phase of the Delphi process. First, the recruitment and retention of participants is highlighted as an important outcome of this study – over 90% of participants were recruited and retained throughout the survey rounds. Second, Round 1 results identify 222 skill statements organized into six categories. Quantitative findings in Round 2 demonstrated consensus on 175 therapist skill statements, which included consensus on sub-categories of skills identified over six main skillsets: Assessment, Family Engagement, Intervention, Reflexivity, Adaptability, and Trauma Informed Care. Finally in Round 3, qualitative findings on training methods able to support therapist skillsets were revealed, along with other important considerations such as preparing the 100 training environment and considering larger systemic issues around therapist training and development. Findings from this Delphi study leave much room for discussion and implications, which will be discussed in the next chapter. 101 CHAPTER 5: DISCUSSION AND IMPLICATIONS The purpose of this mixed methods modified Delphi study (Linstone, 1975) was to determine a foundational therapist skillset necessary for therapists to effectively adopt family focused evidence-based practices in community mental health agencies as well as best approaches to training therapists in this skillset. To achieve this goal, a three round Delphi process was completed with 51 participants representing eleven different family focused EBPs that treat children with serious emotional disturbance symptoms in their family context. The Delphi method was chosen for this study as it is uniquely suited for obtaining expert opinions on complex areas of health research (Jorm, 2015; Keeney et al., 2011; Kennedy, 2004). The first research question was as follows: What are the foundational therapist skills experts identify as important to improving therapist adoption of family focused EBPs targeting SED symptoms in children? To answer this research question, Delphi participants were asked to brainstorm by answering three open-ended questions about therapist skills needed to work with SED youth and families. This resulted in the identification of 222 skills statements that were organized in six main categories and 32 sub-categories. The six main categories of statements were as follows: assessment, intervention, family engagement, trauma informed care, reflexivity, and adaptability. The second research question guiding this study was: What is the consensus among panel experts on the core foundational skills therapists need to acquire to improve adoption of family focused EBPs in community mental health agencies? Results from Round 2 realized consensus on 175 of the 222 items (79%) that went out for ranking, as organized under the six main categories. Last, participants were offered an opportunity to participate in focus groups to discuss the consensus items and training methods pertaining to the third research question: What are the 102 corresponding training methods for each identified foundational therapist skill that can ultimately inform an intervention to improve therapist adoption of EBPs in community mental health agencies treating SED children and their families? The results from these focus groups yielded one main theme, Training Methods, with three subthemes, 1) Preparing the Training Environment, 2) Specific Training Methods for Developing Skillsets, and 3) Additional Considerations Related to EBP Training. The first subtheme, Preparing the Training Environment, included strategies that helped set up the environment or plan for a good training experience, e.g., letting therapists know what to expect when learning EBPs. The second subtheme, Specific Training Methods, included more specific training techniques, e.g., role-play and reflective supervision. The third subtheme, Additional Considerations Related to EBP Training, reflects feedback participants gave that focused on larger systemic issues around therapist training. For example, participants offered statements related to lack of preparatory training in graduate school training programs. Foundational Skillset for EBP Practice The Delphi panelists reached consensus that the following therapist skill categories are essential for therapists to have for successful EBP implementation: 1) assessment skills, 2) family engagement skills, 3) intervention skills, 4) reflexivity, 5) adaptability, and 6) knowledge of trauma informed care. There were 32 total subcategories represented in these broader categories. Participant statements and comments helped illuminate some of the reasons that contributed to the variance in scores. It is important to remember that Delphi processes are not intended to converge with 100% consensus (Keeney et al., 2011). Items of disagreement offer significant opportunities for follow-up discussion and implications. 103 Assessment Skills Twenty-nine assessment skills were included as consensus items. They reflect the following subcategories: observing family behavior, assessing family interactions, assessing family culture, assessing family history, systemic conceptualization, assessment of behavior, and conducting structured assessment. Overall, participants viewed assessment as important in all these sub-categorical areas. Statements with the highest agreement of importance fell in the category of skills in observing behavior, family interactions, and identifying drivers of symptoms. There was also high agreement on recognizing the positive aspects of parent child relationships and assessing family culture. High agreement with these statements is congruent with the literature. For example, skills in observing behavior are regarded as highly important for working with SED youth in community health settings (see Graaf & Snowden, 2020; Hill, 2017; Hodges et al., 2004). Similarly, a focus on recognizing positive aspects in parent/child interactions is a hallmark of many family focused EPBs (e.g., GenPMTO; Forgatch & Patterson, 2010). Further, integrating culture and diversity into operational skillsets, such as teaching assessment, is consistent with research literature (Erolin & Wieling, 2021). The findings in the assessment category reflect that participants placed a high value on the utilization of relational and interaction tools for assessment processes, i.e., observing family interactions to identify strengths, assessing interactional patterns that maintain problems in the family structure, or assessing family culture. It is important to highlight that individualized assessment tools, such as Diagnostic Statistical Manual based assessments (American Psychiatric Association, 2013), were not reflected in participant feedback, and therefore did not make the Round 2 survey for ranking. This is not to suggest that participants do not value these tools in the 104 assessment process, rather that when considering a skillset for therapists working with SED youth and families, assessment skills based in interaction and relationship rise to the top as necessary and worthy of attention. This is also not to suggest that structured tools for assessment were not considered as important. Consensus was reached that structured assessments for risk, safety, and overall clinical concerns are important skills for therapists to have. Findings in the assessment category also confirm consensus on therapist skill development in systemic conceptualization, which was an important conclusion from the literature review (e.g., Karlin & Cross, 2015; Loras, 2018). However, two skill statements within the systemic conceptualization sub-category, staying focused on family process and not the content in which interactions are occurring, and the ability to think abstractly, did not reach consensus as individual items. Failure to reach consensus on these two items may indicate that participants defined the elements of systemic conceptualization differently, or that the skill may have importance, just not toward the aim of conceptualizing the family system. For example, Loras (2108) describes conceptualization components as “circular” and “non-pathologizing” (p.4). In contrast, key phrases like engagement-focused and goal-oriented are considered by Karlin and Cross (2014) when discussing conceptualization elements. It is also possible that these non-consensus statements were more reflective of model specific language, suggesting that their importance to the whole panel would have more variance. For example, staying focused on family process and not content is emphasized in Brief Strategic Family Therapy (Szapocznik & Hervis, 2020) but to a lesser degree in others (e.g., Child-Parent Psychotherapy; Lieberman, VanHorn, & Ippen, 2005). This is a significant point when considering the aim of this study was to find common skillsets that crosscut multiple EBPs. 105 The results from the Assessment category punctuate the importance of therapist skill acquisition in assessment methods that add to the current literature. While systemic conceptualization is documented in the research as an important therapist skillset for promoting EBP adoptions, there is little else to guide the discussion of what other kind of assessment skills would be important for therapists to have to work with SED youth and their families. Findings not only confirm the importance of systemic conceptualization as an assessment skill, but they advance this finding to consider a host of other important assessment skills as needed for successful adoption of EBPs, e.g., observing behavior, assessing family interactions and culture, and utilizing structured assessment for risk and safety. These experts view these four assessment skills as foundational to learning, an approach that is comprehensive and considerate of behavior, interaction, and structured for safety and risk. While case conceptualization in many ways is summary skill that moves assessment into treatment (Sperry, 2005), this list is a reminder that order for therapists to adopt EBPs more seamlessly, it is ideal if they are well versed in assessment skills that span assessment practices. Family Engagement Skills It is not surprising that engagement skills were identified among the core skillset, given the extant EBP family therapy literature on the importance of being able to reach, engage, and maintain families in treatment. For example, engagement is considered as an essential common factor of treatment (Hubble et al., 1999) and a critical basic therapy skill (Norcross & Wampold, 2011). Lack of family engagement is identified as problematic in fidelity studies (Horigian et al., 2016; Karlin & Cross, 2014). Simply stated, without engagement, there is no access to children and families for treatment (Becker & Stirman, 2011). 106 A total of 34 statements reached consensus in this category. Subcategories in this section included: establishing a therapeutic bond, building empathic response, engaging all family members in treatment, facilitating agreement on therapeutic goals, and enhancing client motivation. Statements with the highest consensus reflected establishing rapport with difficult clients, non-biased/non-judgmental approach, support, and encouragement to families, and moving to strength-based instead of deficit frames when working with families. Thematically, these are highly reflective of foundational relationship skills, well supported in the common factors literature (Blow & Sprenkle, 2001). Agreement in this category affirms the significance of engagement which has been long seen as important in the literature (e.g., Becker & Stirman, 2011; Horigian et al., 2016; Karlin & Cross, 2014). Findings continue to validate the importance of foundational relationship skills necessary for family therapy practice. Participants did not reach consensus on engaging families in conversations regarding race/racism, diversity, and equity. This finding is contrary to literature advocating for the importance of engaging families in conversations around diversity and intersections as a key to engagement and therapeutic alliance (PettyJohn et al., 2020). To be clear, highlighting this non- consensus item is not to say that engaging diverse families was not valued by participants; engaging families from a diverse range of ethnic/racial, geographic, and socioeconomic backgrounds and being sensitive to unique family contexts were both consensus items. Where panelists seemed to diverge was in how or when to bring up these conversations. Divergence on this statement may reflect the ways in which different EBP models address topics of diversity and equity. For example, some models may address issues of diversity and equity primarily in response to considerations and life experiences raised by families, while other models may bring up these topics more proactively. While consideration of diversity is seen as important, 107 especially related to sensitivity to various contexts, it is essential to consider how these conversations occur, with what clients, and with what timing. This is the piece most strongly supported in the literature, especially as it relates to building and maintaining therapeutic alliances (PettyJohn et al., 2020). This finding points to a potential need to better understand how EBPs approach teaching skills that attend to diversity and intersectionality and suggests additional guidance for supporting therapists in having well-timed and integrated conversations about topics such as diversity and equity. Intervention Skills This was the largest category of skillset statements (n = 100), which is not surprising, given the many facets of therapeutic treatment. Subcategories in intervention include treatment planning, process-focused treatment skills, conflict management skills, session management skills, therapist organization skills, maintaining treatment alliance, reframing, facilitating change, working with emotion, skills associated with cognitive behavioral therapy, skills in family therapy, knowledge of parenting approaches, active teaching skills, and case management. There was consensus on all subcategories (n=14). Seventy-six percent of the statements in this category met consensus for the final core skillset. The Intervention category was expected to have the fewest percentage of consensus items or the most disagreement among the panel based on two main factors. First, the extant literature that looks at therapist skills most often does so from reviewing fidelity findings (e.g., Allen & Johnson, 2012; Horigian et al., 2016). Fidelity findings tend to highlight adherence to specific model techniques or interventions. Second, specific intervention techniques are often what separate models of treatment, especially when considering that family focused EBPs tend to share similar theoretical underpinnings. For example, Trauma-Focused Cognitive Behavioral 108 Therapy (TF-CBT; Kliethermes, Drewry, & Wamser-Nanney, 2017) has attachment theory underpinnings, yet would utilize different intervention techniques than Attachment-Based Family Therapy, also a model built on attachment theory (Ewing, Diamond & Levy, 2015). These disagreements are further illuminated when reviewing the non-consensus items. Items that did not reach consensus, such as decentralizing in order to let the family execute necessary changes, skill in challenging family reality, and leveraging alliances within the family system to create change in the family structure do in fact reflect structural and strategic skills more commonly known in models such as Brief Strategic Family Therapy (BSFT; Szapocznik & Hervis, 2020). These findings are important to considering the applicability of common intervention skills across multiple EBPs and confirm assertions in the literature that we cannot assume that fidelity findings from specific EBPs automatically translate to therapist skills necessary for family focused EBPs as a whole (Lebensohn-Chialvo et al., 2018). This similar theme of assuring a common skillset continues when considering how reframing (i.e., changing the view) shows up in the therapist skillset. Reframing is heavily noted in the literature as a therapist skill necessary to support EBP adoption and is a common skill inherent to many systemic family therapy models (Weeks & D’Aniello, 2017). Yet, it is one of the subcategories within the intervention category with higher variance (i.e., less consensus). Statements in this subcategory that included the word “reframe” had less agreement than those that used less specific terminology. For example, demonstrate skill in systemic reframing and demonstrate skill in conducting relational reframe had lower medians than their counterparts deliver new perspectives to family members and identify underlying relational patterns in a way that does not shame or judge and instead provides hope or worthiness. Further, reframe negative thoughts to more positive thoughts did not reach consensus. Although there was consensus on the 109 subcategory of reframing, these subtleties suggest that the skill of “reframing” may best be described by other statements (e.g., deliver new perspectives) that broaden its definition and keep it applicable to multiple family focused EBPs. Considering a broader definition of reframing may be helpful to consider when developing future training efforts to be sure it aligns with multiple EPBs. Participants named skills such as case management, cognitive behavioral therapy, and supporting caregivers with parenting strategies as important therapist skills toward the aim of EBP adoption with SED youth and families. This is a novel finding and an important addition to the literature for several reasons. First, it establishes these skills as necessary to systemic family therapy practice, even though they may or may not be specific approaches to certain EBPs. For example, the case management subcategory, which included statements related to advocacy and coordination of care, is commonly reflected as a need for working with SED youth and families in community mental health settings, but not commonly to systemic family therapy practices (Goodcase et al., 2021). Similarly, use of cognitive behavioral approaches met consensus. While cognitive behavioral approaches often underpin systemic EBP treatments, (e.g., TFCBT; Cohen et al., 2006; BSFT; Szapocznik & Hervis, 2020; Functional Family Therapy; Robbins, Alexander, Turner, & Hollimon, 2016), it is not an explicit theory of use for other systemic models. And last, while a few models represented in this study have a primary aim toward teaching strategies that support parents in caregiving roles (e.g., GenPMTO, Forgatch & Patterson, 2010; Triple P; Sanders & Turner, 2005), many other models do not have specific strategies built in but recognize the need for it (i.e., Allen & Johnson, 2012; Iadarola et al., 2017). This finding suggests participants value and recognize a breadth of interventions that may be supportive to sustaining EBP adoption treating SED youth and families that fall outside of 110 traditional EPB training. This is significant implications for developing training that can support acquisition of these additional skills and assuring that families with needs for support in advocacy, parenting, or more specific cognitive behavioral interventions have access to that support. Last, I would be remiss not to discuss the subcategory of skills in family therapy given the study goal to support the advancement of family focused EBPs. Family therapy skills were represented with four unique statements. Three of these items were included in the consensus skillset: conducting family therapy sessions, promoting the parent/caregiver as the agent of change for the child, and the general subcategory skill is family therapy are important. The item that did not reach consensus was demonstrating specific skill in structural and strategic family therapy theoretical approaches. This item not reaching consensus is in line with fidelity reviews that suggest structural and strategic skills are more model specific and not generalizable to most systemic family therapies (Lebensohn-Chialvo et al., 2018). However, I highlight the statements that did reach consensus in this subcategory as important to punctuate when contextualized with the research literature. Although newer therapists and recent graduates report belief that family therapy is important and ideal, their actual practice of it remains varied. This is noted in model specific fidelity feedback (Lebensohn-Chialvo et al., 2018) as well as in studies that assess therapists’ attitudes and practice habits (e.g., Oed & Gonyea, 2019). These findings remind us that even when we are considering only family focused, systemic models of treatment, finding ways to support therapist competency in conducting family sessions remains a salient point. Successful EBP adoption relies on a therapists’ ability to conduct sessions that involve the family. This finding establishes that therapists need training in continuing to obtain the skillset to do so. 111 Reflexivity This category included 23 skill statements within three subcategories: skills in reflective practice, awareness of self of the therapist, and self-care. All but one skill statement reached consensus. This was overall consistent with the literature promoting therapist reflexivity as an important and sometimes overlooked skill to be taught (Aponte et al., 2009). Findings in this category speak to the value participants place on reflexive practice and validate that reflexivity is an important area for skills training. Some have argued that a lack of attention toward therapist reflexive capacity is a common characteristic of EBP adoption (see Allan et al., 2018). In contrast, these findings highlight agreement among the Delphi panel that building therapist emotional capacity is a foundational skill necessary for therapists adopting EBPs. This finding is congruent with scholars who advocate for therapists to practice handling difficult emotions, learn to understand their emotional triggers in treatment, and exercise maintaining emotional presence in session (Aponte et al., 2009, Owen, et al., 2014; Rousmaniere, 2016). Results such as these are encouraging toward assuring that therapists’ emotional needs will be less likely to be overlooked in the work they do as they learn new skills for important for EBP adoption. While many scholars have promoted self of the therapist development (e.g., Aponte et al., 2009) in treatment, Delphi experts support these scholars and have viewed improving a therapists’ ability to understand themselves and their growth in their work as an important foundation to learning EBPs. Further, the skills identified in this category include a therapists’ understanding of how their own diversity impacts their work. This builds on recommendations that therapists practice integrating diversity in their work including a deeper understanding of their own diverse backgrounds (Erolin & Wieling, 2021). Having an endorsed 112 skillset specific to therapists’ reflexive position provides a foundation for continuing to develop this skill area moving forward. Adaptability Intervention researchers often promote cultural adaptations of existing EBPs to intentionally meet the needs of cultural community or population (e.g., Horigian et al., 2016; Parra Cardona et al., 2012). Participants upheld these principles by endorsing adaptability as a skill category important for therapists to have when working with SED youth and families. There were nine statements in this category with two subcategories: 1) tailoring evidence-based practices, and 2) therapist ability to be flexible. The first subcategory reflects the panel’s agreement on skill in engaging diverse families. The fidelity literature clearly connects the ability to engage diverse families with the ability to be able to adapt EBPs to the client (Horigian et al., 2016; Norcross & Wampold, 2011). Five statements represent the general skill of adapting, and of these statements, a clear distinction is made that adapting (or tailoring) is not the same as modifying an EBP. This is important as it was clear participants were not advocating to go off model; rather instead they believed that skills such as tailoring EBPs to specific family situations or cultural contexts were important to promoting fit and acceptance in diverse family systems. The emergence of this consensus skillset is congruent with aims on adaptations of EBPs (e.g., Parra Cardona et al., 2012) and are encouraged for therapists to consider in order to engage cultural communities (Horigian et al., 2016). This skillset is present in more recent psychotherapy supervision models, e.g., multicultural orientation perspective (MCO, Watkins et al., 2019) that emphasize adaptations in full consideration of cultural understanding of client and self of therapist. 113 The panel’s endorsement on adaptability as a skill area is significant for building culturally relevant practice. Most notably, it takes therapist adaptability skills in an important new direction – one that goes a step farther than the literature. While the literature promotes cultural adaption of EBPs, and it promotes therapist flexibility in terms of adapting to new knowledge and being flexible when learning new skills, it falls short in connecting these two as a relevant therapist skillset specifically for working with SED youth and families. Participant endorsement of this adaptability skillset makes a first and important connection to support ongoing efforts to provide culturally relevant EBP practice. Trauma Informed Care Six statements encompassed trauma informed care practice, only one of which did not make the final consensus skillset. There was only one subcategory in this section. There is endorsement from the panel that skills in trauma informed care are important to EBP adoption. Trauma informed care is broadly defined as a set of core principles or ways of being in practice that attend to the complex and sensitive nature of trauma response (National Childhood Traumatic Stress Network Task Force, 2012). I anticipated that trauma would be a larger category of skill statements. This was in part due to increased attention to the prevalence of trauma in community health settings (Alisic et al., 2014; Briggs et al., 2012) and the push to develop trauma competency in practitioners to compliment EBP learning in health settings (IOM, 2007; Layne et al., 2014b). Perhaps because trauma informed care practice is defined as a set of principles to follow and less a specific skill, it is logical that there were not more statements related to skills specific to trauma informed care. Or, that similar to attending to culture and diversity, trauma skill statements were integrated into other more defined skill categories, e.g., reflexivity and assessment. 114 However, the statements meeting consensus give little to support the full range of trauma informed practice competencies, which include: 1) enhancing therapist’s empathic understanding of trauma experiences from the child and family perspective, 2) developing clinical judgement in practitioners who work with traumatized youth and families, 3) increasing therapist interest in developing trauma focused training with evidence based practices, and 4) developing skill in systematically evaluating multiple perspective and the unique circumstances, strengths and needs of each client (Layne et al., 2014b). One consensus skill represented in the final skillset is congruent with this framework, i.e., skill in recognizing, naming, and working with families around the family’s trauma experiences. Similarly, working knowledge of trauma reactions and how to help clients regulate their emotions is a consensus skill statement firmly found in the trauma-informed practice literature (Cohen & Mannarino, 2015). Nonetheless, it is important to note that the need for trauma informed care practices to be infused in EBPs has been a consistent call in the literature as it supports improvement of EBP adoption (Layne et al., 2014b; Loras, 2018). While panelists endorsed an important set of skills for trauma informed practice, additional research efforts may be helpful to understand how family focused EPBs line up with the full range of trauma informed care principles. Overall Skillset Discussion and Summary While it was practical to organize this discussion around the categories to break down the large number of consensus items into manageable chunks for discussion, I did this at the risk of losing a vision for the whole. This last section provides a few overall reflections. First, in the intervention category, I remarked that high variance was expected among the panel experts due to each of them representing models of treatment that have specific, identifiable therapeutic interventions. This did not happen. Instead, experts recognized a large 115 percentage of intervention skills that were common across all models. This assertion holds up when reviewing non-consensus items as well. Here I found that there was more evidence of model-specific language, e.g., structural and strategic focused language such as facilitating family restructuring. As noted by Lebensohn-Chialvo and colleagues (2018), skill deficits in this area are not thought to be generalizable to most family therapy models, and these results seem to support that notion. Overall, the methods of this study worked as envisioned in producing a foundational skillset that would crosscut multiple EBPs. This is not to say that model-specific techniques do not have a place in systemic family therapy; rather it should be noted that I aimed in this study to identify the core skills that would eventually support more specific model technique acquisition at the time of EBP implementation. Second, it is important to note that my goal was to find a core skillset that crosscut multiple family focused EBPs and could be applied to multiple mental health disciplines as commonly employed in community health settings. With little clarity in the literature specific to therapist skills needed for family focused EBPs, we conceptualized this study in a common factors framework with the idea in mind that skills can be summarized in ways that emphasize elements important to multiple therapies and approaches and that these commonalities exist (Sprenkle, Blow, & Dickie, 1999). While not a complete alignment, consensus skills mapped onto several important categories considered in a common factors approach, e.g., client engagement, therapeutic alliance, systemic conceptualization of problems. Further, these findings continue to contextualize the importance of the results of this study in promoting training opportunities that will benefit practitioners preparing for EBP training, recognizing that effective treatment operates through EBPs, not the other way around (Sprenkle & Blow, 2004). The inventory of identified therapist skills important to improving therapist adoption of family 116 focused EBPs with SED demonstrates the commonalities among family focused EBPS and therapist skills needed to treat SED youth (Blow, Sprenkle, & Davis, 2007). The development of this therapist foundation skillset advances previous studies that have examined components of effective practice across models, without getting into specific model studies centered on fidelity (e.g., Aarons et al., 2019; Horigian, et al., 2016). Even without consensus identification, the data contribute to a clearer understanding of the range of skills EBP experts consider as important for therapists utilizing family focused EBPs in community health settings. Training Methods to Support Learning of Core Skillset Results from the focus groups yielded one main theme, Training Methods, with three subthemes: 1) Preparing the Training Environment, 2) Specific Training Methods for Developing Skillsets, and 3) Additional Considerations Related to EBP Training. Overall, the results show several strategies and methods that are supportive toward training therapists in the core skillset identified. Preparing the Training Environment Preparing therapist for the demands of EBP training. These findings suggest that there is significant benefit to having conversations with therapists ahead of time about the training commitment involved with the EBP and their willingness to do the training. This finding is important to promoting transparency and assuring fit. This finding corresponds to other established research that supports therapists’ pre-knowledge of the EBP predicts greater participant in training and implementation activities (e.g., Pemberton et al., 2017; Shapiro et al., 2012). This is seconded by Akin et al. (2014) noting the fit of provider and EBP is essential. However, I also consider here that EBP practice choice may or not be supported at the practitioner level, despite significant evidence that choice matters. Findings suggest 117 organizations should consider how to weigh this important evidence of choice with organizational aims and goals. Providing a safe space for processing and supervision. Several participant responses supported the need to establish up front safe spaces to process EBP training and for supervisory relationships to be reflective about training processes. The findings fit especially well when considering the insufficient literature on therapists’ experiences in training especially in the vulnerable place of learning new things (Allan et al., 2018; Nel, 2006). This finding also speaks to the risk of learning environments not designed for reflective processing or the opportunity to be vulnerable. This conclusion is further contextualized considering the findings of Allan et al. (2018) on the importance of learning systemic EBPs not just on the cognitive level but having access to supervision and a space to sort out personal issues or feelings that may arise in the training process. Participants advise organizations considering training in EBPs to consider the context for development of supervisory relationships that can support EBP learning and the processing necessary to support therapists in the process. Assessing therapist readiness for training. Participants supported assessment for therapist readiness for EBP training prior to starting training. This finding is novel, yet has limitations in how it can be operationalized, largely because there is not agreement on a specific method or set of skills with which to do this. Participant responses indicate their assessment strategies (pre-EBP) in areas such as discussing theoretical orientation or assessing for certain personality traits. Ways to formally evaluate readiness through research base or standardized protocol were not reflected in the group feedback. They are also not well defined in the literature, outside of therapist attitudes toward EBPs (Aarons, 2004) or more general guidelines for clinical practice (e.g., APA, 2006). This is an area necessary for further research. 118 Specific Training Methods for Developing Skillsets Participants shared several specific methods through which to support training therapists in core EBP skills. I highlight here that the methods discussed map on to what the extant literature reports about the benefits of adult learning and grounding training in active, adult learning principles (Bryan et al., 2009; Kenner & Weinerman, 2011). These findings contribute to the growing literature about the use of active strategies for supporting adult learning. Assessment. Participants endorsed significant importance regarding therapist having skill in assessment practices and gaining this skill through interactive methods. Specifically, findings suggest opportunities for practicing assessment skills through role play, reviewing and discussing case vignettes, and working with experienced clinicians to develop assessment skills. These findings are consistent with the literature as summarized by Frank and colleagues (2020) that intensive training and ongoing consultation are core components of learning assessment practices. However, these findings add to the current literature by suggesting that a team approach with practice components, e.g., role play is significant toward the aim of supporting assessment skills for therapists learning family focused EBPs. Participants also emphasized the use of experienced clinicians as teammates or partners in learning assessment practices. Utilizing teammates within the organization to teach assessment skills may be a highly sustainable option for those who have seasoned clinicians on staff who have demonstrated good clinical assessment skills. Family engagement. These findings underscore the importance of training methods that include reflective supervision, reframing engagement, and assessing (therapist) resistant cognitions. Other skill categories did not reflect so heavily in these more interpersonal training methods; however, it is not surprising given engagement is a critical skill to not only sustain EBP 119 (Karlin & Cross, 2014), but to maintain the treatment relationship in general (Sori & Sprenkle, 2004). These findings reflect an emphasis on working with trainees to understand their blocks to engaging families instead of seeing resistance as a property that the family possesses. This has even more importance when preparing a training environment for emotionally safe, reflective space to process. While best practices exist for intervention on engagement (see Szapocznik et al., 2015), future research should look more closely at clearly defining the training methods and reflective processes that can supporting therapists in building engagement with families. Intervention. Experiential, active strategies were well supported for therapists learning intervention skillsets. This finding builds on previous findings that didactic workshops and cognitive only learning are not effective in learning EBPs or systemic treatment (Frank, Becker- Haimes, & Kendall, 2020; Allan et al., 2018). These findings also extend to considering the benefits of therapists being taught intervention skills in ways that model how the skills look in practice. This is contextualized in the literature on process skills utilized in interventions like GenerationPMTO, where the training methods utilized mirror the interventions that therapists learn to deliver to families (Knutson, Forgatch, Rains, & Sigmarsdóttir, 2009). These findings provide additional possibilities for continued research on training process skills. Reflexivity. Participants regarded supervision and peer support as primary methods for building reflective capacity in therapists. Self-reflection and focus on the person of the therapist are well established in the research literature (Scott et al., 2021) and methods such as supervision and support are important pathways for that reflective process to occur (Aponte et al., 2009). Debate has been established in the literature with regard to how much attention is paid to and how much capacity there is for reflexive practice in light of EPB adoption, which is where these particular findings have their most salience. Supervision is a necessary vehicle for successful 120 EBP adoption (Beidas & Kendall, 2010), yet supervisors are known to struggle with time to deliver supervision in community mental health settings (Dorsey et al., 2017; J. Bayardo, personal communication, 30 October 2018; 17 May 2019). Supervisors in community mental health settings often prioritize their time supporting case conceptualization followed by therapeutic interventions, with personal support to practitioners falling last (Dorsey et al., 2017). It is important to contextualize this finding to thoughtfully consider the capacity of organizations to implement reflective supervision practices as a strategy to promote reflective practice skills in therapists. This juxtaposition may lend additional credibility to the peer consultation groups also endorsed by participants as tools for helping therapists develop reflective capacity. This is an area that may benefit from further research and would perhaps require changes at the organizational level to promote reflective supervision as a priority and support resources toward that aim. Therapist adaptability. Salient themes for promoting therapist adaptability included strategies that facilitate engagement and cultural sensitivity when working with families. Participants valued promoting strategies that facilitated therapists evaluating their work toward the aims of cultural humility and looking at the ways in which they were resistant in the engagement cycle with families. They suggested formal interventions to support skills in engagement, i.e., motivational interviewing (MI; Miller & Rollnick, 2012). This finding extends the current literature in a few ways. First, it promotes training methods that introduce other formal training interventions, as suggested with using MI as a supplemental training strategy. One hallmark of MI is teaching participants that interventions they use can either increase or decrease resistance and that resistance is not a property of the client. MI is prior endorsed as part of the therapist skillset under Engagement in this current study. Participants endorse training in 121 this intervention as part of the core therapist skillset and an important training method to improve culturally sensitive engagement in treatment. Additional research could consider examining the use of Motivational Interviewing as conjoint learning with family focused EBPs. Second, it is important to highlight that participants valued strategies to improve adaptability that focused on therapist reflective process and evaluation of their own work, carefully challenging the idea that resistance is a client issue. This challenges to some degree current literature which often depicts clients as resistant and downplays the therapists’ role in engagement to one that is dependent on their theoretical orientation (Ibebunjo, 2021). This set of findings do overlap with the findings on reflexivity where reflective supervision is a promising strategy to train therapists to see themselves in their work. However, this category is differentiated by the aim to promote adaptability skills as a way to enhance engagement with diverse families. Much like discussed with reflexivity, there are potential considerations on the feasibility of reflective supervision opportunities. Nonetheless it should be considered strongly as a means to promote helping therapists adapt treatments to engage families. Trauma informed care practice. Reflective supervision was supported by participants as a primary strategy to help teach skills in trauma informed practice. Findings in this category also described the importance of training therapists to keep trauma at the forefront of their interventions, which meant that reflective supervision was a vehicle to achieve that end. This finding is consistent with the literature that suggests supervision address supporting therapists to assess and address trauma in family systems, while at the same time providing therapists safety for processing the traumatic nature of their work (National Child Traumatic Stress Network [NCTSN], 2012). 122 Participant responses in this category did extend to trauma informed care practice to the systemic or organizational level. Defining trauma informed practices as a set of principles that target both the practitioner and the system they work in is congruent with a comprehensive view of addressing the impact of trauma (IOM, 2017; National Child Traumatic Stress Network [NCTSN], 2012). As trauma-informed practice is more commonly considered an organizational issue in health care (IOM, 2007) and in human services organizations (Bassuk, Unick, Paquette, & Richard, 2017), these findings support additional research to identify training strategies that can continue to promote skill development at the practitioner and system level. Additional Considerations Related to EBP Training Deficits in graduate training programs. Participants suggested there is a significant lack of preparation for systemic therapy skills coming from graduate training programs. Participants also suggested substantial difficulties arise when they are training new graduates without basic clinical skills, as the onus of teaching falls to them. This finding suggests there may be opportunity to better understand and potentially respond to gaps between graduate programs and clinical practice. Some initiatives are noted in the literature around partnerships between internships and jobs to improve skills training toward this aim (e.g., Grauf-Grounds & Sellers, 2006). However, this finding points to a potentially broader question of what skills should be covered in graduate school and what kind of standards and competencies should be expected. This an area with significant potential for additional research. Therapist competencies. Participants stated a need for competency-based learning opportunities and for opportunities for gatekeeping or assessing a therapist’s ability to practice with competency throughout their clinical training and internships. There was disagreement among participants on where this should happen. Some suggested graduate schools play a role in 123 this, and others suggested that post graduate externships or on the job learning should account for the possibility that therapists would need to develop additional skills to learn EBP practice. This is a complex recommendation to unpack. Research literature provides little definition for practicing organizations around evaluating competency for employees or gatekeeping activities for trainees in community settings (Homrich & Henderson, 2018). On the other hand, graduate schools have clinical training standards and competency guidelines through their accrediting bodies, e.g., Commission on Accreditation of Marriage and Family Therapy Education (COAMFTE, n.d.) or Council on Social Work Education (CSWE, n.d.) that can guide students learning in clinical training sites and internships or placements concurrent with student’s graduate clinical training programs. How these guidelines translate to post graduate practice are undefined. The skills taught in accredited programs may not be the skills required to successfully adopt EBPs. In some cases, graduate training may work against EBP adoption, especially when training programs are disconnected from EBPs or when faculty may dismiss evidence-based approaches in support of their own pet theories. Findings certainly affirm a gap which has implications for community health organizations who operate with less guidance than graduate training programs, yet often are in the position to have to evaluate core skill development of therapists. This is a promising area for future research. Organizational factors. Perhaps not surprisingly given what we know about considering the context of community health settings in EBP implementations (Aarons 2009), participants discussed concern that while agreed upon skills and training methods are known, nothing can come of them without organizational support. Participants named variables such as funding or time for therapists to process their learning and receive the support necessary for that process as barriers to implementing skill development opportunities. This is a concern echoed in the 124 literature; many studies highlight organizational challenges inherent to EBP implementations in community mental health systems of care that impact therapist development (see Aarons et al., 2009; Beidas et al., 2014; Rodriguez et al., 2021). At the same time, participants were equally clear in that their experiences have led them to understand promising methods to train and support therapists in EBP adoption with SED youth and families. Despite organizational factors that may exist, participants were clear in what they understand to be the best methods for training. This suggests that while organizational factors may be a consideration, this should not undermine what is known about the most promising resources to support therapists in EBP adoption with SED youth and families. Limitations of the Study Diversity of panel. The diversity of this participant group was a study limitation. The first priority for recruitment was to select participants with expert credentials, which meant recruitment had less focus on population representation. The participants were a largely female, Caucasian group of EBP experts that tended to have many years of clinical experience (M=22) and were highly educated. This sample is under-representative of male (or non-binary) participants as well as those with a more diverse ethnic identification. Limits to qualitative data. While the qualitative data in this study provided important information about therapist skillsets and corresponding training methods, its limitations should be noted. First, qualitative rounds in Delphi studies are iterative and open to bias in the coding process. This is particularly true for this study given the large amount of data generated from having a large participant sample. Further, while efforts were made to mitigate bias, I acknowledge that the construction of the surveys and interpretations remain subjective and biased even with methodological safeguards in place (Creswell & Poth, 2018; McPherson, 125 Reese, & Wendler, 2018). Second, due to time constraints of the focus group, there was a lack of saturation on specific tools that could be utilized to provide the training methods suggestions. Identifying more specific tools and best practices that could be utilized for training is an important area for future research. Lengthy Round 2 survey. Another limitation is that this study had a lengthy survey process in Round 2, which resulted in a substantial demand on participants’ time. Moreover, the coding scheme organized data in categories, of which one was very large. It is possible participants may have had survey fatigue in the process which could have impacted agreement rankings. Study Strengths Participation and expertise. A true strength of this study is the panel participation and expertise. This study’s panel represented diversity in participant age and role identity (within their EBP). All 11 EBPs were represented through each round of inquiry. In addition, there was a significant amount of interest in this study that was maintained through each of the Delphi rounds. This study had a 90% or better participation in all rounds. Consistent participation is key to assure Delphi findings can be both relevant and generalizable (Hsu & Sandford, 2007). Many participants indicated not only a strong interest in the topic area of EBP implementation in community health settings with SED youth, but also revealed a desire or opportunity to have their voice heard. Academic-community partnership. The framework for this study is rooted in a community-academic partnership which not only informed the need for the study but affirmed trustworthiness of findings along the way. Balanced attention to community partner feedback and research literature proved valuable toward triangulating findings and providing opportunities 126 for critical reflection throughout the course of the study, lending additional trustworthiness. This partnership is key to assuring that the results of this study move forward since it keenly answers a research question needed at the organizational level to inform how they invest in and plan for EBP training moving forward. In addition, findings are immediately applicable to the partner who hopes to utilize them in strategic planning for training initiatives. Pathway for advancing EBP implementation. Perhaps the biggest strength of this study is its contribution to a practical question about therapist training not yet answered in the research literature: What are the skills (and corresponding training methods) necessary to improve the adoption of EBPs treating SED youth in community health settings? Delphi inquiries are intended to start the process of answering complex research questions and identify next steps of inquiry (Lund, 2020). This study provides a foundation with which to consider future training interventions. Group opinion elicited in this study provided an inventory of 175 skills categorized over six key skill areas with which to guide future training research and implementation. In addition, this study has great impact in affirming the use of active teaching methods grounded in adult learning principles. Study Implications Implications for Research The identification of the therapist skills important to improving adoption of family focused EBPs advances our knowledge of skills needed to effectively treat SED youth in community health settings. Consensus at both the category and individual skill level has significant implications for moving these findings into a pilot training intervention based on the core skills content. A pilot training intervention could be a primer training for community health organizations that are either moving into or trying to sustain EBP implementations. It may also 127 hold promise as a standalone intervention for organizations that have yet to adopt family focused EBPs but hope to improve outcomes for families. The aim of such an intervention study would be to support the future development or sustainment of model-specific techniques already well- research in in systemic family therapy. Continued research is needed to determine if or how this core therapist skillset could be utilized to assess therapist readiness for EBP training. While participants offered important suggestions around preparing therapists for EBPs by offering pre-workshops and looking at fit, they highlighted a lack of formal methods applicable for assessment purposes. This is similarly undefined in the literature, outside of therapist attitudes toward EBPs (Aarons, 2004) or more general guidelines for clinical practice (e.g., APA, 2006). Assessment of therapist readiness has significant implications toward organizational investment in training therapists in EBPs. Research is needed to explore the development of specific competency-based learning opportunities for each of the core skillset categories. This will support attention to individual learning styles and needs and provide a pathway with which to determine competencies at the clinical training program level. As it stands right now, there is too big a leap from general discipline specific guidelines for competent practice (e.g., APA, 2006) and the general therapeutic skills needed to conduct EBPs with SED youth and their families. This will offer consistent, methodical ways of determining therapist skills, strengths, and needs. Research efforts should consider defining, evaluating, and validating training processes and tools that can be used to support core skill development. Currently, there is scant literature outside of individual model fidelity tools that support ways in which to evaluate therapist skill development across multiple systemic family therapies (Lebensohn-Chialvo et al., 2018). Promising research processes are in place for some EBPs that could be translatable for 128 examining the methods of core skill development, e.g., Generation PMTO fidelity process skills (Forgatch & Patterson, 2010). While panelists endorsed an important set of skills specific to trauma informed practice, additional research efforts may be helpful to understand how family focused EBPs line up with the full range of trauma informed care principles that are intended to have impact at the organizational and practitioner level. Research efforts such as this could be beneficial in continuing to promote trauma informed care health care systems, as recommended by IOM (2007; NCTSN, 2012). The development of this therapist foundation skillset utilizing a common factors framework advances previous studies that have examined components of effective practice across models, without getting into specific model studies centered on fidelity. The data contributes to a clearer understanding of the range of skills EBP experts consider as important for therapists utilizing family focused EBPs in community health settings. Further research could explore models to analyze common factors and continue to refine these findings. Additional research may be applicable to studying methods for developing therapists’ engagement skills. While structural models exist for supporting engagement of families in systemic treatment (see Szapocznik et al., 2015), there is an opportunity for future research that can more clearly identify the reflexive processes and supervision methods that best support therapists learning engagement strategies. Similarly, Motivational interviewing (MI) was discussed as a training method supportive of building therapist capacity for engagement. Additional research could be beneficial to examine concurrent MI and EBP training to see if it had impact on EBP adoption with SED youth and their families. 129 One final important area for additional research is considering where does training of core skills best fit. This study examined core skills at the community health setting intersection as it relates to EBP adoption. However, participants suggested this warranted further research to see about fit for these skills to be applied perhaps earlier in clinical training. Organizational and Training Program Implications Community health organizations are able to utilize this core skillset to guide strategic planning efforts and workforce development plans. These organizations can benefit from introducing comprehensive training plans that provide core skills training either prior to, or concurrently with, EBP training. This has significant potential to help organizations improve on their EBP investment return and help assure readiness at the provider level before jumping into EBP training. This implication includes acknowledgement that the findings from this study did not endorse workshop training without opportunity for reflective supervision, consultation, or other methods to practice or process skills learned. Community health settings will likely need to consider the feasibility of adopting suggested methods of training and support to therapists learning EBPs, particularly reflective supervision practices which are already known to be stressed in community health settings (Dorsey et al., 2017; J. Bayardo, personal communication, 30 October 2018; 17 May 2019). Strategic initiatives should be incorporated to build this capacity should the organization have limited supervision capacity. This is an important consideration that has impact on therapists’ skill development and well-being (Curry & Epley, 2020) therefore should not be overlooked. In addition, organizations should commit to infrastructure that supports either live supervision or capacity to record sessions if they do not have this in place already. Time for therapists to watch 130 their own work as well as process their work with supervision is critical to growth and reflective practice (Curry & Epley, 2020). Community health organizations and clinical training programs should assess their commitment to trauma informed principles at the organizational and practitioner level. Guidance is available from organizations such as the National Childhood Traumatic Stress Network or the University of Buffalo (Koury & Green, 2017) toward these aims. This is in line with recommendations on assessing trauma informed care in human services organizations (Bassuk, Unick, Paquette, & Richard, 2017). Enhancing trauma informed care at the organizational level has positive impact on EBP adoptions (Courtois & Gold, 2009). Further, although many models are infused with trauma informed concepts, many EBPs are not which makes this more important to consider. Community health settings and clinical training programs should consider how to provide therapists with choice in training in EBPs, as participants in this study suggested that not giving participants a choice has negative consequences on the training and EBP adoption process. This is supported in the literature by Akin et al., 2014 in suggesting the importance of a good fit between therapist and EBP for successful adoption. Choice should be weighed as part of the organizational or training program mission and decided prior to the implementing an EBP, along with providing therapists expectations of the EBP. Pre-knowledge before EPB training is a supportive measure for supporting implementation activities (Pemberton et al., 2017). Based on the findings, clinical training programs and community health organizations need to be attentive building culture and promoting competency to assure family sessions are conducted when adopting family focused EBPs. This requires continued attention to helping therapists build competency around conducting family sessions and seeing presenting problems 131 through a relational lens. Clinical training programs should consider required coursework in systemic family approaches as a basic requirement for program completion to provide a base for this skillset. Attention to practice habits in conducting family therapy is important as therapists’ thoughts about the importance of family sessions and actually doing them tend not to align favorably (Oed & Gonyea, 2019). Clinical Practice, Supervision, and Training Perhaps the most overarching finding for clinical training, supervision, and practice is that therapists should learn and become proficient in the core skillset identified in this study to best promote adoption of family focused EBPs in community health settings with SED youth and their families. This is a tall order that requires many additional pieces to ensure its success. Some of the more salient implications of this study for practice, supervision, and training follow. Clinical practice. Therapists should learn core skills in assessment that promote systemic conceptualization and develop skill in observing family behavior, assessing family interactions and family culture, and conducting conducing structured assessments around risk and safety. These foundational assessment skills contribute to a much more comprehensive way of conducting assessments with families. Ongoing consultation and team support are important methods toward the aim of supporting therapists learning assessment skills (Frank et al., 2020). Therapists should have opportunities to learn how to have well-timed and integrated conversations about topics such as diversity and equity (Erolin & Wieling, 2021). This can be achieved through practice and supported in a supervision relationship. This is in line with an approach to engage diverse families in treatment and is also an important part of building and maintaining a treatment alliance (PettyJohn et al., 2020; Sori & Sprenkle, 2004). 132 Therapists should learn skill in case management, parenting support, CBT strategies or at least have access to work in partnership with colleagues who have these skills. These skills may or may not be a part of family focused EBPs, however they are foundational skills needed in working with SED families in community mental health settings skills (e.g., Goodcase et al., 2021). Skill in these areas help assure families with needs for advocacy, parenting strategies, or more specific cognitive behavioral interventions. Supervision. Therapists need to have opportunity for reflective supervision that can support their growth in reflective practice, adapting interventions, and having a safe space to practice learning new interventions (NCTSN, 2012). In order to build skills, opportunities for supervision should be made available to therapists and organizations may need to build capacity for this to happen. The use of consultation groups and peer groups to supplement learning, supervision, and personal support is suggested as helpful in supplementing reflective supervision opportunities, however reflective supervision is a critical foundation for therapist skill development and reflexive practice (Curry & Epley, 2020) Training. For therapists to be proficient in this core skillset, training efforts need to utilize methods that are rooted in adult learning principles, such as practice, rehearsal, role-play, and having meaningful opportunities reflect on their practice as they are learning. This finding is congruent with scholars who in general advocate for therapists to practice handling difficult emotions, learn to understand emotional triggers in treatment, and maintain presence in session (Aponte et al., 2009, Owen, et al., 2014; Rousmaniere, 2016). Adult learning principles also suggest preparation in learning activities for adults and training efforts (Bryan et al., 2009). Overviews of EBPs or pre-trainings may help therapists, trainers, and supervisors assess fit and readiness for EBP training. 133 Last, graduate clinical training programs should consider the use of core skills to help guide curriculum development and practicum expectations. This holds opportunity to bridge expectations from clinical training programs to post graduate employment if utilized in both settings. Training programs are often limited in delivering any or a wide range of EBPs to their students due to time or copyright limitations by model developers. However, it is not difficult for these programs to help their students achieve competencies in the skills highlighted in this study. Skills are arranged by category with statement indicators that positions them well for turning into a rubric or pre-written set of competencies or learning agreements. Implications for Future Delphi Studies Recruitment and retainment. Much of the literature on Delphi studies establishes the need for successful recruitment and retainment efforts which are key to the success of the methodology (e.g., Green et al., 1999; Keeney et al., 2011; McKenna, 1994). However, few studies discuss factors of participant engagement, or as stated by McKenna (1994) the pieces that build relationship or add personal touches that are important to recruitment efforts. For as many benefits as the Delphi offers in garnering a consensus voice (McPherson et al., 2018), understanding more about the context for why people give to the process rises in importance and can be helpful when designing future Delphi studies. Time and study design. Delphi studies are time intensive processes. More recently, research literature defines engagement in and of itself as a preliminary round of inquiry, one in which additional methodological rigor and established practices could be suggested (Hung et al., 2008). These findings have implications for considering the context of engagement and considering planning preliminary rounds that address engagement issues early. For example, Hsu & Sandford (2007) promoted at the time of their study the use of well-timed, frequent mailings 134 to help with recruitment and retainment. Even more prevalent today with the ease of electronic mail, strategies such as this can promote more efficient and effective follow up between rounds of inquiry. These results should be considered when designing a Delphi study particularly as it relates to estimating time and effort in interest of high return on participant recruitment and retention. Demographic reporting. Literature searches to date have not yielded information on established best practices for demographic reporting in Delphi studies. Studies reviewed demonstrated demographic reporting limited specifically to expert criteria (e.g., Hordijk et al., 2019; Domlyn & Wandersman, 2018), and sample demographic sheets guiding Delphi studies suggest minimal demographic information and spaces to assure expert criteria is met (Keeney et al., 2011). Yet, diversity in panels is still the best opportunity to mitigate bias, include different perspectives on inquiries and generalize results (Trvelyan & Robinson, 2015). Participant demographic collection and reporting should be considered toward both aims of assuring expert participants and attending to diverse perspectives to improve generalizability and reduce opportunity for bias. This can be an important consideration in future Delphi research designs to evaluate how providing detailed demographic information is important for the aims of the study. Final Summary/Conclusions Families need access to evidence-based care to address rising mental health needs. At the same time, there is a significant need to understand how to better support therapists in acquiring skills that support EBP adoptions, so that these practices can reach children with SED and their families in community health settings. The study need was identified in the literature as well as through a community-academic research partnership. 135 A community-engaged mixed method Delphi study provided consensus on a 175-item therapist skillset that is most likely to support implementation of family focused EBPs in community mental health settings. Consensus was gained on categories of assessment, engagement, intervention, reflexivity, adaptability, and trauma informed care. See Appendix K for the complied core therapist skillset. Methods supportive of helping therapists acquire this skillset include key indicators for preparing the training environment for success, employing training methods such as practice, video supervision, and role play, and working with EBP training challenges such as organizational commitment to training and time. With these skills and training methods identified, there are significant implications for research, community health providers, and clinical training programs. The foundation is set to continue to advance this research through a pilot study of a training intervention using this core skill set. Community health organizations can readily use these data to inform strategic planning for workforce development. And finally, clinical training programs can adopt this set of skills to inform training curriculum in graduate programs. Delphi studies are considered one of the most promising methods for finding valid and reliable group opinions and an important way to produce significant data on complex areas of health research inquiry (Keeney et al., 2011; McKenna et al., 2006). Finally, finding agreement on a therapist skillset able to promote the adoption of family focused EBPs that treat children with SED symptoms is a critical pathway to improving the sustainability of EBPs in communities (Beidas et al., 2014; Stirman et al., 2012). Efforts such as this current study hold promise toward understanding additional pieces of the research-to- practice puzzle and support improved family outcomes from treatment in community settings (Aarons et al. 2009; NIH, 2018; SAMHSA, 2020). 136 APPENDICES 137 Appendix A: Additional Table for Methods Chapter Table A.1 Family Focused EBP Models Utilized for Participant Recruitment PROGRAM RATING DESCRIPTION (CEBC, 2020) Adolescent-Focused 2, NIH Adolescent FBT’s goal is to result in positive outcomes in such areas as alcohol and drug Family Behavior use, depression, conduct problems, family dysfunction, and days absent from Therapy (Adolescent work/school. Adolescent FBT is designed to be used with youth, multiple ethnicities, FBT) differing types of substance abuse (alcohol, marijuana, and hard drugs), and across genders. Drafts of standardized client record keeping forms and quality assurance may be customized to fit agency needs. Attachment-Based 3 ABFT is an attachment-based, trauma-informed, emotion-focused intervention for youth Family Therapy with suicide, depression anxiety, and/or trauma. Treatment strengthens secure parent- (ABFT) child relationships which can reduce family conflict and buffer against stress. The model is structured yet flexible, requiring therapists to be focused as well as emotionally attuned. Adolescents with elevated depression symptoms, suicidal ideation and behavior, anxiety. Brief Strategic 2, NIH BSFT is a brief intervention used to treat adolescent drug use that occurs with other Family Therapy problem behaviors. These co-occurring problem behaviors include conduct problems at home, at school, oppositional behavior, delinquency, associating with antisocial peers, aggressive and violent behavior, and risky sexual behavior. BSFT is a family systems approach. 138 Table A.1 (cont’d) Child-Parent 2 CPP is a treatment for trauma-exposed children aged 0-5. Typically, the child is seen Psychotherapy with his or her primary caregiver. CPP examines how the trauma and the caregivers’ relational history affect the caregiver-child relationship and the child’s developmental trajectory. A central goal is to support and strengthen the caregiver-child relationship as a vehicle for restoring and protecting the child’s mental health. Functional Family M+, 2 FFT is a family intervention program for dysfunctional youth with disruptive, Therapy (FFT) externalizing problems. FFT has been applied to a wide range of problem youth and their NIH families in various multi-ethnic, multicultural contexts. Target populations range from at- risk pre-adolescents to youth with moderate to severe problems such as conduct disorder, violent acting-out, and substance abuse. Generation PMTO M and 1 GenerationPMTO (PMTO®) is a parent training intervention that can be used in family contexts including two biological parents, single-parent, re-partnered, grandparent-led, reunification, adoptive parents, and other primary caregivers. This behavioral family systems intervention can be used as a preventative program and a treatment program. GenerationPMTO interventions have been tailored for specific child/youth clinical problems, such as externalizing and internalizing problems, school problems, antisocial behavior, conduct problems, deviant peer association, theft, delinquency, substance abuse, and child neglect and abuse. Multidimensional 1, NIH MDFT is a family-based treatment for adolescent substance use, delinquency, and other Family Therapy behavioral and emotional problems. Therapists work simultaneously in four interdependent domains: the adolescent, parent, family, and community. Multisystemic M+, 1 MST is a juvenile crime prevention program designed to improve the real-world Therapy (MST) functioning of youth by changing their natural settings - home, school, and neighborhood NIH Adolescent - in ways that promote prosocial behavior while decreasing antisocial behavior. 139 Table A.1 (cont’d) Multisystemic M, 1 MST – PSB is a juvenile sex offender treatment program designed to reduce criminal and Therapy (MST) antisocial behavior, especially problem sexual behavior, by providing intensive family Problem Sexual therapy services in the youth' natural environment over a 5-7-month period. Behavior Parent-Child 1 Parent-Child Interaction Therapy (PCIT) is a dyadic behavioral intervention for children Interaction Therapy (ages 2 - 7 years) and their parents or caregivers that focuses on decreasing externalizing child behavior problems (e.g., defiance, aggression), increasing child social skills and cooperation, and improving the parent-child attachment relationship. Trauma-Focused 1, NIH TF-CBT is a conjoint child and parent psychotherapy model for children who are Cognitive experiencing significant emotional and behavioral difficulties related to traumatic life Behavioral Therapy events. It is a components-based hybrid treatment model that incorporates trauma- sensitive interventions with cognitive behavioral, family, and humanistic principles. Triple P (level 4) 1, P Level 4 Triple P is one of the five levels of the Triple P - Positive Parenting Positive Parenting Program® System and is intended for moderate to severe behavior in children. Level 4 Program Triple P helps parents learn strategies that promote social competence and self-regulation in children as well as decrease problem behavior. Rating key: Blueprints Ratings: M+ = Model plus (highest rating), M = Model rating, P = Promising rating California Evidence-Based Clearinghouse: 1 = well supported by evidence, 2 = supported by research evidence, 3 = promising research evidence National Institute of Health – National Institute on Drug Abuse: NIH = noted as an endorsed EBP 140 Appendix B: Sample Recruitment Email My name is Debra Miller, and I am a doctoral candidate at Michigan State University in East Lansing, MI. With the support of my advisors Dr. Adrian Blow and Dr. Kendal Holtrop, I am conducting a research study entitled: Core Therapist Skills Supporting Implementation of Evidence-Based Practices (EBP) with Severe Emotionally Disturbed (SED) Youth in Community Mental Health Settings: A Mixed-Method Delphi Study, in partial fulfillment of the requirements for the degree of Doctor of Philosophy. I am specifically reaching out to you as an expert in a family focused EBP that targets severe emotional disorders/symptoms in youth, due to your work with [EBP]. [EBP] was one of 12 models that came up in my systematic search of highly ranked, family-focused EBPs that target severe emotional symptoms in youth. The purpose of this study is to identify and gain consensus on the foundational skillset therapists need to have to better support EBP adoption in community mental health practice. An additional aim of this study is to summarize this skillset and propose a set of training methods that can better support EBP implementations. I am recruiting up to five experts that can represent [EBP] in my study. I am aiming for a diverse mix of master trainers, experienced supervisors, consultants, implementation coordinators and/or model developers - preferably with implementations in public mental health settings. Each round of the Delphi should require no more than 20-30 minutes of a participant’s time with exception of the optional focus group at the end of the study. I am providing gift cards for reimbursement of time and effort (up to $150). Would you be willing to participate in and/or identify some additional participants to represent [EBP]? I will be emailing survey links to identified participants in the next 2-3 weeks. If you have anyone you can recommend, please connect me with their email address so I can introduce myself and start the process. Study is determined exempt per MSU IRB and I can forward the informed consent document to you per your request. Thanks in advance for your support. I look forward to having [EBP] represented as a part of this study! 141 Delphi Round 2- Core Therapist Skills Study Information Dear Participant, Thank you for your continued support of this research study. Round 2 provides a series of statements reflecting the core therapist skills you identified in Delphi Round 1 as important to supporting evidence-based practice implementation for families with children experiencing severe emotional disturbance symptoms (SED). The goal for Round 2 is to build consensus on the therapist skillset most important to consider as it relates to evidence-based practice implementation. We recognize that many of the statements you will see in this round are highly important to clinical practice with families. However, we ask that you consider endorsing the statements you believe are the best reflection of the core skills necessary for evidence-based practice implementation with SED youth and their families. Instructions: 1. For each of the statements listed, please rank your level of agreement on a 7-point Likert scale. While items require a response to move to the next set of statements, a middle neutral option is available for statements you are less certain about endorsing. In addition, there is an optional comment box at the end of each section where you can include any additional thoughts you may have. You can also use this text box to indicate if there are any statements you did not understand fully. 2. Please proceed through all six categories. Categories are bolded and reference a number of sections that follow. This is intentional to break up the statements into categorical areas and hopefully decrease survey fatigue. The estimated time for completion is 26-45 minutes according to Qualtrics and our testers. 3. Your survey is set to auto-save through your web browser. If it is helpful, you are welcome to start the survey and come back to it later. Please make sure if you leave the survey and return that you are using the same web browser. Breaks are ok and encouraged to stay fresh in reviewing. Do this as many times as you need. Thank you again for participating in this important work. Now, let’s begin! 142 Appendix C: Informed Consent Form for Participants 18 years and older Dear Participant, My name is Debra Miller, and I am a doctoral candidate at Michigan State University in East Lansing, MI. With the support of my advisors Dr. Adrian Blow and Dr. Kendal Holtrop, I am conducting a research study entitled: Core Therapist Skills Supporting Implementation of Evidence-Based Practices (EBP) with Serious Emotionally Disturbed (SED) Youth in Community Mental Health Settings: A Mixed-Method Delphi Study in partial fulfillment of the requirements for the degree of Doctor of Philosophy - Department of Human Development and Family Studies. As you may know, Delphi studies engage expert input on research topics of interest. I am specifically reaching out to you as an expert in a family focused EBP that targets serious emotional disorders/symptoms in youth. The purpose of this study is to identify and gain consensus on the foundational skillset therapists need to have to better support EBP adoption in community mental health practice. An additional aim of this study is to summarize this skillset and propose a set of training methods that can better support EBP implementations. Thank you for considering participating in this study. Your participation will involve taking part in three, possibly four rounds of surveys. Expectations for each round and anticipated time frame are as follows: • Round 1: Provide thorough, narrative responses to three open-ended question regarding skills you identify as important for therapists to have working with families and SED youth (approximately 30 minutes). • Round 2: Provide ratings on a series of statements generated from the responses in Round 1, based on a 5-point Likert scale. You will have an optional opportunity to comment on your ranking choices for each statement (20-30 minutes). • Round 3 (tentative): This round may be needed to further narrow consensus on statements from Round 2. If needed, you will be asked to provide agreement ratings on a series of statements generated from the responses in Round 2, based on a 5-point Likert scale. You will also be given information on the median responses of statements ranked in Round 2 to help you decide your rankings (20-30 minutes). • Round 4: This round is a small focus group of 2-5 participants where you will be asked to discuss the consensus statements and how they translate to training methods most likely to support therapists learning EBPs. Focus groups will be conducted via Zoom and last 60-90 minutes. Delphi surveys are usually fully anonymous. This is true for this study with the exception of the optional Round 4 in which you will be participating with other EBP experts. You can choose to participate or not in any of the study components and withdraw your participation at any time. Links to the survey will be sent via email and data will be collected and stored on Qualtrics, a secure survey platform stored on the MSU server. Only the principle investigators for this study (as named in this consent) will have access to the data. Names will not be collected as part of the 143 survey process and a unique code identifier will be used. In addition, demographic information compiled will be identified only in summary form, such as ranges of responses and means. Specific EBP models will not be identified by name in the results. There is no foreseeable risk to you participating in this study. However, we recognize there is a significant commitment of time on your part. For your participation, we will be offering $150 in gift cards (one for completion of each round x three rounds). This is to thank you for your efforts in this project. If you have questions concerning the study, now or at any time, please reach me at (989) 859- 3096 .Further, if at any time you have questions or concerns about your rights as a research participant or would like to offer input or register a complaint about this study you may contact the Michigan State University Research Protection Program at (517)355-2180; FAX: (517)432- 4503; email: irb@ora.msu.edu; or postal mail at 4000 Collins Rd., Suite 136, Lansing MI, 48910. By clicking the button below, you indicate your voluntary agreement to participate in this series of online surveys. 144 Appendix D: Additional Figure for Methods Chapter Figure D.1 Delphi Process Flow with Community-Engaged Scholarship Study Validations and Research Questions EBP Expert Selection Qualitative Questions (RQ1) Literature Round 1 Review Community-Academic Partner Relationship (RQ1) Quantitative Survey (RQ2) Round 2 Survey Revised No and Administered Consensus? (Round 3) Yes Propose Therapist Skillset Qualitative Focus Group (RQ3) Round 3 Propose Training Methods 145 Appendix E: Round 1 Open-ended Questions Welcome to Round 1 and thank you for your participation. I am seeking your expert feedback on the breadth and depth of therapist skills you see as an important foundation for therapists to have when they are learning and adopting systemically- oriented evidence-based practices (EBPs) that work with family systems. Your expertise is an important part of this study. Please answer the questions in reflection of your experiences training, supervising, and consulting with therapists learning EBPs that treat serious emotional disturbance (SED) symptoms in youth within their family system. You will be asked three questions. Each are free-writing activities intended to generate as many possibilities as you can think of for each question. In both questions, you will be asked to list and describe a set of therapist skills. Please note, your answers to Question 1 may have some overlap to the skills you list in Question 2. If this happens, this is ok. The lists you generate may include basic interpersonal skills, common therapeutic skills, or even more advanced theoretical application skills. All answers are helpful. Question 1: Considering your own EBP model and other family-based EBPs you are familiar with, what are the key skills you believe therapists need to learn to be successful working with families with children experiencing serious emotional disturbances? Please list and briefly describe these skills. [Expert response here] Question 2: Finish this sentence: When I think about therapists who were successful learning my EBP, I recall that they came into my training with the following foundational skills or competencies. Please also consider family diversity and cultural competency as part of this question. Please list and briefly describe these skills. [Expert response here] Question 3: Separate from the skills and competencies listed above, what are the challenges you see to therapist adoption of EBP? Please briefly list and describe these challenges. [Expert response here] 146 Appendix F: Demographic Information (Qualtrics) (All open text boxes) Evidence-based practice name: Implementations in USA? International? Gender identity: Age: Race/ethnicity: Degree and license: Years of experience with specific EBP: Years of experience (total): All roles with EBP: Main geographic implementation area: Main role identification (check one): -Model developer -Master Trainer -Implementation Coordinator -Consultant -Supervisor -Other (please describe) 147 Appendix G: Directions and Sample Survey Delphi Round 2- Core Therapist Skills Study Information Dear Participant, Thank you for your continued support of this research study. Round 2 provides a series of statements reflecting the core therapist skills you identified in Delphi Round 1 as important to supporting evidence-based practice implementation for families with children experiencing severe emotional disturbance symptoms (SED). The goal for Round 2 is to build consensus on the therapist skillset most important to consider as it relates to evidence-based practice implementation. We recognize that many of the statements you will see in this round are highly important to clinical practice with families. However, we ask that you consider endorsing the statements you believe are the best reflection of the core skills necessary for evidence-based practice implementation with SED youth and their families. Instructions: 1. For each of the statements listed, please rank your level of agreement on a 7-point Likert scale. While items require a response to move to the next set of statements, a middle neutral option is available for statements you are less certain about endorsing. In addition, there is an optional comment box at the end of each section where you can include any additional thoughts you may have. You can also use this text box to indicate if there are any statements you did not understand fully. 2. Please proceed through all six categories. Categories are bolded and reference a number of sections that follow. This is intentional to break up the statements into categorical areas and hopefully decrease survey fatigue. The estimated time for completion is 26-45 minutes according to Qualtrics and our testers. 3. Your survey is set to auto-save through your web browser. If it is helpful, you are welcome to start the survey and come back to it later. Please make sure if you leave the survey and return that you are using the same web browser. Breaks are ok and encouraged to stay fresh in reviewing. Do this as many times as you need. Thank you again for participating in this important work. Now, let’s begin! 148 Therapist Assessment Skills (7 sections) Neither Section 1: Strongly Somewhat Agree Strongly Disagree Somewhat Agree Observing Disagree (2) Disagree nor Agree (5) (6) Agree Family (1) (3) Disagree (7) Behavior (4) Therapists need to... ... be keen observers of the behavior and communications o o o o o o o of their clients. (1) ... have the ability to identify critical relational processes in the o o o o o o o moment during sessions. (2) ... be able to observe how family members interact and use these observations to identify areas of o o o o o o o strength and struggle within the family structure. (3) ... be able to identify family interactions that maintain problems within o o o o o o o the family system. (4) 149 In general, skill in observing family behavior o o o o o o o is important. (5) Q36 Any comments on Observing Family Behavior statements? ________________________________________________________________ 150 Appendix H: Full Interview Guide Round 3 *Italics note scripted content read directly to participants Introductions (5 minutes) a. Facilitators introduce themselves briefly. b. We would like to offer the chance for you to introduce yourself briefly to the group. We say briefly as we have two questions to get to today, which doesn’t sound like a lot, but our experience so far tells us you all have amazing things to say and we want to be sure we get to all of it. Please include your name, the model you identify with, and tell us a little bit about why you prioritized participating in this study? Question 1 (25 minutes): PURPOSE: The purpose of this focus group today is to discuss the consensus areas and the corresponding training methods that can best facilitate skill development for therapists who are preparing to learn evidence-based practices targeted to treat severe emotional disturbance (SED) symptoms in families in community mental health settings. We know that EBP implementation is a challenge, and that therapist skills play a part in this big picture. Often therapists come into EBP trainings with a variance of skills or competencies, unfortunately sometimes missing pieces that are foundational to work with SED youth and families. In order to maximize the impact of EBP training and implementation, we need your help to determine the best training methods to address consistency with these skills before they get to your EBP training. Round 2 yielded consensus in 6 categories of skills. We will review the consensus items via screen sharing and ask that you to discuss the following question: How do you think therapist acquisition of these skills could be best accomplished so that they are ready for your EBP? What are the methods or strategies you would use? a. Assessment b. Engagement 1. Anything more specific about engaging families about race/racism, diversity, or equity? c. Intervention 1. Anything more specific about reframing negative thoughts to more positive ones? d. Reflexivity e. Adaptability 151 1. Anything more specific for tailoring EBPs to unique families? Prompt if this does not emerge. f. Trauma informed care Question 2 (20 minutes): Much of the consensus on therapist skills needed for treating SED youth and their families involved skills that are not easily teachable, didactic skills. For example, this may be skill in “establishing a therapeutic bond” or “being able to hear feedback and try new strategies.” While we (trainers, supervisors, implementors, etc.) play a role in the success of teaching skills to therapists in training, we also know that who the therapists are plays a role in skill acquisition. With this in mind, we ask you to discuss the following: How do you teach core therapist skills that are fundamentally interactive or reflexive? AND How do these skills impact your assessment of fit when looking at readiness for starting your EBP training? Question 3 (time permitting): There were four items that did not reach consensus and were heavily supported in the literature as key to EBP implementation among varying treatment models. Please discuss the importance of these items and why you think they did not have consensus? Closing (3-5 minutes): Are there any other comments or lingering thoughts you have regarding this study or the information discussed today? Thank you for your time and participation! Gift cards will be emailed within 24 hours. 152 Appendix I: Round 3 Informed Consent Thank you for your participation in Round 3. My name is Debra Miller, and I am a doctoral candidate at Michigan State University in East Lansing, MI. With the support of my advisors Dr. Adrian Blow and Dr. Kendal Holtrop, I am conducting a research study entitled: Core Therapist Skills Supporting Implementation of Evidence-Based Practices (EBP) with Severe Emotionally Disturbed (SED)Youth in Community Mental Health Settings: A Mixed-Method Delphi Study, in partial fulfillment of the requirements for the degree of Doctor of Philosophy - Department of Human Development and Family Studies. The purpose of this study is to identify and gain consensus on the foundational skillset therapists need to have to better support EBP adoption in community mental health practice and to propose a set of training methods that can better support EBP implementations. While you have prior indicated your consent to participate in this study at the beginning of Round 1, we would like to assure that we continue to have your consent for the focus group (Round 3) portion. Your participation in Round 3 will allow you the opportunity to discuss the consensus statements and how they translate to training methods most likely to support therapists learning EBPs among other EBP experts participating in this study. Unlike other rounds, your anonymity can no longer be kept within your focus group. Your consent is indicated through your participation today. You are free to leave the call at any time without penalty. This focus group will last 60 minutes and will be recorded to help us best capture and analyze the items discussed. Only the investigators for this study (as named in this consent) will have access to the recorded data. Specific EBP models or individuals representing the models in the focus group will not be identified in the results of the study in such a manner that would allow for identification of a participant’s name to the responses. There is no foreseeable risk to you participating in this study. However, in recognition of your time to this study, we are offering a $50 Amazon gift card. Amazon will receive your email address for delivery of this gift card. You have already consented to this via email, however if you have changed your mind, please let me know. If you have questions concerning the study, now or at any time, please reach me at (989) 859- 3096, or thoma328@msu.edu. Further, if at any time you have questions or concerns about your rights as a research participant or would like to offer input or register a complaint anonymously about this study, you may contact the Michigan State University Research Protection Program at (517)355-2180; FAX: (517)432-4503; email: irb@ora.msu.edu; or postal mail at 4000 Collins Rd., Suite 136, Lansing MI, 48910. 153 Appendix J: Additional Table for Results Chapter Table J.1 Consensus Categories and Sub-Headings (Therapist Skillset) ASSESSMENT SKILLS ENGAGEMENT SKILLS INTERVENTION SKILLS • family culture • establishing a • maintaining treatment • assessing family therapeutic bond alliance history • build empathic • treatment planning • systemic response • process-focused conceptualization • engaging families in treatment skills • behavior treatment* • conflict management • observing family • facilitating agreement skills behavior on therapeutic goals • session management • conducting • enhancing client skills structured motivation • therapist organizational assessment skills • skills in reframing* TRAUMA INFORMED • facilitating change REFLEXIVITY SKILLS CARE PRACTICE • working with emotion • reflective practice • conducting trauma- • skills associated with • managing own self- informed practice cognitive behavioral care theoretical approaches • awareness of self of • skills in family therapy the therapist issues • knowledge of parenting approaches • active teaching skills ADAPTABILITY • skills in case SKILLS management • tailoring evidence- based practices to unique families * Denote subcategories that had • therapist ability to some statements with non- be flexible consensus. These were included as additional prompts for participants in the interview guide. 154 Appendix K: Additional Table for Discussion and Implications Chapter Table K.1 Summarized Therapist Skillset Core Therapist Skillset Combined All Categories Assessment Skills (n=29) Be keen observers of the behavior and communications of their clients. Have the ability to identify critical relational processes in the moment during sessions. Be able to observe how family members interact and use these observations to identify areas of strength and struggle within the family structure. Be able to identify family interactions that maintain problems within the family system. In general, skill in observing family behavior is important. Assess the family based on their interactions. See beyond the presenting issues and consider drivers of symptoms rather than just the “tip of the iceberg.” Recognize the positive aspects of the caregiver/child relationship (e.g., "Look at the big smile on her face when you did that!). Assess all ecological factors contributing to identified problems in the family system. In general, assessing family interactions is important. Identify gaps in cultural knowledge. Assess and understand a family's unique cultural identity. Apply a sociocultural lens to assessment practices and be skilled in cultural humility. In general, the skill of assessing family culture is important. Assess for historical trauma in family systems. Understand a family’s history and recognize how it impacts the family's current circumstances and perspectives. In general, the skill of assessing family history is important. Conceptualize behaviors, interactions, and interventions. Comprehend and conceptualize underlying causes of problems and facilitate solutions based on these. Think systemically (not linearly). 155 Table K.1 (cont’d) Demonstrate assessment skills that include multiple perspectives and multiple domains of the person's ecology. In general, the skill of systemic conceptualization is important. Understand how behaviors are reinforced or extinguished. In general, assessment of behavior is important. Conduct a basic assessment that includes a risk assessment. Assess (initially and ongoing) for safety with children with severe emotional disturbances. Assess for trauma symptoms and they should do this with each family they work with. Utilize assessment of clinical concerns to develop treatment plans based on prioritization of needs. In general, skill in conducting structured assessment is important. Family Engagement Skills (n=35) Establish rapport with difficult clients. Talk to individuals with a non-biased approach and listen non-judgmentally. Relate to the concerns of all family members. Support and encouraging families. Demonstrate skill in the use of humor. Join systemically and strategically. Match to youth and to the adult members of their family in ways that are authentic. Use self-disclosure when appropriate for joining purposes. Demonstrate skill in both verbal and non-verbal communication. Move from an individual, deficit-based focus to a relational, strength-based focus. Relate to all ages, genders, family roles, sexual preferences, belief-systems, and styles with acceptance and understanding. In general, skill in establishing a therapeutic bond is important. Embody the emotions the client is verbalizing or communicating non-verbally. Appreciate and sincerely reflect concern for others. Be sensitive to each family’s unique context. Empathize with the difficulty of parenting a child with significant behavioral challenges. In general, the ability to build empathic response is important. 156 Table K.1 (cont’d) Build and maintain multiple alliances (e.g., with the child and parents). Demonstrate skill in empowering family members and not take sides. Engage new support persons when they are added to the treatment team. Engage families and youth with a history of failed therapy attempts. Engage multi-stressed families. Engage younger kids. Engage families from a diverse range of ethnic/racial, geographic, and socioeconomic backgrounds. Engage other family members needed for treatment that are not present from the start. In general, skill in engaging families in treatment is important. Align families around treatment goals. Connect problems identified in the family and how the treatment will address those problems in clear, concise, and easy to understand terms. Motivate all family members to accomplish goals that positively impact the entire system. In general, facilitating agreement on therapeutic goals is an important skill. Demonstrate skill in basic motivational interviewing (open ended questions, meeting the clients where they are, reflections, asking permission, etc.) to increase parent involvement Enhance motivation in adolescents who are not otherwise motivated to attend therapy. Enhance each family member’s own motivation for change. Motivate all family members to accomplish goals that positively impact the entire system. In general, skill in enhancing client motivation is important. Intervention Skills (n=76) Understand family culture and weave it into their treatment plan. Demonstrate skill in developing a plan for treatment interventions based on a systemic case conceptualization. Demonstrate skill in facilitating a family centered approach to treatment planning. Develop treatment plans based on assessment of clinical concerns and prioritization of those concerns. In general, skills in treatment planning are important. 157 Table K.1 (cont’d) Demonstrate skill in describing the impact of positive parent-child interactions in real time (e.g.,"Look at the big smil on her face when you did that!) Coach the family to engage in different ways of communicating and behaving with one another. Know when they are being pulled into the middle of the family's conflict. Demonstrate skill in noticing feelings in the moment and then helping parents and children make sense of those feelings. Help family members communicate clearly without putting self or others down. Demonstrate skill in and have desire to intervene with families directly. In general, process-focused treatment skills are important. Demonstrate skill in being directive with both parents and children to contain situations that could potentially get out of hand (e.g., a child becoming aggressive in session. Demonstrate skill in conflict resolution and diplomacy to help families deal with conflicts they experience. Demonstrate skill in conflict resolution and diplomacy to support families with systems such as probation, neighbors, school staff, etc. Mange conflict when it arises in session so that it can be targeted for change. In general, conflict management skills are important. Demonstrate preparedness to teach or implement their plan for the session. Demonstrate good time management skills. Demonstrate the ability to follow and maintain a session structure. Manage multiple people in session. Set session agendas responsive to the family’s needs. Take leadership in a session by asserting themselves when necessary. Demonstrate exemplary listening skills balanced with the ability to supportively interrupt. Directly address session goals. In general, session management skills are important. Balance time for sessions, contact, and assuring documentation is complete. Manage urgent needs with families as they arise. 158 Table K.1 (cont’d) Demonstrate skill in structuring the therapeutic relationship by being reliable, i.e., showing up on time, contacting family with any changes needed. In general, therapist organizational skills are important. Reconnect with family members who are challenged in a session. Demonstrate skill in acknowledging and working with resistance when it occurs. Balance alignment throughout a session. In general, skill in maintaining treatment alliance is important. Effectively manage negative affect expressed by parents. Deliver new perspectives to family members in a manner that influences their willingness to see themselves and each other in new or more hopeful ways. Demonstrate skill in systemic reframing, i.e., providing a more adaptive version of the family’s view of the presenting problem. Demonstrate skill in conducting relational reframes early in treatment to help engage the whole family in treatment. Identify underlying relational patterns within the family in a way that does not shame or judge and instead provides hope or worthiness. Demonstrate skill in normalizing family situations. In general, skills in reframing are important. Facilitate conversations between family members when appropriate. Balance amplification of distress and generation of hope to increase motivation to change in the session. Help youth and parents identify and work through the blocks or barriers to adopting new behaviors. Demonstrate skill in the use of enactments, i.e., guiding and directing family members to have new (more adaptive) experiences of each other in session. Use a range of interpersonal (warmth) and directive (teaching) strategies in sessions to achieve changes. Motivate clients through questioning boundaries and setting expectations. Ask youth and family members to interact differently with each other. In general, skill in facilitating change is important. Use emotions skillfully to prompt change. Manage, evoke, and leverage emotional intensity for change. Elicit feelings of understanding, love, and connection. 159 Table K.1 (cont’d) Create or enhance a healthy emotional experience for a youth and their family. Create emotional healing experiences in session. In general, skills in working with emotion are important. Use descriptive praise - praising what exactly is liked about actions or thought processes of family members. Engage family members in basic cognitive-behavioral coping skills. Engage family members in basic problem-solving skills. Engage clients in regulation skills, such as mindfulness and anxiety reduction strategies. Demonstrate skill in specific CBT approaches, e.g., Dialectical Behavioral Therapy, Cognitive Restructuring. In general, skills associated with cognitive behavioral theoretical approaches are important. Conducting a family therapy session. Promote the parent/caregiver as the agent of change for the child. In general, skills in family therapy are important. Demonstrate skill in basic parenting knowledge ready to use in sessions, e.g., how to help parents decide on rules, or incentives and consequences that will work for their family. Coach parents in the moment on handling parenting situations. Share knowledge of positive parenting practices. Implement relevant behavior-management skills, which are often needed for children presenting with severe emotional disturbances. In general, knowledge of parenting approaches is important. Demonstrate skill in active teaching that engages parents in the learning process. Demonstrate skill in effectively using active teaching strategies such as role play, brainstorming, eliciting, movement in session. In general, active teaching skills are important. Demonstrate general skills in case management, including an ability to advocate for families. Demonstrate skills in advocacy with regard to oppression, racism, or inequities experienced by family system. Coordinate care and collaborate with other service systems. In general, skills in case management are important. Reflexivity Skills (n=22) Be reflective about the meaning of their work. 160 Table K.1 (cont’d) Understand the impact they are having on the family relationship in treatment. Be reflective about relational and sociocultural influences on families. Take perspective with clients. Evaluate whether their clinical strategies are effective. In general, skills in reflective practice are important. Be aware of their biases. Understand their physiological, affective, and cognitive responses to their clients. Be reflective about how their work impacts them. Be aware of their own emotional wounds and potential triggers. Hear supervision feedback and try suggestions given. Understand their resistance to supervision feedback or opportunities for clinical growth. Understand their own social location (i.e., their experiences with power, privilege, and subjugation as it relates to race, class, gender, sexual orientation Acknowledge their limited understanding of interactions between family members based on cultural and family norms. Understand how their diversity (i.e., identity and differences) impacts their work. Understand how their own culture (i.e., language, customs, beliefs, and values) impacts their practice. In general, awareness of self of the therapist issues is important. Manage their self-care while in session, at work, and at home. Demonstrate knowledge of trauma responses and how to ground themselves. Maintain a calm presence in the presence of chaos or conflict. Self-regulate their own emotions. In general, skills in managing own self-care are important. Adaptability Skills (n=9) Tailor (not modify) evidence-based practices in order to best engage families in treatment. Assess family cultural identity and subsequently tailor (not modify) the evidence-based practices Creatively tailor intervention strategies to specific families and situations. Understand the difference between tailoring and modifying an evidence-based practice. 161 Table K.1 (cont’d) Adapt treatment to the family based on their unique strengths and difficulties. In general, skill in tailoring evidence-based practices to unique families is important. Adjust their therapeutic approach when needed. Incorporate new skills into their therapeutic skillsets. In general, therapist ability to be flexible is important. Trauma Informed Care Practice (n=5) Talk about difficult and complex issues such as trauma, neglect, abuse, oppression, and power with families. Demonstrate skill in recognizing, naming, and working with families around the family's trauma experiences. Demonstrate working knowledge of trauma reactions and how to help clients regulate their emotions. Ground themselves when working with family trauma. In general, skill in conducting trauma-informed practice is important. 162 REFERENCES 163 REFERENCES Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Mental health services research, 6(2), 61-74. doi:10.1023/B:MHSR.0000024351.12294.65 Aarons, G. A., & Sawitzky, A. C. (2006). Organizational culture and climate and mental health provider attitudes toward evidence-based practice. Psychological Services, 3(1), 61–72. doi:10.1037/1541-1559.3.1.61 Aarons, G. A., Wells, R. S., Zagursky, K., Fettes, D. L., & Palinkas, L. A. (2009). Implementing evidence-based practice in community mental health agencies: A multiple stakeholder analysis. American Journal of Public Health, 99(11), 2087-2095. doi:10.2105/AJPH.2009.161711 Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4-23. doi:10.1007/s10488-010-0327-7 Adler, M. & Ziglio. E. (1996). Gazing into the oracle: The Delphi Method and its application to social policy and public health. London: Jessica Kingsley Publishers. Akin, B. A., & McDonald, T. P. (2018). Parenting intervention effects on reunification: A randomized trial of PMTO in foster care. Child abuse & neglect, 83, 94-105. Akin, B. A., Mariscal, S. E., Bass, L., McArthur, V. B., Bhattarai, J., & Bruns, K. (2014). Implementation of an evidence-based intervention to reduce long-term foster care: Practitioner perceptions of key challenges and supports. Children and Youth Services Review, 46, 285-293. Akins, R. B., Tolson, H., & Cole, B. R. (2005). Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Medical Research Methodology, 5, 37. doi:10.1186/1471-2288-5-37 Alisic, E., Zalta, A. K., Van Wesel, F., Larsen, S. E., Hafstad, G. S., Hassanpour, K., & Smid, G. E. (2014). Rates of post-traumatic stress disorder in trauma-exposed children and adolescents: meta-analysis. The British Journal of Psychiatry, 204(5), 335-340. doi:10.1192/bjp.bp.113.131227 Allan, R., Ungar, M., & Eatough, V. (2018). “Now I Know the Terrain”: Phenomenological exploration of CFTs learning an evidence-based practice. Contemporary Family Therapy, 40(2), 164-175. doi:10.1007/s10591-017-9448-4 164 Allen, B., & Johnson, J. C. (2012). Utilization and implementation of trauma-focused Cognitive– Behavioral therapy for the treatment of maltreated children. Child Maltreatment, 17(1), 80-85. doi:10.1177%2F1077559511418220 American Psychiatric Association (2013). Diagnostic and statistical manual of mental disorders: DSM-5. Arlington, VA. American Psychological Association Center for Workforce Studies (2018). How diverse is the psychology workforce? Retrieved from: https://www.apa.org/monitor/2018/02/datapoint. American Psychological Association Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285. doi:10.1037/0003-066X.61.4.271 American Psychological Association Task Force on Evidence-Based Practice with Children and Adolescents. (2008). Disseminating evidence-based practice for children and adolescents: A systems approach to enhancing care. Retrieved from https://www.apa.org/pi/families/resources/task-force/evidence-based Aponte, H. J., Powell, F. D., Brooks, S., Watson, M. F., Litzke, C., Lawless, J., & Johnson, E. (2009). Training the person of the therapist in an academic setting. Journal of Marital and Family Therapy, 35(4), 381-394. doi:10.1111/j.1752-0606.2009.00123.x Balazs, C. L., & Morello-Frosch, R. (2013). The three Rs: How community-based participatory research strengthens the rigor, relevance, and reach of science. Environmental Justice, 6(1), 9-16. doi:10.1089/env.2012.0017 Barwick, M. A., Boydell, K. M., Stasiulis, E., Ferguson, H. B., Blase, K., & Fixsen, D. (2008). Research utilization among children's mental health providers. Implementation Science, 3(1), 19. doi:10.1186/1748-5908-3-19 Bassuk, E.L., Unick, G.J., Paquette, K., & Richard, M.K. (2017). Developing and instrument to measure organizational trauma-informed care in human services: The TICOMETER. Psychology of Violence, 7(1), 150-157. Bateson. C. (1979). Mind and nature: A necessary unity. New York: Dutton. Bearman, S. K., Weisz, J. R., Chorpita, B. F., Hoagwood, K., Ward, A., Ugueto, A.M., & Bernstein, A. (2013). More practice, less preach? The role of supervision processes and therapist characteristics in EBP implementation. Administration and Policy in Mental Health, 40, 518–529. doi:10.1007/s10488-013-0485-5 165 Becker, K. D., & Stirman, S. W. (2011). The science of training in evidence-based treatments in the context of implementation programs: Current status and prospects for the future. Administration and Policy in Mental Health and Mental Health Services Research, 38, 217–222. doi:10.1007/s10488-011-0361-0 Beidas, R. S., & Kendall, P. C. (2010). Training therapists in evidence‐based practice: a critical review of studies from a systems‐contextual perspective. Clinical Psychology: Science and Practice, 17(1), 1-30. doi:10.1111/j.1468-2850.2009.01187.x Beidas, R. S., Edmunds, J., Ditty, M., Watkins, J., Walsh, L., Marcus, S., & Kendall, P. (2014). Are inner context factors related to implementation outcomes in cognitive-behavioral therapy for youth anxiety? Administration and Policy in Mental Health, 41, 788–799. doi:10.1007/s10488-013-0529-x Bishop, D. V., Snowling, M. J., Thompson, P. A., Greenhalgh, T., & Consortium, C. (2016). CATALISE: A multinational and multidisciplinary Delphi consensus study. Identifying language impairments in children. PLOS ONE, 11(7), e0158753. doi:10.1371/journal.pone.0168066 Bisson, J. I., Tavakoly, B., Witteveen, A. B., Ajdukovic, D., Jehel, L., Johansen, V. J., Nordanger, D., Garcia, F.O., Punamaki, R., Schnyder, U., Wittmann, L., Olff, M. & Sezgin, A. U. (2010). TENTS guidelines: development of post-disaster psychosocial care guidelines through a Delphi process. The British Journal of Psychiatry, 196(1), 69-74. doi:10.1192/bjp.bp.109.066266 Blow, A. J. (1999). Common factors across theories of marriage and family therapy: A modified delphi study (Order No. 9951917). Available from Dissertations & Theses @ CIC Institutions; ProQuest Dissertations & Theses Global. (304522028). Retrieved from http://ezproxy.msu.edu/login?url=https://www.proquest.com/dissertations- theses/common-factors-across-theories-marriage-family/docview/304522028/se- 2?accountid=12598 Blow, A. J., & Sprenkle, D. H. (2001). Common factors across theories of marriage and family therapy: A modified Delphi study. Journal of Marital and Family Therapy, 27(3), 385- 401. doi:https://doi.org/10.1111/j.1752-0606.2001.tb00333.x Blow, A. J., Sprenkle, D. H., & Davis, S. D. (2007). Is who delivers the treatment more important than the treatment itself? The role of the therapist in common factors. Journal of Marital and Family Therapy, 33, 298-317. doi:10.1111/j.1752-0606.2007.00029.x Blueprints for Healthy Youth Development. (2020). Retrieved from: https://www.blueprintsprograms.org/programs-search. 166 Bond, G. R., Drake, R. E., McHugo, G. J., Peterson, A. E., Jones, A. M., & Williams, J. (2014). Long-term sustainability of evidence-based practices in community mental health agencies. Administration and Policy in Mental Health and Mental Health Services Research, 41(2), 228-236. doi:10.1007/s10488-012-0461-5 Bowers, N. R., & Bowers, A. (2017). General systems theory. Social work treatment: Interlocking theoretical approaches, 240. New York: Oxford. Braun, V., & Clarke, V. (2012). Thematic analysis. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbooks in psychology®. APA handbook of research methods in psychology, Vol. 2. Research designs: Quantitative, qualitative, neuropsychological, and biological (p. 57–71). American Psychological Association. doi:10.1037/13620-004 Briggs, E. C., Greeson, J. K. P., Layne, C. M., Fairbank, J. A., Knoverek, A. M., Pynoos, R. S. (2012). Trauma exposure, psychosocial functioning, and treatment needs of youth in residential care: Findings from the NCTSN core data set. Journal of Child & Adolescent Trauma, 5, 1–15. doi:10.1080/19361521.2012.646413 Bronfenbrenner, U. (1979). The ecology of human development: Experiments by nature and design. Cambridge, MA: Harvard University Press. Brown, E., Cohen, J. A., & Mannarino, A. P. (2020). Trauma-Focused Cognitive Behavioral Therapy: The Role of Caregivers. Journal of Affective Disorders. Brown, V.A., & Lambert, J. A. (2013). Collective learning for transformational change: A guide to collaborative action. New York: Routledge. Bryan, R. L., Kreuter, M. W., & Brownson, R. C. (2009). Integrating adult learning principles into training for public health practice. Health promotion practice, 10(4), 557-563. Burnard, P. (1991). A method of analysing interview transcripts in qualitative research. Nurse Education Today 11(6), 461-466. doi:10.1016/0260-6917(91)90009-Y California Evidence-Based Clearninghouse for Child Welfare. (2020). Information and resources for child welfare professionals. Retrieved from: https://www.cebc4cw.org. Carr, A. (2019). Family therapy and systemic interventions for child‐focused problems: the current evidence base. Journal of Family Therapy, 41(2), 153-213. doi:10.1111/1467- 6427.12226 Centers for Disease Control. (2018). Youth Risk Behavior Survey/ Retrieved February 24, 2020. from: https://www.cdc.gov/healthyyouth/data/yrbs/pdf/trendsreport.pdf. 167 Chorpita, B. F., Rotheram-Borus, M. J., Daleiden, E. L., Bernstein, A., Cromley, T., Swendeman, D., & Regan, J. (2011). The old solutions are the new problem: How do we better use what we already know about reducing the burden of mental illness? Perspectives on Psychological Science, 6, 493–497. doi:10.1177/1745691911418240 Cohen, J. A., Mannarino, A. P., & Deblinger, E. (2006). Treating trauma and trauma grief in children and adolescents. New York, NY: Guilford. Cohen, J. A., Mannarino, A. P., Deblinger, E., & Berliner, L. (2009). Cognitive-behavioral therapy for children and adolescents. In E. B. Foa, T. M. Keane, M. Friedman, & J. A. Cohen (Eds.), Effective treatments for PTSD: Practice guidelines from the international society for traumatic stress studies. New York, NY: Guilford. Cohen, J. A., & Mannarino, A. P. (2015). Trauma-focused Cognitive Behavior Therapy for traumatized children and families. Child Adolescent Psychiatric Clinics of North America, 24(3), 557-570. doi:10.1016/j.chc.2015.02.005 Coulter, S. (2011). Systemic family therapy for families who have experienced trauma: A randomised controlled trial. British Journal of Social Work, 41(3), 502–519. doi:10.1093/bjsw/bcq132. Courtois, C. A., Gold, S. N. (2009). The need for inclusion of psychological trauma in the professional curriculum: A call to action. Psychological Trauma: Theory, Research, Practice, and Policy, 1, 3-23. doi:10.1037/a0015224 Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five traditions. Thousand Oaks, CA: Sage. Creswell, J. W., Plano Clark, V. L., Gutmann, M., & Hanson, W. (2003). Advanced mixed methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 209–240). Thousand Oaks, CA: Sage. Creswell, J. W., & Plano Clark, V. L. (2011). Choosing a mixed methods design. Designing and conducting mixed methods research, 2, 53-106. Creswell, J.W. and Poth, C.N. (2018). Qualitative inquiry and research design: Choosing among five approaches. 4th Edition, SAGE Publications, Inc., Thousand Oaks. Cunningham, P. B., & Henggeler, S. W. (1999). Engaging multiproblem families in treatment: Lessons learned throughout the development of multisystemic therapy. Family Process, 116 (38), 265–286. doi:10.1111/j.1545-5300.1999,00265.x Curry, A., & Epley, P. (2020). “It makes you a healthier professional”: The impact of reflective practice on emerging clinicians’ self-care. Journal of Social Work Education, 1-17. 168 Dalkey, N. C., & Helmer, O. (1969). An experimental application of the Delphi method to the use of experts. Report No. RM-727-PR (Abridged). Santa Monica, CA: RAND Corporation. Dari, T. H. (2017). Development and Validation of Community-Based Participatory Research (CBPR) Competencies: A Delphi study. University of Toledo, Toledo, Ohio. Dari, T., Laux, J. M., Liu, Y., & Reynolds, J. (2019). Development of Community-Based Participatory Research Competencies: A Delphi Study Identifying Best Practices in the Collaborative Process. Professional Counselor, 9(1), 1-19. Davis, K.E. (2014). Expenditures for treatment of mental health disorders among children, ages 5–17, 2009–2011: Estimates for the US civilian noninstitutionalized population. Statistical brief no. 440; Medical Expenditure Panel Survey. Rockville (MD): Agency for Healthcare Research and Quality; June 2014. Davis, K. L., Kliewer, B. W., & Nicolaides, A. (2017). Power and reciprocity in partnerships: Deliberative civic engagement and transformative learning in community-engaged scholarship. Journal of Higher Education Outreach and Engagement, 21(1), 30-54. Davis, S. D., Lebow, J. L., & Sprenkle, D. H. (2012). Common factors of change in couple therapy. Behavior Therapy, 43(1), 36-48. doi:10.1016/j.beth.2011.01.009 Doberneck, D. M., Bargerstock, B. A., McNall, M., Van Egeren, L., & Zientek, R. (2017). Community Engagement Competencies for Graduate and Professional Students: Michigan State University's Approach to Professional Development. Michigan Journal of Community Service Learning, 24(1), 122-142. Doberneck, D. M., & Dann, S. L. (2019). The Degree of Collaboration Abacus Tool. Journal of Higher Education Outreach and Engagement, 23(2), 93-107. Domlyn, A. M., & Wandersman, A. (2019). Community coalition readiness for implementing something new: using a Delphi methodology. Journal of community psychology, 47(4), 882-897. Retrieved from https://onlinelibrary.wiley.com/doi/10.1002/jcop.22161 Dorsey, S., Pullmann, M. D., Kerns, S. E., Jungbluth, N., Meza, R., Thompson, K., & Berliner, L. (2017). The juggling act of supervision in community mental health: Implications for supporting evidence-based treatment. Administration and Policy in Mental Health and Mental Health Services Research, 44(6), 838-852. Dowling, E., Cade, B., Breunlin, D. C., Frude, N., & Seligman, P. (1982). A retrospective survey of students’ views on a family therapy training programme. Journal of Family Therapy, 4(1), 61–72. doi:10.1046/j..1982.00577.x. Erolin, K. S., & Wieling, E. (2021). The Experiences of Couple/Marriage and Family Therapists of color: A Survey Analysis. Journal of Marital and Family Therapy, 47(1), 3-20. 169 Etikan, I., Musa, S. A., & Alkassim, R. S. (2016). Comparison of convenience sampling and purposive sampling. American journal of theoretical and applied statistics, 5(1), 1-4. doi:10.11648/j.ajtas.21060501.11 Ewing, E. S. K., Diamond, G., & Levy, S. (2015). Attachment-based family therapy for depressed and suicidal adolescents: theory, clinical model and empirical support. Attachment & human development, 17(2), 136-156. Fernandez, M. A., & Eyberg, S. M. (2009). Predicting treatment and follow-up attrition in parent–child interaction therapy. Journal of Abnormal Child Psychology, 37(3), 431-441. doi:10.1007/s10802-008-9281-1 Finstad, K. (2010). Response interpolation and scale sensitivity: Evidence against 5-point scales. Journal of usability studies, 5(3), 104-110. Fish, L. S., & Busby, D. M. (1996). The Delphi method. Research methods in family therapy, 469-482. Fixsen, D., Blase, K., Naoom, S., & Wallace, F. (2009). Core implementation components. Research on Social Work Practice, 19(5), 531. doi:10.1177%2F1049731509335549 Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute. Forgatch, M. S., & Patterson, G. R. (2010). Parent Management Training—Oregon Model: An intervention for antisocial behavior in children and adolescents. In J. R. Weisz & A. E. Kazdin (Eds.), Evidence-based psychotherapies for children and adolescents (p. 159– 177). The Guilford Press. Frank, H. E., Becker‐Haimes, E. M., & Kendall, P. C. (2020). Therapist training in evidence‐ based interventions for mental health: a systematic review of training approaches and outcomes. Clinical Psychology: Science and Practice, 27(3), e12330. Friedman, R. M., Katz-Leavy, J. W., Manderscheid, R., & Sondheimer, D. (1996). Prevalence of Serious Emotional Disturbance in Children and Adolescents. In R. Manderscheid and M. Sonnenschein (Eds.) Mental Health, United States: 1996 (pp. 71-89). Washington, DC: U.S. Government Printing Office, DHHS Publication Number (SMA)96-3098. Fruggeri, L. (2012). Different levels of psychotherapeutic competence. Journal of Family Therapy, 34(1), 91-105. doi:10.1111/j.1467-6427.2011.00564.x Garson, G. D. (2014). The Delphi method in quantitative research. Asheboro, NC: Statistical Associates Publishers 170 Gellatly, R., Brookman-Frazee, L., Barnett, M., Gonzalez, J. C., Kim, J. J., & Lau, A. S. (2019). Therapist reports of EBP client engagement challenges in sessions with diverse youth and families in community mental health settings. In Child & youth care forum (Vol. 48, No. 1, pp. 55-75). Springer US. Goodcase, E. T., Brewe, A. M., White, S. W., & Jones, S. (2021). Providers as stakeholders in addressing implementation barriers to youth mental healthcare. Community Mental Health Journal, 1-15. Goodyear, M., Hill, T. L., Allchin, B., McCormick, F., Hine, R., Cuff, R., & O'Hanlon, B. (2015). Standards of practice for the adult mental health workforce: Meeting the needs of families where a parent has a mental illness. International Journal of Mental Health Nursing, 24(2), 169-180. doi:10.111.inm.12120 Graaf, G., & Snowden, L. (2021). State strategies for enhancing access and quality in systems of care for youth with complex behavioral health needs. Administration and Policy in Mental Health and Mental Health Services Research, 48(2), 185-200. Green, D., & Kirby-Turner, N. (1990). First steps in family therapy: A personal construct analysis. Journal of Family Therapy, 12(2), 139–154. doi:10.1046/j..1990.00379.x Green, B., Jones, M., Hughes, D., & Williams, A. (1999). Applying the Delphi technique in a study of GP’s information requirements. Health & social care in the community, 7(3), 198-205. Hardy, D.J., O'Brien, A.P., Gaskin, C.J., O'Brien, A.J., Morrison-Ngatal, E., Skews, G., Ryan, T., & McNulty, N. (2004). Practical application of the Delphi technique in a bicultural mental health nursing study in New Zealand. Journal of Advanced Nursing, 46, 95-109. doi:10.1111/j.1365-2648.2003.02969.x. Hill, K. (2017). Prevalence, experiences, and characteristics of children and youth who enter foster care through voluntary placement agreements. Children and Youth Services Review, 74, 62–70. https: doi.org/10.1016/j.child youth .2017.01.025. Hodges, K., Xue, Y., & Wotring, J. (2004). Use of the CAFAS to evaluate outcome for youths with serious emotional disturbance served by public mental health. Journal of Child and Family Studies,13(3), 325-339. doi:10.1023/B:JCFS.0000022038.62940.a3 Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried, A., & Liddle, H. A. (2008). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76(4), 544. doi:10.1037/0022-006X.76.4.544 Holey, E. A., Feeley, J. L., Dixon, J., & Whittaker, V. J. (2007). An exploration of the use of simple statistics to measure consensus and stability in Delphi studies. BMC Medical Research Methodology, 7(1), 52. doi:10.1186/1471-2288-7-52 171 Homrich, A. M., & Henderson, K. L. (2018). Gatekeeping in the mental health professions. American Counseling Association. Hordijk, R., Hendrickx, K., Lanting, K., MacFarlane, A., Muntinga, M., & Suurmond, J. (2019). Defining a framework for medical teachers’ competencies to teach ethnic and cultural diversity: Results of a European Delphi study. Medical Teacher, 41(1), 68-74. doi:10.1080/0142159X.2018.1439160 Horigian, V. E., Anderson, A. R., & Szapocznik, J. (2016). Taking Brief Strategic Family Therapy from Bench to Trench: Evidence Generation Across Translational Phases. Family Process, 55(3), 529-542. doi:10.1111/famp.12233 Hsu, C.-C., & Sandford, B. A. (2007). Minimizing non-response in the Delphi process: How to respond to non-response. Practical Assessment, Research, and Evaluation, 12(1), 17. Hubble, M. A., Duncan, B. L., & Miller, S. D. (1999). The heart and soul of change: What works in therapy (pp. xxiv-462). American Psychological Association. Hung, H. L., Altschuld, J. W., & Lee, Y. F. (2008). Methodological and conceptual issues confronting a cross-country Delphi study of educational program evaluation. Evaluation and program planning, 31(2), 191-198. Iadarola, S., Levato, L., Harrison, B., Smith, T., Lecavalier, L., Johnson, C., Swiezy, N., Bearss, K., & Scahill, L. (2018). Teaching parents behavioral strategies for autism spectrum disorder (ASD): Effects on stress, strain, and competence. Journal of autism and developmental disorders, 48(4), 1031-1040. Retrieved from https://link.springer.com/content/pdf/10.1007/s10803-017-3339-2.pdf Ibebunjo, Elisha (2021). Examining therapist experience with resistant clients. Dissertations. 580. https://digitalcommons.nl.edu/diss/580 Institute of Medicine Committee on the Quality of Health Care in America. (2001). Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press. Institute of Medicine. (2007). The state of quality improvement and implementation research: Expert views. Workshop summary. Washington, DC: The National Academies Press. Jacoby, B. (2014). Service-learning essentials: Questions, answers, and lessons learned. San Francisco, CA: Jossey-Bass. Jorm, A. F. (2015). Using the Delphi expert consensus method in mental health research. Australian & New Zealand Journal of Psychiatry, 49(10), 887– 897. doi:10.1177/0004867415600891 172 Karekla, M., Lundgren, J., & Forsyth, J. (2004). A survey of graduate training in empirically supported and manualized treatments: A preliminary report. Cognitive and Behavioral Practice, 11, 230–242. doi:10.1016/S1077-7229(04)80034-8 Karlin, B. E., & Cross, G. (2014). From the laboratory to the therapy room: national dissemination and implementation of evidence-based psychotherapies in the US Department of Veterans Affairs Health Care System. American Psychologist, 69(1). doi:10.1037/a0033888 Kazak, A. E., Hoagwood, K., Weisz, J. R., Hood, K., Kratochwill, T. R., Vargas, L. A., & Banez, G. A. (2010). A meta-systems approach to evidence-based practice for children and adolescents. American Psychologist, 65(2), 85. doi:10.1037/a0017784 Kazdin, A. E. (2008). Evidence-based treatment and practice: New opportunities to bridge clinical research and practice, enhance the knowledge base, and improve patient care. American Psychologist, 63, 146– 159. doi:10.1037/0003-066X.63.3.146 Keeney, S., Hasson, F., & McKenna H. (2011). The Delphi technique in nursing and health research. Chichester, West Sussex: Wiley-Blackwell. Kennedy, H. P. (2004). Enhancing Delphi research: methods and results. Journal of advanced nursing, 45(5), 504-511. doi:10.1046/j.1365-2648.2003.02933.x Kenner, C., & Weinerman, J. (2011). Adult learning theory: Applications to non-traditional college students. Journal of College Reading and Learning, 41(2), 87-96. doi:10.1080/10790195.2011.10850344 Kittell-Limerick, P. (2005). Perceived barriers to completion of the academic doctorate: A Delphi study. Texas: Texas A&M University-Commerce. Kliethermes, M. D., Drewry, K., & Wamser-Nanney, R. (2017). Trauma-focused cognitive behavioral therapy. In Evidence-based treatments for trauma related disorders in children and adolescents (pp. 167-186). Springer, Cham. Knutson, N. M., Forgatch, M. S., Rains, L. A., & Sigmarsdóttir, M. (2009). Fidelity of implementation rating system (FIMP): The manual for PMTO™. (Rev. ed.). Eugene, OR: Implementation Sciences International. Koury, S. P., & Green, S. A. (2017). Developing trauma-informed care champions: A six-month learning collaborative training model. Advances in Social Work, 18(1), 145-166. Lambert, M.J. (1992). Psychotherapy outcome research: Implications for integrative and eclectic therapists. In J.C. Norcross & M.R. Goldfried (Eds.). Handbook of psychotherapy integration. (pp.94-129). New York: Wiley. 173 Lapsley, S., Suto, M. J., Barnes, S. J., Scott, M., Hou, S., & Michalak, E. E. (2017). The Delphi consensus consultation method: Using community engagement to identify effective self-management strategies for bipolar disorder. SAGE Publications Ltd. Lauriks, S., de Wit, M. A., Buster, M. C., Arah, O. A., & Klazinga, N. S. (2014). Composing a core set of performance indicators for public mental health care: A modified Delphi procedure. Administration and Policy in Mental Health and Mental Health Services Research, 41(5), 625-635. doi:10.1007/s10488-013-0506-4 Layne, C. M., Steinberg, J. R., & Steinberg, A. M. (2014). Causal reasoning skills training for mental health practitioners: Promoting sound clinical judgment in evidence-based practice. Training and Education in Professional Psychology, 8(4), 292. doi:10.1037/tep0000037 Layne, C. M., Strand, V., Popescu, M., Kaplow, J. B., Abramovitz, R., Stuber, M., Amaya- Jackson, L., Ross, L. & Pynoos, R. S. (2014). Using the core curriculum on childhood trauma to strengthen clinical knowledge in evidence-based practitioners. Journal of Clinical Child & Adolescent Psychology, 43(2), 286-300. doi:10.1080/15374416.2013.865192 10.1080/15374416.2013.865192 Leavy, P. (2017). Research design: Quantitative, qualitative, mixed methods, arts-based, and community-based participatory research approaches. Guilford Publications. Lechowicz, J. S., & Gazda, G. M. (1975). Group counseling instruction: Objectives established by experts. Counselor Education and Supervision, 15(1), 21-27. doi:10.1002/j.1556-6978.1975.tb00980.x Leedy, P. D., & Ormrod, J. E. (2010). Practical research: Planning and design (9th ed.). Upper Saddle River, NJ: Pearson. Lincoln, Y. S., & Guba, E. G. (1985). Establishing trustworthiness. Naturalistic inquiry, 289(331), 289-327. Linstone, H. A., & Turoff, M. (Eds.). (1975). The delphi method. Reading, MA: Addison-Wesley. Lorås, L. (2018). Systemic Family Therapy Competences in Child and Adolescent Mental Health Care. Contemporary Family Therapy, 40(1), 1-9. doi:10.1007/s10591-017-9440-z Lund, B. D. (2020). Review of the Delphi method in library and information science research. Journal of Documentation. Lupien, S. J., McEwen, B. S., Gunnar, M. R., Heim, C. (2009). Effects of stress throughout the lifespan on the brain, behavior and cognition. Nature Reviews Neuroscience, 10, 434– 445. doi:10.1038/nrn2639 174 Manuel, J. I., Mullen, E. J., Fang, L., Bellamy, J. L., & Bledsoe, S. E. (2009). Preparing social work practitioners to use evidence-based practice: A comparison of experiences from an implementation project. Research on Social Work Practice, 19(5), 613-627. doi:10.1177%2F1049731509335547 Marty, D., Rapp, C., McHugo, G., & Whitley, R. (2008). Factors influencing consumer outcome monitoring in implementation of evidence-based practices: Results from the National EBP Implementation Project. Administration and Policy in Mental Health and Mental Health Services Research, 35(3), 204-211. doi: 10.1007/s10488-007-0157-4 McHugo, G. J., Drake, R. E., Whitley, R., Bond, G. R., Campbell, K., Rapp, C. A., Goldman, H.H., Lutz, W.J., & Finnerty, M. T. (2007). Fidelity outcomes in the national implementing evidence-based practices project. Psychiatric Services, 58(10), 1279-1284. doi:10.1176/ps.2007.58.10.1279 McKenna, H. P. (1994). The essential elements of a practitioners' nursing model: a survey of psychiatric nurse managers. Journal of Advanced Nursing, 19(5), 870-877. McPherson, S., Reese, C., & Wendler, M. C. (2018). Methodology update: Delphi studies. Nursing research, 67(5), 404-410. Mendel, P., Meredith, L., Schoenbaum, M., Sherbourne, C., & Wells, K. (2008). Interventions in organizational and community context: A framework for building evidence on dissemination and implementation in health services research. Administration and Policy in Mental Health and Mental Health Services Research, 35(1), 21–37. doi:10.1007/s10488-007-0144-9 Merikangas, K. R., He, J. P., Brody, D., Fisher, P. W., Bourdon, K., & Koretz, D. S. (2010). Prevalence and treatment of mental disorders among US children in the 2001–2004 NHANES. Pediatrics, 125(1), 75-81. Merlo, L. J., Storch, E. A., Lehmkuhl, H. D., Jacob, M. L., Murphy, T. K., Goodman, W. K., & Geffken, G. R. (2010). Cognitive-behavioral therapy plus motivational interviewing improves outcome for pediatric obsessive-compulsive disorder: A preliminary study. Cognitive Behaviour Therapy, 39, 24–27. doi:10.1080/16506070902831773 Metz, A. (2016). Practice Profiles: A Process for Capturing Evidence and Operationalizing Innovations. White Paper. National Implementation Research Network. Miao, T.-A., Umemoto, K., Gonda, D., & Hishinuma, E. S. (2011). Essential elements for community engagement in evidence-based youth violence prevention. American journal of community psychology, 48(1-2), 120-132. doi:10.1007/s10464-010-9418-6 Miller, W. R., & Rollnick, S. (2012). Motivational interviewing: Helping people change. Guilford press. 175 Mitchell, B. S., Kern, L., & Conroy, M. A. (2019). Supporting students with emotional or behavioral disorders: State of the field. Behavioral Disorders, 44(2), 70-84. Mock, M. R. (2008). Visioning social justice: Narrative of diversity, social location, and personal compassion. In M. McGoldrick & K. V. Hardy (Eds.), Re-visioning family therapy (pp. 425–441). New York: Guilford. Nachmias, D., & Nachmias, C. (1981). Research methods in the social sciences (2nd ed.). New York: St. Martin’s Press. National Childhood Traumatic Stress Network Core Curriculum on Childhood Trauma Task Force (2012). The 12 core concepts: Concepts for understanding traumatic stress responses in children and families. Core Curriculum on Childhood Trauma. Los Angeles, CA, and Durham, NC: UCLA-Duke University National Center for Child Traumatic Stress. National Institute of Drug Abuse. (2020). Family-Based Approaches. Retrieved from https://www.drugabuse.gov/publications/principles-adolescent-substance-use-disorder- treatment-research-based-guide/evidence-based-approaches-to-treating-adolescent- substance-use-disorders/family-based-approaches. National Institutes of Health. (2018). Dissemination and implementation research in health (R01) Web site. Retrieved February 28, 2020 from: https://grants.nih.gov/grants/guide/pa- files/PAR-16-238.html National Institute of Mental Health. (2018). Mental health information surveillance. Retrieved February 28, 2020 from: https://www.nimh.nih.gov/health/topics/substance-use-and- mental-health/index.shtml. Nel, P. W. (2006). Trainee perspectives on their family therapy training. Journal of Family Therapy, 28(3), 307–328. doi:10.1111/j.1467-6427.2006.00354.x. Nelson, T. D., Steele, R. G., & Mize, J. A. (2006). Practitioner attitudes toward evidence-based practice: Themes and challenges. Administration and Policy in Mental Health and Mental Health Services Research, 33, 398–409. doi:10.1007/s10488-006-0044-4 Nelson, M. M., Stanley, J. R., Funderburk, B. W., & Bard, E. (2012). Therapists’ attitudes toward evidence-based practices and implementation of parent–child interaction therapy. Child Maltreatment, 17(1), 47–55. doi:10.1177/1077559512436674. Norcross, J. C., & Wampold, B. E. (2011). Evidence-based therapy relationships: Research conclusions and clinical practices. Psychotherapy, 48(1), 98. doi:10.1037/a0022161 Oed, M. M., & Gonyea, J. (2019). Examining the Beliefs and Attitudes of Current Students and Recent Graduates on the Inclusion of Children in Family Therapy. Contemporary Family Therapy, 41(2), 135-146. 176 Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: an example, design considerations and applications. Information & management, 42(1), 15-29. doi:10.1016/jm.2003.11.002 Onken, L. S., Carroll, K. M., Shoham, V., Cuthbert, B. N., & Riddle, M. (2014). Reenvisioning clinical science: Unifying the discipline to improve the public health. Clinical Psychological Science, 2, 22–34. doi: 10.1177/2167702613497932 Orlinsky, D. E., & Ronnestad, M. H. (2005). How psychotherapists develop: A study of therapeutic work and professional growth. Washington, DC: American Psychological Association. doi:10.1037/11157-000. Ovenstad, K. S., Ormhaug, S. M., Shirk, S. R., & Jensen, T. K. (2020). Therapists’ behaviors and youths’ therapeutic alliance during trauma-focused cognitive behavioral therapy. Journal of Consulting and Clinical Psychology, 88(4), 350-361. doi:10.1037/ccp0000465 Owen, J. E. (2016). Fostering critical reflection: Moving from a service to a social justice paradigm. New directions for student leadership, 2016 (150), 37-48. Owen, J., Duncan, B. L., Reese, J., Anker, M. G., & Sparks, J. A. (2014). Accounting for therapist variability in couple therapy: What really matters? Journal of Sex and Marital Therapy, 40(6), 488–502. doi:10.1080/0092623X.2013.772552. Palinkas, L. A., Horwitz, S. M., Green, C. A., Wisdom, J. P., Duan, N., & Hoagwood, K. (2015). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and policy in mental health and mental health services research, 42(5), 533-544. doi:10.1007/s10488-013-0528-y Pallant, J. (2011). Survival manual. A step by step guide to data analysis using SPSS. New York, NY: Open University Press. Parra Cardona, J. R., Domenech‐Rodriguez, M., Forgatch, M., Sullivan, C., Bybee, D., Holtrop, K., Escobar-Chew, A.R., Tams, L., Dates, B., & Bernal, G. (2012). Culturally Adapting an Evidence‐Based Parenting Intervention for Latino Immigrants: The Need to Integrate Fidelity and Cultural Relevance. Family process, 51(1), 56-72. Patton, M.Q. (2015). Qualitative research and evaluation methods. (4th ed.). Thousand Oaks, CA: Sage. Pemberton, J. R., Conners-Burrow, N. A., Sigel, B. A., Sievers, C. M., Stokes, L. D., & Kramer, T. L. (2017). Factors Associated with Clinician Participation in TF-CBT Post-workshop Training Components. Administration and Policy in Mental Health and Mental Health Services Research, 44(4), 524-533. doi:10.1007/s10488-015-0677-2 177 Perou, R., Bitsko, R. H., Blumberg, S. J., Pastor, P., Ghandour, R. M., Gfroerer, J. C., Hedden, S. L., Crosby, A. E., Visser, S. N., Schieve, L. A., Parks, S. E., Hall, J. E., Brody, D., Simile, C. M., Thompson, W. W., Baio, J., Avenevoli, S., Kogan, M. D., Huang, L. N., & Centers for Disease Control and Prevention (CDC) (2013). Mental health surveillance among children—United States, 2005-2011. MMWR supplements, 62(2), 1–35. Perepletchikova F, Treat TA, Kazdin AE. (2007). Treatment integrity in psychotherapy research: Analysis of the studies and examination of the associated factors. Journal of Consulting and Clinical Psychology, 75(6):829–841. [PubMed: 18085901] doi:10.1037/0022-006X.75.6.829 PettyJohn, M.E., Tseng, C.-F. and Blow, A.J. (2020), Therapeutic utility of discussing therapist/client intersectionality in treatment: When and how? Family Process, 59, 313- 327. https://doi.org/10.1111/famp.12471 President’s New Freedom Commission on Mental Health. (2003). Achieving the promise: Transforming mental health care in America. DHHS Publication No. SMA-03-3832, Rockville, MD. Prinz, R. J., & Miller, G. E. (1994). Family-based treatment for childhood antisocial behavior: Experimental influences on dropout and engagement. Journal of Consulting and Clinical Psychology, 62, 645–650. doi:10.1037/0022-006X.62.3.645 Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009). Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36(1), 24-34. doi:10.1007/s10488-008-0197-4 Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey R., & Hensley, M. (2011). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and policy in mental health and mental health services research, 38(2), 65-76. Qualtrics. (2005). Retrieved from https://www.qualtrics.com. Provo, UT. Rapp, C. A., Etzel-Wise, D., Marty, D., Coffman, M., Carlson, L., Asher, D., Callaghan, J., Holter, M. (2010). Barriers to evidence-based practice implementation: Results of a qualitative study. Community Mental Health Journal, 46(2), 112-118. doi:10.1007/s10597-009-9238-z Robbins, M. S., Alexander, J. F., & Turner, C. W. (2000). Disrupting defensive family interactions in family therapy with delinquent adolescents. Journal of Family Psychology, 14(4), 688. doi:10.1037/0893-3200.14.4.688 Robbins, M. S., Alexander, J. F., Turner, C. W., & Hollimon, A. (2016). Evolution of functional family therapy as an evidence‐based practice for adolescents with disruptive behavior problems. Family Process, 55(3), 543-557. 178 Rodriguez, A., Kim, J. J., Zhan, C., Lau, A. S., Hamilton, A. B., Palinkas, L. A., Gellatly, R., & Brookman-Frazee, L. (2021). A mixed-method analysis on the impacts of a system- driven implementation of multiple child evidence-based practices on community mental health providers. Professional Psychology: Research and Practice, 52(1), 67. Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York, NY: Free Press. Rohrbaugh, M. J. (2014). Old wine in new bottles: Decanting systemic family process research in the era of evidence-based practice. Family Process, 53(3), 434–444. doi:10.1111/famp.12079. Ross, A. M., Kelly, C. M., & Jorm, A. F. (2014). Re-development of mental health first aid guidelines for suicidal ideation and behaviour: A Delphi study. BMC Psychiatry, 14, doi:10.1186/s12888-014-0241-8 Roundfield, K. D., & Lang, J. M. (2017). Costs to Community Mental Health Agencies to Sustain an Evidence-Based Practice. Psychiatric Services, 68(9), 876-882. doi:10.1176/appi.ps.201600193 Rousmaniere, T. (2016). Deliberate practice for psychotherapists: A guide to improving clinical effectiveness. Taylor & Francis. Rowe, G., & Wright, G. (1999). The Delphi technique as a forecasting tool: issues and analysis. International journal of forecasting, 15(4), 353-375. doi:10.1016/S0169-2070(99)00018-7 Saldana, L., & Chamberlain, P. (2012). Supporting implementation: the role of community development teams to build infrastructure. American journal of community psychology, 50(3-4), 334-346. doi:10.1007/s10464-012-9503-0 Saldana, L. (2014). The stages of implementation completion for evidence-based practice: protocol for a mixed methods study. Implementation Science, 9(1), 43-43. doi:10.1186/1748-5908-9-43 Sandelowski, M. (2000). Combining qualitative and quantitative sampling, data collection, and analysis techniques in mixed‐method studies. Research in nursing & health, 23(3), 246- 255. Sanders M. & Turner K. (2005). Reflections on the challenges of effective dissemination of behavioural family intervention: Our experience with the Triple P—Positive parenting program. Child and Adolescent Mental Health, 10,158–169. doi:10.1111/j.1475-3588.2005.00367.x Scott, J., Yap, K., Bunch, K., Haarhoff, B., Perry, H., & Bennett‐Levy, J. (2021). Should personal practice be part of cognitive behaviour therapy training? Results from two self‐ practice/self‐reflection cohort control pilot studies. Clinical Psychology & Psychotherapy, 28(1), 150-158. 179 Sekayi, D., & Kennedy, A. (2017). Qualitative Delphi method: A four round process with a worked example. The Qualitative Report, 22(10), 2755-2763. Sexton, T., Gordon, K. C., Gurman, A., Lebow, J., Holtzworth-Munroe, A., & Johnson, S. (2011). Guidelines for classifying evidence-based treatments in couple and family therapy. Family Process, 50(3), 377–392. doi:10.1111/j.1545-5300.2011.01363.x. Shapiro, C. J., Prinz, R. J., & Sanders, M. R. (2012). Facilitators and barriers to implementation of an evidence-based parenting intervention to prevent child maltreatment: The Triple-P Positive Parenting Program. Child Maltreatment, 17, 86–95. doi:10.1177/ 1077559511424774. Sholomskas, D.E., Syracuse-Siewert, G., Rounsaville, B.J., Ball, S.A, & Nuro, K.F. (2005). We don’t train in vain: A dissemination trial of three strategies of training clinicians in cognitive behavioral therapy. Journal of Consulting and Clinical Psychology,73:106– 115. doi:10.1037/0022-006X.73.1.106 Skulmoski, G. J., Hartman, F. T., & Krahn, J. (2007). The Delphi method for graduate research. Journal of Information Technology Education: Research, 6(1), 1-21. Sori, C. F., & Sprenkle, D. H. (2004). Training family therapists to work with children and families: A modified Delphi study. Journal of Marital and Family Therapy, 30(4), 479– 495. doi:10.1111/j.1752-0606.2004.tb01257.x. Sperry, L. (2005). Case conceptualization: A strategy for incorporating individual, couple and family dynamics in the treatment process. The American Journal of Family Therapy, 33(5), 353-364. Sprenkle, D. H., & Blow, A. J. (2004). Common factors and our sacred models. Journal of Marital and Family Therapy, 30(2), 113-129. doi:10.1111/j.1752-0606.2004.tb01228.x Sprenkle, D. H., Blow, A. J., & Dickey, M. (1999). Common factors and other nontechnique variables in marriage and family therapy. In M. A. Hubble, B. L. Duncan & S. Miller (Eds.), The heart and soul of change: What works in therapy (pp. 329-360). Washington, DC: The American Psychological Association. Sprenkle, D. H., Davis, S. D., & Lebow, J. (2009). Common factors in couple and family therapy: The overlooked foundation for effective practice. New York: Guilford Press. Staller, K. M. (2006). Railroads, runaways, & researchers: Returning evidence rhetoric to its practice base. Qualitative Inquiry, 12(3), 503–522. doi:10.1177/1077800406286524. State of Michigan. Department of Health and Human Services. (2020). Services for Severe Emotional Disturbances. Retrieved from: https://www.michigan.gov/mdhhs/ 0,5885,7- 339-71550_2941_4868_80988-427532--,00.html 180 Stirman, S. W., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implementation Science, 7(1), 17. doi:10.1186/1748-5908-7-17 Stratton, P., Silver, E., Nascimento, N., McDonnell, L., Powell, G., & Nowotny, E. (2015). Couple and family therapy outcome research in the previous decade: What does the evidence tell us? Contemporary Family Therapy, 37(1), 1-12. doi: org/10.1007/s10591-014-9314-6 Substance Abuse and Mental Health Services Administration, Center for Mental Health Services. (1993). Definition of children with a serious emotional disturbance. Federal Register, 58(96), 29425. Substance Abuse and Mental Health Services Administration. (2020). Strategic plan FY 2019 – FY 2023. Rockville, MD: United States Department of Health and Human Services. Sue, D. W., & Sue, D. (2015). Counseling the culturally diverse: Theory and practice (7th ed.). Hoboken, NJ: Wiley and Sons. Szapocznik, J., & Hervis, O. E. (2020). Brief strategic family therapy. American Psychological Association. Washington, DC. doi:10.1037/0000169-000 Szapocznik, J., Muir, J. A., Duff, J. H., Schwartz, S. J., & Brown, C. H. (2015). Brief strategic family therapy: Implementing evidence-based models in community settings. Psychotherapy research, 25(1), 121-133. Trevelyan, E. G., & Robinson, N. (2015). Delphi methodology in health research: how to do it? European Journal of Integrative Medicine, 7(4), 423-428. doi:10.1016/j.eujim.2015.07.002 Van der Oord, S., & Tripp, G. (2020). How to improve behavioral parent and teacher training for children with ADHD: Integrating empirical research on learning and motivation into treatment. Clinical child and family psychology review, 1-28. VERBI Software. (2019). MAXQDA 2020 [computer software]. Berlin, Germany: VERBI Software. Available from maxqda.com. Von Bertalanffy, L. (1968). General system theory: foundations, development, applications. George Braziller Inc., New York. Wampold, B.E. & Brown, G.S. (2005). Estimating therapist variability: A naturalistic study of outcomes in managed care. Journal of Consulting and Clinical Psychology, 73, 914-923. doi:10.1037/0022-006X.73.5.914 181 Wampold, B.E. (2006). Not a scintilla of evidence to support empirically supported treatments as more effective than other treatments. In J.C. Norcross, L.E. Beutler, & R.F. Levant (Eds.), Evidence-based practices in mental health (pp. 194–197). Washington DC: American Psychological Association. Watkins Jr, C. E., Hook, J. N., Owen, J., DeBlaere, C., Davis, D. E., & Van Tongeren, D. R. (2019). Multicultural orientation in psychotherapy supervision: Cultural humility, cultural comfort, and cultural opportunities. American journal of psychotherapy, 72(2), 38-46. Watson, M. (1993). Supervising the person of the therapist: Issues, challenges, and dilemmas. Contemporary Family Therapy, 15, 21–31. doi:10.1007/BF00903485 Weeks, G. R., & D’Aniello, C. (2017). Teaching reframes in a master’s level marriage and family therapist program. Journal of Family Psychotherapy, 28(4), 303-316. Weisz, J. R., & Kazdin, A. E. (2010). Evidence-based psychotherapies for children and adolescents: Guilford Press. Wittenborn, A. K., Blow, A. J., Holtrop, K., & Parra-Cardona, J. R. (2019). Strengthening research in marriage and family therapy: Challenges and multilevel solutions. Journal of Marital and Family Therapy, 45(1), 20-32. doi:10.1111/jmft.12338 Yeager, K. A., & Bauer-Wu, S. (2013). Cultural humility: Essential foundation for clinical researchers. Applied Nursing Research, 26(4), 251-256. 182