MAPPING THE INTERVENTION PROCESS OF A PARENT-MEDIATED INTERVENTION FOR AUTISM SPECTRUM DISORDER USING MIXED METHODS. By Kyle Michele Frost A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of Psychology – Doctor of Philosophy 2023 ABSTRACT Empirical support for naturalistic developmental behavioral interventions (NDBIs) is growing, suggesting that this class of early interventions are effective for supporting social communication development in young autistic children. NDBIs are frequently implemented as parent mediated interventions, in which caregivers are taught specific treatment techniques to implement with their child. Parent mediated NDBIs are complex, involving two levels of service delivery (coach to parent, and parent to child) each comprised of multiple intervention elements. Despite their promise, little research has examined the active ingredients and mechanisms of change underlying NDBIs, and studies have yet to consider unique features of parent-mediated approaches that may affect how they work. The present study used an exploratory sequential mixed methods design grounded in the theory of change framework to explore the active ingredients and mechanisms of change of Project ImPACT, an empirically supported NDBI for young children with autism or social communication delays. The first study aim was to develop a comprehensive Theory of Change by obtaining stakeholder perspectives on how Project ImPACT works. Ten intervention experts, 22 community providers, and 12 caregivers participated in semi structured interviews which were subsequently coded using qualitative methods. We used joint displays to develop and visualize causal models for each intervention element, which we then consolidated in a broader visual model of intervention process at the parent and child level. The second aim was to provide proof- of-concept of our model using quantitative analyses conducted with archival data from treatment trials of Project ImPACT. Through completion of aim 1, we developed two integrated models of the change process for Project ImPACT, with the first detailing the caregiver learning process as the caregiver works with a coach, and the second detailing the child learning process as the caregiver implements the intervention techniques. The caregiver learning model emphasized the importance of both learning and motivational process in supporting fidelity and sustainment. The child learning model described how developmental techniques aimed at supporting children’s attention and engagement lay the foundation for more adult-directed learning opportunities and subsequent child skill growth. However, aim 2 hypotheses were not supported by path analyses conducted using archival data. Our qualitative data also described key themes relating to long-term outcomes (e.g., sustainment and quality of life for children and families), need for a service like Project ImPACT (e.g., structured and systematic, child-centered, parent-driven service), and contextual factors that affect fit (e.g., child- and family-level characteristics, cultural and linguistic factors). Developing, refining, and testing program theory is a core element of research in complex interventions, with several benefits for research and implementation in practice. Although parent mediated NDBIs are increasingly being studied in research contexts, their active ingredients and mechanisms of change are understudied, and their complexity presents a barrier to widespread implementation in the community. In the future, this research can be used to develop and prioritize nuanced research questions related to the timing, optimization, and mechanistic process underlying NDBIs. This dissertation is dedicated to my family – especially Benji, Chelsea, and Dax; Mom and Dad; Eric, Christie, and Huxley; who have been sources of unwavering love and support throughout my graduate training. iv ACKNOWLEDGEMENTS First, I would like to thank my mentor, Dr. Brooke Ingersoll, for her enthusiastic support and encouragement throughout my graduate training; I have grown so much as a scientist and scholar thanks to you. I would also like to thank Drs. Aubyn Stahmer and Paul Yoder; your mentorship and guidance in completing my dissertation was invaluable. Thanks also to my lab mates, particularly Katherine Pickard, Karis Casagrande, and Diondra Straiton; it has been a joy to learn from and alongside you as friends and professional colleagues. Thank you to Danielle Duggan and Delaney Weedmark; your dedication to coding and insightful engagement with our qualitative data made this work possible. I am also grateful for the support of a National Research Service Award from the Eunice Kennedy Shriver National Institute of Child Health and Human Development (F31HD103209), which provided funding for this work. Finally, I would like to thank the clinicians and families whom I’ve had the opportunity to train and coach in Project ImPACT; my training and research have benefited so much from your insights and expertise. v TABLE OF CONTENTS CHAPTER 1: Introduction ............................................................................................................. 1 A mixed methods approach to studying complex interventions ................................................. 1 Naturalistic developmental behavioral interventions for autism................................................. 3 Complexity in parent-mediated interventions ............................................................................. 5 Study Aims: Developing a Theory of Change of a Parent-Mediated NDBI............................... 8 CHAPTER 2: Method ................................................................................................................... 11 Overview ................................................................................................................................... 11 Conceptual framework: Theory of Change ............................................................................... 11 Intervention of interest: Project ImPACT ................................................................................. 12 Phase 1: Qualitative analysis ..................................................................................................... 13 Phase 2: Using connected integration to develop a theory of change ....................................... 26 Phase 3: Quantitative analysis ................................................................................................... 27 CHAPTER 3: Results; Service need and context ......................................................................... 30 Service need .............................................................................................................................. 30 Context: Fit for family............................................................................................................... 34 CHAPTER 4: Results; Coach-parent level intervention processes .............................................. 43 Phases 1 and 2 ........................................................................................................................... 43 Phase 3....................................................................................................................................... 57 Merged Integration .................................................................................................................... 61 CHAPTER 5: Results; Parent-child level intervention processes ................................................ 64 Phases 1 and 2 ........................................................................................................................... 64 Phase 3....................................................................................................................................... 82 Merged Integration .................................................................................................................... 86 CHAPTER 6: Discussion and conclusions ................................................................................... 89 Strengths .................................................................................................................................... 89 Limitations ................................................................................................................................ 91 Directions for future research .................................................................................................... 94 Conclusion................................................................................................................................. 97 REFERENCES ............................................................................................................................. 98 APPENDIX A: PROJECT IMPACT INTERVENTION ELEMENTS ..................................... 109 APPENDIX B: QUANTITATIVE VARIABLE CORRELATIONS......................................... 110 vi CHAPTER 1: Introduction A mixed methods approach to studying complex interventions Developing, refining, and testing program theory is a core element of research in complex interventions (Patient-Centered Outcomes Research Institute, 2019; Skivington et al., 2021) yet these theories are often underdeveloped (Sermeus, 2015). Randomized controlled trials, widely considered the gold standard in efficacy research, employ a “black box” evaluation approach that prioritizes study of outcomes while largely neglecting the complexity of multi- component interventions (Scriven, 1994). Studying multi-component interventions as a package obscures key active ingredients as well as potential redundant, ineffective, or harmful intervention elements which should be trimmed from an intervention to optimize its effectiveness (Collins et al., 2005; Dimidjian & Hollon, 2010). Mixed methods have numerous strengths for advancing complex interventions research, including their ability to examine multifaceted and complex phenomena, and the ability to integrate stakeholder perspectives which support practical, real-world benefits (Borglin, 2015; Craig et al., 2008; Curry & Nunez-Smith, 2015). As such, mixed methods are increasingly being used in clinical, health services, and implementation research (Curry & Nunez-Smith, 2015). However, to our knowledge, researchers have yet to capitalize on mixed methods integration to develop and test causal explanatory models of the component parts of complex interventions. Here, we illustrate how mixed methods can be used to take a theory-based perspective on evaluation, focusing on exploring intervention processes: how and why complex intervention components bring about change in outcomes. Developing models of how interventions work is an ongoing and iterative process with several benefits for research and implementation in practice (Sermeus, 2015). Well-developed causal explanatory theories support the design of innovative empirical research agendas that 1 advance beyond questions of intervention efficacy. For example, such theories point to key constructs regarding how and for whom treatments work, enabling the development of nuanced research questions related to the timing, optimization, and mechanistic process underlying efficacious interventions (Leviton & Trujillo, 2017). Furthermore, while many intervention programs are packaged or manualized for the purpose of conducting research, there are shared components or commonalities among similar classes of interventions (e.g. cognitive-behavioral therapy), or programs for specific concerns or populations (e.g. childhood disruptive behavior disorders; Chorpita et al., 2005; Garland et al., 2008; Kaehler et al., 2016). Intervention theory can elucidate meaningful similarities and differences in the functions or mechanisms of these interventions, aiding the development of a cumulative intervention science (Davidoff et al., 2015; Leviton & Trujillo, 2017; Weiss, 1995). Moreover, the underdevelopment of intervention theory is a barrier to understanding the generalizability or external validity of interventions as they move from research to implementation in real-world settings (Leviton, 2017). Intervention theory can inform important contextual factors related to implementation (e.g. necessary preconditions for adoption of a new practice, organization- or provider-level moderators of effectiveness), which reduces uncertainty about whether an intervention is likely to generalize to a new setting (Leviton & Trujillo, 2017) and can support effective replication in new contexts (Davidoff et al., 2015). Causal explanatory intervention theories can also inform implementation constructs at the level of the intervention, such as treatment fidelity and adaptation. Complex interventions developed in research settings are frequently adapted as they move into the community to improve fit to context or individual clients. Adaptation has the potential to reduce efficacy if active ingredients are not implemented (Chambers & Norton, 2016), therefore specifying the 2 core functions of an intervention has been recommended to inform adaptation in practice. Core functions describe the mechanisms or processes through which intervention activities bring about intended outcomes (Kirk et al., 2019), recognizing that it is possible (and perhaps necessary) to make adaptations to specific intervention activities while addressing the same underlying process. Well-developed intervention theory, consistent with identifying these core functions, supports a “planned adaptation” approach in which providers can maintain essential features of the treatment while making adaptations that improve fit to context and to individual clients (Kirk et al., 2019; Lee et al., 2008). Naturalistic developmental behavioral interventions for autism Autism spectrum disorder (ASD) is a relatively prevalent neurodevelopmental disability affecting 1 in 59 children in the United States (Christensen et al., 2018). Policy guidance recommends high-quality early intervention as soon as children are diagnosed in order to improve social, communication, cognitive, and behavioral outcomes (National Research Council, 2001). Numerous efficacious interventions focused on supporting autistic children’s social communication development have been shown to lead to functional improvements in child social communication. These interventions have emerged from both developmental and behavioral theoretical orientations; however current best practices have converged on a combination of strategies from both perspectives (Schreibman et al., 2015). This class of interventions, called naturalistic developmental behavioral interventions (NDBIs), has growing empirical support for improving social communication for young autistic children (Sandbank et al., 2020; Tiede & Walton, 2019). NDBIs are complex interventions which share several common treatment elements, such as following the child’s lead, modeling appropriate language, using communicative temptations, 3 and using prompting techniques to teach new skills (Frost et al., 2020). They emphasize teaching in naturalistic contexts, and focus on developmentally informed treatment targets such as imitation, joint attention, play, joint engagement, and communication skills (Bruinsma et al., 2020; Schreibman et al., 2015). Despite their promise, little research has examined the active ingredients and mechanisms of change underlying NDBIs. Because NDBIs are complex interventions, particularly when implemented by parents, there are a variety of potential mechanistic processes at play. For example, evidence suggests that both adult responsiveness and growth in ‘pivotal skills’ mediate treatment outcomes in early interventions for social communication delays. Increased “mirrored pacing” and “parental synchrony,” both related to adult responsiveness, have been associated with treatment outcomes (Aldred et al., 2012; Gulsrud et al., 2016; Pickles et al., 2015). In addition, growth in child imitation and intentional communication have been shown to indirectly affect later expressive language (Yoder et al., 2021a), providing support for the idea that directly targeting early social communication skills has downstream effects on other developmental outcomes. Indeed, supporting social engagement may enable children to learn optimally from their environment; recent work found that caregiver language modeling was more strongly associated with children’s sentence diversity when caregivers’ NDBI strategy use increased (Clark-Whitney et al., 2022). There are numerous barriers to understanding mechanisms underlying NDBIs. First, an emphasis on studying packaged programs as a whole (a “black box” evaluation approach) largely neglects the complexity of multi-component interventions (Scriven, 1994) and obscures similarities and differences among packaged NDBIs. NDBIs have several components which could have additive effects (i.e. the total treatment effect is equal to the sum of treatment effects 4 for each active ingredient alone) or interactive effects; interactive effects may be synergistic (i.e., the total treatment effect exceeds the sum of treatment effects for each active ingredient alone) or antagonistic (i.e., the total treatment effect is less than the sum of treatment effects for each active ingredient alone) in nature (Upton & Cook, 2014). As such, it is not clear whether some ingredients are more essential than others when it comes to supporting positive outcomes. In addition, NDBIs target a variety of developmental outcomes which are expected to occur on different time scales. Early, context-dependent changes in social communication are thought to have downstream effects on more distal, generalized social communication outcomes. Typical RCT designs which include pre-intervention, post-intervention, and (potentially) follow-up assessments may not be optimally timed to capture these developmental processes as they unfold. In sum, the complexity of NDBIs, which includes many intervention elements and outcomes, makes it difficult to understand the intervention process. Complexity in parent-mediated interventions Involving caregivers in the implementation of therapeutic interventions is a common approach to therapy for a variety of early childhood psychiatric disorders and developmental disabilities, including disruptive behavior disorders (Michelson et al., 2013), language impairment (Roberts et al., 2019), ADHD (Rimestad et al., 2019), and autism (Nevill et al., 2018). Parent mediated interventions (PMIs) are designed to teach caregivers specific treatment techniques to implement with their child in the home and community, with the child being the intended beneficiary of the intervention (Bearss et al., 2015). PMI is particularly prevalent within NDBIs (Schreibman et al., 2015). Given their emphasis on the child’s natural environment, several NDBIs were developed explicitly as PMIs (e.g. Social ABCs; Brian et al., 2016; Project ImPACT; Ingersoll & Wainer, 2013; World Health 5 Organization Caregiver Skills Training; Salomone et al., 2019) and others have been adapted to parent-mediated formats. From an implementation perspective, PMI are thought to have several unique advantages for increasing service access. Because caregivers can potentially implement techniques throughout their child’s day at home or in the community, PMI may provide children with a higher dose of treatment while requiring fewer hours of direct service provision by trained professionals (Wetherby et al., 2018). As such, PMI are a relatively low-intensity service compared to many direct service models for ASD, which may make them easier to implement in low-resource settings and existing service delivery systems (Reichow et al., 2013; Wetherby et al., 2018). PMI are complex and involve two levels of service delivery for a given family; first, a professional teaches a caregiver to use the intervention, and the caregiver subsequently implements the intervention with their child. As such, professionals must be proficient in multiple skill sets including adult coaching and child-directed intervention to deliver PMIs with high fidelity. Contextual features further contribute to complexity for PMI. Features of the family system, social interactions, cultural contexts, and service delivery systems may all affect whether and how the treatment works (Nilsen & Bernhardsson, 2019). Lastly, sustained implementation of these interventions involve dyadic interactions between caregivers and their children; feedback loops and collateral changes or “side effects” for other behaviors are likely to occur in this dynamic system (Sterman, 2006), which may contribute to intervention use or long- term outcomes. Given their potential benefits, a better understanding of how PMIs work is crucial for several reasons. Though much research in early intervention for autism has shifted in emphasis from therapist- to parent-implemented intervention models, it has been assumed that the 6 treatments work essentially the same way regardless of who delivers them. Indeed, a variety of packaged NDBI programs for this population have been adapted to parent-implemented formats (e.g. Early Start Denver Model; Estes et al., 2014; Enhanced Milieu Teaching; Kaiser et al., 2000; Joint Attention, Symbolic Play, Engagement & Regulation; Kasari et al., 2014), taking interventions developed for therapist use and teaching them to caregivers. Assuming these treatments work the same way when implemented by caregivers neglects broader outcomes of the caregiver learning process, which is an important intervention unto itself with a variety of potential outcomes. Although improved fidelity of implementation and subsequent improvements in child developmental functioning are the primary intended outcomes, PMIs may affect caregiver mental health, parenting stress, and self-efficacy (Estes et al., 2014; Ingersoll et al., 2016; Tonge et al., 2006) or have other intervening effects on the caregiver-child relationship. Often treated as ‘secondary outcomes’ or ‘collateral benefits,’ these factors may be essential in understanding the change processes at play in PMI. Another key assumption of PMIs is that caregivers sustain their implementation of PMIs when their therapist is not present, both in the short term (i.e., between therapy appointments) and in the longer term, following termination of the coaching process. This is a significant limitation because the ‘work’ of PMIs is largely thought to occur outside of treatment sessions and in the family’s natural environment. Although caregivers are able to implement interventions with fidelity when observed by a research team (e.g., Hardan et al., 2015; Yoder et al., 2021b), it is less clear how frequently caregivers use the techniques in their daily lives and whether they maintain these practices long-term. Indeed, one of the only long-term follow up studies of a parent-mediated early intervention found no difference in the primary parent outcome, parent synchrony, at six-year follow-up, although treatment effects for enrolled 7 children were maintained (Pickles et al., 2016). A better understanding of caregiver learning processes in PMIs may point to key mechanisms or moderators of sustainment, which remains a key gap in the literature. Last, despite their potential benefits and growing evidence, parent-mediated NDBIs are underutilized in community settings (Hume et al., 2005; Straiton, 2020; Vivanti et al., 2018). The complexity of PMI, particularly for autism, may present barriers to implementation in community-based service delivery systems (Wood et al., 2015). Recent research suggests that providers working in community service delivery systems (e.g. Part C Early Intervention) may not receive enough training on teaching caregivers, as they demonstrated low fidelity to evidence-based caregiver coaching practices (Pellecchia et al., 2022). In addition, in a community-based implementation of Project ImPACT, families receiving the study intervention did not have significantly higher fidelity to the intervention than families who received usual care, although differences were found in a more general measure of positive parenting behaviors (Stahmer et al., 2019). In sum, although parent mediated NDBI have the potential to support more widespread access to services, there remain several knowledge gaps pertaining to how these interventions work and how they can be implemented optimally in the community. Study Aims: Developing a Theory of Change of a Parent-Mediated NDBI Parent-mediated NDBIs are complex interventions comprised of several putative “active ingredients” or intervention elements thought to cause change in downstream developmental outcomes. These active ingredients are thought to operate via a variety of mechanisms of change, with social communication development unfolding over time as children participate in engaged social interactions with a caregiver. Common methodologies for examining active ingredients and mechanisms of change have significant limitations for studying this type of complex 8 intervention. Additive and dismantling RCTs, for example, are unable to detect interactions between intervention elements (except in factorial designs), require large samples to examine the effects of a single intervention element, and may not capture associated mechanisms depending on the timing of repeated assessments (Papa & Follette, 2015). Single-case experimental designs, which require fewer participants, can be effective for linking active treatment phases and context-dependent effects or mechanisms of change, provided that they change with a brief latency (Kratochwill et al., 2010). Yet, given that NDBIs target generalized social communication outcomes which are thought to develop gradually, such a design is unlikely to capture such effects. Here, we used the Theory of Change framework, a pragmatic, theory-based evaluation framework, to explore the active ingredients and mechanisms of change of Project ImPACT, an empirically supported NDBI for young children with autism or social communication delays. In developing a Theory of Change, we considered not only putative active ingredients and mechanisms of change, but also the service need and context in which the intervention is delivered, as well as short-, mid-, and long-term outcomes associated with the intervention. The aims of this exploratory sequential mixed methods study were to: 1) Develop a comprehensive Theory of Change of Project ImPACT using stakeholder perspectives on potential active ingredients and mechanisms of change, and 2) Provide proof-of-concept of the Theory of Change model using archival data from treatment trials of Project ImPACT. In engaging various end- users of the intervention to develop a theory of change, we were able to consider aspects of the intervention not identified a priori by intervention developers. This approach allowed us to explore different elements of the intervention as they unfold over time and consider a broader scope of intervention processes and outcomes than is typically measured in experimental studies. 9 In the long-term, these results may be used to design prospective experimental studies of the change processes underlying this complex intervention. Chapter 2 presents a description of our methodological approach, which used a phased mixed methods research design. Chapter 3 presents qualitative findings regarding service need and contextual factors around the implementation of Project ImPACT. Chapter 4 presents findings as they relate to the active ingredients, mechanisms of change, and outcomes of the parent coaching model. Chapter 5 presents our findings as they relate to the active ingredients, mechanisms of change, and outcomes of the child-directed intervention as delivered by caregivers. Chapter 6 provides general discussion of the strengths and limitations of the present study as well as future directions for this work. 10 CHAPTER 2: Method Overview This project used an exploratory sequential mixed methods design (Curry & Nunez- Smith, 2015) with multiple phases (Figure 2.1). Phase 1 (QUAL) involved conducting and analyzing semi-structured interviews from key stakeholders. Phase 2 (Mixed) focused on integrating the qualitative data and situating it within the context of existing research and theory to build a plausible quantitative model through a process of connected integration. Phase 3 (quan) used archival data from clinical trials to test hypothesized relationships modeled in Phase 2. Figure 2.1. An exploratory sequential design developed to build and test a theoretical model of intervention process. Conceptual framework: Theory of Change We relied substantially on the Theory of Change framework (Weiss, 1995, 1997) in both conceptualizing and designing this study. Theory of Change is a pragmatic framework for theory-based evaluation which incorporates intervention components, short-term, mid-term, and long-term outcomes, as well as the rationale or mechanistic links that connect them (De Silva et al., 2014; Weiss, 1995, 1997). A key tenet of the framework is that interventions are “based on 11 explicit or implicit theories about how and why the program will work” (Weiss, 1995, p. 66). Theory-based evaluations are meant to identify and test those theories and assumptions. Intervention of interest: Project ImPACT PMI are complex in that they involve delivery of multiple treatment elements at two levels of service delivery (i.e., a provider coaching the caregiver, and the caregiver using the intervention with their child) and take place in reciprocal, dyadic interactions within a dynamic family system (Figure 2.2). Project ImPACT is a manualized, evidenced-based, parent-mediated NDBI for improving social communication outcomes in young children with autism or other social communication delays (Ingersoll & Dvortcsak, 2010, 2019). Because of evidence supporting the efficacy and effectiveness of this intervention (e.g. Ingersoll et al., 2016, 2017; Stadnick et al., 2015; Stahmer et al., 2019; Yoder et al., 2021b), as well as its increasing clinical use in community settings, we believe a theory-based evaluation focusing on refining and testing the program theory with an eye toward optimization is warranted. Figure 2.2. Illustration of multi-level nature of a Theory of Change model of parent-mediated interventions. 12 Phase 1: Qualitative analysis 1.1 Recruitment and study sample Recruitment focused on three key stakeholder groups: intervention experts, community providers, and caregivers. Intervention experts were defined as certified trainer consultants in Project ImPACT. These are individuals who are certified to provide workshops and consultation to train providers in other agencies and possess deep knowledge of the intervention, underlying theoretical foundation, and research base. All certified trainer consultants are known to the developers of Project ImPACT and were contacted by email up to three times to participate. All 10 who were not directly involved in this study participated in interviews. Community providers were defined as providers who are certified in Project ImPACT. The certification process involves reading the intervention manual, completing an online tutorial about the techniques, attending the Project ImPACT Introductory Training workshop, receiving group or individual consultation from a certified trainer consultant, and meeting criteria for fidelity of implementation (including direct implementation of the techniques with a child, coaching an adult, and engaging in collaborative goal setting). As of February 2021, a total of 58 individuals were certified in Project ImPACT. Each individual was contact by email up to three times. Two additional providers received the study flyer from a Project ImPACT certified supervisor and contacted the first author about participation; given their high level of training, they were invited to participate. A total of 22 community providers participated in interviews. Expert and provider demographics are available in Table 2.1. Caregivers were eligible to participate if they received Project ImPACT from a certified coach within 1 year of the time of the interview, regardless of the manner in which the intervention was delivered (i.e. in a group or individual; in-person or via telehealth). Caregivers 13 were recruited through certified community providers (as described above); this was to ensure that caregivers were likely to have received a relatively high-fidelity implementation of Project ImPACT (i.e. representative of how the program is meant to be delivered, and likely to include all of the intended treatment elements comprising the intervention). Community providers were sent a digital copy of the study flyer, as well as a script that could be used by email or text to share with eligible caregivers. The flyer and written script included contact information for caregivers to contact the study team directly about participation. A total of 12 caregivers (including 1 couple) participated in interviews. In terms of treatment modality, 4 caregivers participated in-person using the individual model; 7 caregivers participated in the individual model via telehealth, and 1 caregiver participated in a group model, which began in person but transitioned to telehealth. Family demographics are available in Table 2.2. Table 2.1. Expert and provider demographics. Provider Expert n (%) n (%) Gender Male 0 0% 1 5% Female 10 100% 21 95% Race White or Caucasian, non- Hispanic/Latinx 8 80% 16 73% Black or African American 0 0% 2 9% Asian 0 0% 2 9% Multiracial/multiethnic 1 10% 1 5% Hispanic or Latinx 1 10% 1 5% Level of education Bachelor’s degree 0 0% 2 9% Master’s degree 6 60% 17 77% Doctoral degree 4 40% 3 14% 14 Table 2.1. (cont’d) (% of Provider (% of Expert n cases*) n cases*) Discipline Applied Behavior Analysis 0 0% 5 23% Psychology (School, Clinical, Counseling) 6 60% 5 23% Speech/Language Pathology 4 40% 7 32% General Education 0 0% 3 14% Special Education 0 0% 9 41% Early Childhood Education 0 0% 6 27% Occupational Therapy 0 0% 1 5% Other 0 0% 2 9% Employment Setting Private practice 4 40% 1 5% Specialty center 1 10% 1 5% Hospital/medical center 4 40% 3 14% School: Public 1 10% 3 14% Early intervention program 0 0% 13 59% University clinic 1 10% 3 14% University-academic appointment 4 40% 3 14% Other 2 9% Service delivery setting Clinic setting 4 40% 8 36% In client’s homes 6 60% 7 32% School setting 0 0% 1 5% Telehealth 9 90% 21 96% Other 0 0% 1 5% Note. Respondents could select multiple options for discipline, employment setting, and service delivery setting; percentage of cases are presented and thus do not add to 100%. 15 Table 2.2. Family demographics. Caregiver n (%) Child n (%) Gender Male 2 17% 8 73% Female 10 83% 3 27% Race White or Caucasian, non-Hispanic/Latinx 9 75% 6 55% Black or African American 0 0% 0 0% Asian 1 8% 1 9% Multiracial 0 0% 2 18% Hispanic or Latinx 2 17% 2 18% Level of education Some college/specialized training 3 25% Associate's degree 2 17% Bachelor's degree 3 25% Master's degree 4 33% Marital Status Married; living with partner 12 100% 1.2 Interview guide We developed a semi-structured qualitative interview guide based on knowledge of Project ImPACT and the Theory of Change framework (Kallio et al., 2016). The questions focused on participants’ background, opinions, experiences, and knowledge about the intervention (Morris, 2015). Consistent with the Theory of Change framework, the interview guide covered five major topic areas: 1) service need; 2) coach-parent level active ingredients, mechanisms, and short-term outcomes; 3) parent-child level active ingredients, mechanisms, and mid-term outcomes; 4) family context; and 5) long term outcomes. We took a “narrow and deep” approach to the Theory of Change, focusing narrowly on the scope of the intervention but with a detailed examination of mechanisms and outcomes (Taplin et al., 2013). Although a traditional Theory of Change development process begins with outcomes and works backwards, we adapted the process to improve the flow of individual interviews. The 16 interview guide was comprised of open-ended questions and follow-up probes mapping onto each element of the Theory of Change framework, with an emphasis on causal processes related to specific intervention elements. Additional probes to support elaboration and clarification of ideas were also used liberally throughout interviews. We tailored the language in interview guides for each stakeholder group, with all three interview guides covering the breadth of the Theory of Change framework (Table 2.3). Table 2.3. Sample questions from the semi-structured interview guide. Topic Example Question Example follow up probe Topic 1: Service need From your perspective, what is the Why do you think that’s most important thing that families important for the families gain from Project ImPACT that you work with? they might not get from other services? Topic 2: Coach-parent How did your coach help you learn What effect did [program level active ingredients, the techniques during sessions? element] have on you, your mechanisms, and short- parenting, or your ability to term outcomes use the ImPACT techniques? Topic 3: Parent-child How do children respond when How does [technique] level active ingredients, adults do [technique] in the bring about that mechanisms, and short- moment? [response]? term outcomes Topic 4: Context How do you feel like ImPACT fit How does ImPACT fit or with your role as a parent? conflict with your parenting style? Your values? Your cultural background or upbringing? Topic 5: Long-term Do you think training in Project Are there any ways in outcomes ImPACT affects families more which it affects parents in long-term, for example, a year out the future? from participating? Prior to the start of data collection, the interview was pilot tested using expert assessment, internal testing with colleagues, and a practice interview with a caregiver (Kallio et al., 2016). As data collection unfolded, the provider and caregiver interview guides were adapted to include a 17 common graphic from the Project ImPACT manual which summarizes the intervention elements to facilitate recall and put individuals at ease. 1.3 Interview procedure Interviews were conducted and audio-recorded over Zoom, ranging in duration from 30 to 66 minutes (mean = 47 minutes, SD = 9 minutes). Across participants, this amounted to a total of 33 hours, 7 minutes, 47 seconds of recorded conversation, and 315,607 words of transcribed conversation. The interviewer (KMF) took short-hand notes during interviews to record major topics discussed; these notes were consolidated and used to track data saturation in terms of gaining participant perspectives on all components of the intervention. 1.4 Analysis: Framework Method We used the Framework Method, a codebook-based thematic analysis approach, to analyze the data (Gale et al., 2013; Ritchie & Lewis, 2003). This approach provides structure useful for research teams with multiple coders, is theoretically and philosophically flexible, and is appropriate for use with semi-structured interview data (Gale et al., 2013). Our approach to the data was consistent with a subtle realist paradigm (Hammersley, 1992), which attempts to represent a true reality that exists independent of peoples’ beliefs, but acknowledges that this reality is “only accessible to us via the respondents’ interpretations” with different perspectives engendering “different types of understanding” (Ritchie & Lewis, 2003). This reality is further interpreted by the researcher(s), emphasizing a need for reflexivity throughout the analysis (Gale et al., 2013). The Framework Method proceeds in several stages which are described in detail below: 1) transcription, 2) familiarization, 3) coding, 4) developing a working analytical framework, 5) applying the analytical framework, 6) charting the data into the framework matrix, and 7) interpreting the data. 18 1.4.a. Transcription and Familiarization. Interviews were transcribed by trained undergraduate research assistants and subsequently verified by the lead author. The qualitative coding team, comprised of two research assistants and the lead author (an advanced doctoral student), familiarized themselves with the data through reading and re-reading of transcripts prior to initiating the coding process (Gale et al., 2013). During this time, the lead author began to develop a preliminary codebook, which included deductive codes from the content of the intervention manuals. 1.4.b. Coding. Coding was conducted with computer-assisted qualitative data analysis software (MAXQDA 2020). Multiple first cycle coding methods were applied to the interview transcripts representing a combination of inductive and deductive codes. A detailed codebook was developed and refined throughout the coding process. A subset of sample codes and code types are presented in Table 2.4. We applied structural codes to delineate specific sections of the interview that aligned with constructs of interest (Saldaña, 2013). These codes were developed deductively and include constructs in the Theory of Change framework (e.g., need, long-term outcomes) as well as specific intervention components from the treatment manual (e.g., practice and feedback). The structural codes allowed for indexing and categorizing portions of each interview to facilitate later charting of the data. Alongside the deductive structural codes, we used an “initial” or “open coding” approach throughout to identify information relating to service need, contextual and implementation factors, and long-term outcomes. These inductive codes were identified during the interview and initial consensus coding process, and iteratively refined over time. 19 Table 2.4. Sample codes to illustrate types of codes and hierarchical coding system. Codes Code types Need Deductive structural code Structured program Inductive code Parent-directed intervention elements Category used to organize codebook Demonstration/modeling* Deductive structural code Practice and feedback* Deductive structural code Parent mechanisms and outcomes Category used to organize codebook Hands on learning* Inductive code Fidelity of implementation* Inductive code Note. *Examples of codes used for causation coding, which include treatment elements, mechanisms, and outcomes. Last, we used causation coding to code portions of the interview focused on intervention ingredients, mechanisms, and outcomes at both the coach-parent and parent-child treatment levels. Causation coding focuses on “why questions,” by identifying sequences of causes, outcomes, and mediating variables (or mechanisms) that link causes to outcomes (Saldaña, 2013). These sequences can occur in three parts (cause-mechanism-outcome) or may involve more complex processes involving multiple variables. In addition, different participants may describe the same construct as occurring in different parts of the sequence. Rather than identifying a “true” or “accurate” causal process, causation coding helps identify or hypothesize about potential causal pathways (Saldaña, 2013). Most ‘causes’ (i.e., Project ImPACT treatment elements) were developed deductively and applied as structural codes, and mechanisms and outcomes were identified inductively from the content of the interviews. Because of the complexity of the data, causes, mechanisms, and outcomes were coded separately in the first coding cycle. A second cycle of coding was conducted during the final audits of transcripts. At this stage, comments were created within MAXQDA to describe the causal chains linking cause(s), mechanism(s), and outcome(s) according to each participant. An example from our dataset is illustrated in Figure 2.3. 20 Figure 2.3. An illustration of how causation coding was implemented in MAXQDA2020. 1.4.c. Developing and Applying the analytical framework. We used a collaborative, team-based approach to coding (Hemmler et al., 2020) which is outlined in Figure 2.4. Codebook development was a collaborative and iterative process which began with consensus coding. Initially, all three members of the coding team coded the same transcripts independently and met twice per week to discuss assigned codes and refine the codebook. Nine interviews were coded in this manner, at which point good working agreement on code definitions was achieved and new codes were rarely being added. Next, all coders independently coded 3 transcripts, and inter- coder consistency was evaluated. Both undergraduate raters reached exact agreement of at least Kappa = .60, suggesting good working agreement of the codebook. A consensus-based process was used to code the remaining transcripts, whereby both undergraduate coders independently coded all transcripts, their codes were merged with each other, and then the lead author audited all transcripts and finalized the codes. Biweekly meetings continued throughout this process to provide a venue for reflexive discussion, resolution of discrepancies and questions and prevention of coder drift. To facilitate ongoing conversation and support around codes, an “Other/Unsure” code was added such that coders could mark and comment on sections to discuss as a group (Hemmler et al., 2020). The first twelve interviews coded were subsequently audited near the end of the coding process to account for the final codebook and coding conventions. 21 Figure 2.4. Flow diagram of codebook development and consensus-based coding process. 1.4.d. Charting the data. Data, including coded segments and researcher comments, were exported from MAXQDA and charted using spreadsheets in order to summarize the data by category (Gale et al., 2013). Table 2.5 provides a sample of a charted data matrix including causation codes and researcher comments. The lead author analyzed data for each intervention element by examining co-occurring codes, researcher comments, and reading and re-reading coded excerpts. A developer of the intervention with a deep level of knowledge and clinical experience, reviewed the data charts and interpretation to further refine the analysis. 22 By charting and sorting the data, the research team was able to examine potentially- meaningful variation in descriptions across stakeholder groups via triangulation (Farmer et al., 2006). We specifically considered the concepts of convergence (i.e., agreement), dissonance (i.e., contradiction), and silence (i.e., identified by one stakeholder group but not another). Supplementing this process, analytic memos were written throughout to record researcher observations, particularly recurrent themes which cut across structural codes (e.g., the relationship between intervention elements). Table 2.5. Sample of charted comment data for a parent-directed intervention element. Participant Exported comments pertaining to “Practice and feedback” Caregiver practice and feedback > increase self efficacy practice and feedback + fidelity > attunement to child positive response > reinforces parent > parent positive emotions Expert practice and feedback > corrective feedback > fidelity practice and feedback > attunement to child & positive feedback> parent empowerment Provider practice and feedback > hands on learning > content knowledge/memory 1.5 Research team and reflexivity The lead author conducted all interviews with study participants. The lead author and two undergraduate research assistants conducted the qualitative analysis as a team. The lead author is a doctoral candidate with a master’s degree in a clinical science program within a department of psychology. She is a certified master trainer in Project ImPACT and has experience providing Project ImPACT as a direct clinical service, delivering it in research and clinical settings as a parent training curriculum, and conducting professional workshops and consultation with community providers. Because of her extensive involvement with the intervention, several of the intervention experts interviewed as part of the study were known to her as colleagues. None of the community providers interviewed were her consultees. The two undergraduate research assistants had no prior experience with early interventions for young children on the autism 23 spectrum or with other social communication delays. Prior to coding, both research assistants completed the Project ImPACT Online Tutorial (www.project-impact.org/online-tutorial), a 6- hour web-based training. They also read the Project ImPACT manual for parents, and chapters 1 and 3 of the Project ImPACT manual for coaches in order to familiarize themselves with the content of the intervention and the names of different intervention techniques. 1.6 Trustworthiness The trustworthiness or rigor of qualitative analysis can be described by several interrelated concepts including dependability, credibility, and transferability, which are akin to the concepts of reliability, validity, and generalizability in quantitative inquiry (Lincoln & Guba, 1985; Morse, 2015). Several strategies were used to improve the rigor of this work, including an audit trail which documented the study procedures. The data collection, coding, and analysis process are detailed in depth in this manuscript to further support transferability of our findings (Mays & Pope, 2000). Reflexivity, or attention to how the research team and study design may shape the analysis process, was discussed and considered throughout the process (see “Research team and reflexivity” section above for additional detail). A semi-structured interview with frequent use of probes allowed the interviewer to elicit in-depth descriptive information from respondents. Thus, the dataset is comprised of a thick, rich description of intervention processes. Furthermore, the purposive sampling strategy, which targeted a relatively large sample from different stakeholder groups, included individuals with extensive expertise in the topic as well as the intended end-users of the intervention program, indicative of incorporating a range of perspectives about the intervention process (Mays & Pope, 2000). Taken together, these contribute to the credibility and transferability of the findings (Morse, 2015). Development of a codebook and evaluation of inter-coder agreement, coupled 24 with our consensus-based coding approach with frequent reflexive discussion among the coding team, support the dependability of the findings. In addition, the thick descriptions afforded by in- depth interviewing, provided opportunities for duplication or replication of results across participants within and across stakeholder groups. 1.7 Sample adequacy What is considered an adequate sample size is a widely debated topic in qualitative inquiry (Malterud et al., 2016; O’Reilly & Parker, 2013). Although the concept of ‘saturation’ has been widely applied, its meaning is not clearly defined across qualitative methods, and some have argued that the idea of ‘completeness’ implied by saturation is both logically impossible and contrary to the qualitative research process (Bowen, 2008; O’Reilly & Parker, 2013). To describe the adequacy of the sample for this project, we draw on the concept of information power, which is modified by five dimensions that vary across studies: breadth of the study aim, specificity of the sample, use of established theory, quality or depth of the interview, and analysis strategy (Malterud et al., 2016). Several characteristics of this study increased our information power to address our research questions. First, we purposively recruited participants with highly specific knowledge about the intervention across three stakeholder groups, allowing for a breadth of experiences while focusing on individuals with adequate knowledge of the research question. Our use of an established theoretical framework as a guide (i.e., Theory of change) and the ability to draw on established theory to analyze and integrate results (e.g., behavioral or learning theory, social interactionist theory) further increase the information power of this work. The circumscribed nature of our primary qualitative research question, focusing on individual components of a specific manualized treatment, increases our information power. However, broader facets of theory of change (e.g., need, context), might limit information power 25 in those areas. Quality or depth of dialogue in the interviews varied across respondents; although most respondents provide detailed, descriptive accounts of information process, some respondents had a difficult time articulating specific change processes. However, the lead author’s expertise in the intervention, as well as her experience working with providers and families, enabled her to connect with participants and obtain detailed information from most respondents. Last, our approach to analysis using the framework method allows for within- and across-case analysis, suggesting a need for enough cases to compare. Taken together, this study had relatively high information power, suggesting that a limited number of interviews is sufficient to address the research questions. We also believe the adequacy of our sample was reflected in the coding process; codes were consistently being added to the codebook early in the coding process and new codes were no longer being added for the last several interviews, suggested that saturation was achieved. Phase 2: Using connected integration to develop a theory of change 2.1 Connected integration using joint displays Phase 2 of the research was characterized by connected integration (or “building”). Charted comments and interview excerpts from the qualitative analysis were used to develop plausible causal models of processes underlying the intervention of interest (Curry & Nunez- Smith, 2015; Fetters et al., 2013). Causal models (often involving multiple potential mechanisms and outcomes) were visualized for each intervention element. To facilitate and document the analytical process, we developed joint displays to simultaneously present the quantitative causal models alongside qualitative codes, interview quotes, and a narrative description of the change process. 26 2.2 Theory of Change development and integration with mid-level theory. Following the development of the causal models for individual intervention elements, we combined them by creating two integrated models. The first focused on the change process at the coach-parent level of the intervention, and the second focused on the change process at the parent-child level of intervention delivery. We specifically looked for shared mechanisms and outcomes and intervention elements thought to work in similar ways in order to create integrated, cohesive, and parsimonious Theory of Change models. In addition, we considered how the change processes related to relevant established psychological theories, which would provide further support for the models. Phase 3: Quantitative analysis To examine preliminary proof-of-concept for our model, we examined four mediation models developed from the results of Phases 1 and 2 of the present study. Models 1 and 2 focused on the coach-parent intervention level (presented in Chapter 4) and Models 3 and 4 focused on the parent-child intervention level (presented in Chapter 5). We used the integrated theory of change models to identify select paths that could be tested quantitatively using available archival data from two studies (a pilot and full-scale RCT) examining the efficacy of Project ImPACT delivered using an online telehealth platform (Ingersoll et al., 2016). The online platform included various materials such as a digital treatment manual, narrated slideshows with video examples, and a video library. In the full-scale RCT, participants were randomized to one of three groups: therapist-assisted (receiving twice-weekly telehealth coaching comprised of content review and practice with feedback), self-directed (receiving access to the Project ImPACT online platform without coaching), and resource 27 support (receiving access to a general online resource library and monthly support phone calls). The pilot study included only the therapist-assisted and self-directed groups. 3.1 Participants Participants in this study included 92 children who were 18 to 93 months old (M=46.6, SD=17.0) and a primary caregiver who participated in either a pilot RCT or full-scale RCT of Project ImPACT Online. Children were included in the studies if they had a community diagnosis of autism spectrum disorder or suspected autism, confirmed by administration of the Autism Diagnostic Observation Schedule (ADOS-G or ADOS-2; Lord et al., 2000, 2012), and limited language skills (i.e., expressive language age equivalent of less than 4 years at study entry). Caregivers had to be proficient in English, but could speak other languages in the home. Child and caregiver demographics can be found in Table 2.6. 3.2 Statistical Approach We selected specific active ingredients (predictors), mechanisms (mediators), and outcomes from the integrated models developed in phases 1 and 2 pragmatically based on available data. Because we relied on archival data to conduct phase 3, the statistical models represent what was feasible for the scope of this project rather than ideal analyses. Detailed descriptions of each model to be tested, along with the study measures used to quantify relevant constructs, can be found in the respective results sections in Chapters 4 & 5. 28 Table 2.6. Demographics. Primary Caregiver Demographics n % Child Demographics n % Gender Caregiver-reported sex Male 12 13 Male 68 26 Female 80 87 Female 24 74 Race Race White 52 57 White 67 73 Black or African American 4 4 Black or African American 7 8 Asian 6 7 Asian 6 7 More than one race 2 2 More than one race 11 12 American Indian/Alaska Native 1 1 Other race 1 1 Missing 27 29 Ethnicity Ethnicity Hispanic or Latino 5 5 Hispanic or Latino 7 8 Not Hispanic or Latino 60 65 Not Hispanic or Latino 58 63 Missing 27 29 Missing 27 29 Education Level Some high school 1 1 High school graduate 6 7 Some college/specialized training 35 38 4-year college 24 26 Graduate degree 26 28 Marital Status Married; living with partner 62 67 Single; divorced or separated 9 10 Single; living with partner 5 5 Single; never married 12 13 29 CHAPTER 3: Results; Service need and context Two key components of the Theory of Change framework include defining the need a program or intervention addresses as well as the context in which the intervention takes place. Service need Important aspects of need for a service like Project ImPACT fell into four primary themes: 1) Parent-driven and parent-led service; 2) Structured and systematic program; 3) General knowledge and understanding about child development; and 4) Child-centered service. Together, these themes are consistent with family-centered care, which was described as lacking in some community service contexts: “in my experience with other clinics in this area, they usually are not family centered. It’s more you come in, you do your therapy, you leave. And maybe they know what you’re working on and maybe they don’t” [provider]. Across themes, respondents often discussed links to caregiver empowerment with regard to caregivers’ ability to move through the program successfully, foster positive connections with their child, understand their child’s development, and participate in a service that felt like a good fit for their family. Each theme (as well as subthemes, where applicable) alongside supporting quotations are described below, and summarized in Figure 3.1. 30 Figure 3.1. Qualitative themes pertaining to need for a service like Project ImPACT. Theme 1: Parent-driven and parent-led service Respondents in all stakeholder groups talked about the need for family-centered services that prioritize caregiver goals, values, and involvement in their child’s therapy. The coaching relationship was often described as a collaboration or partnership (“they’re actually a partner in their child’s therapy and in their child’s learning” [provider]) with the goal of supporting the caregiver in being able to implement the techniques without provider support (“it ultimately works us out of the equation, which is exactly what we need to have happen over time” [provider]). Importantly, respondents linked caregivers’ direct involvement in implementing the intervention techniques, to caregiver empowerment: caregivers are “able to own the progress of their children a lot more than relying on a provider” [provider]. Through increased knowledge and skill development, caregivers feel more efficacious in fostering positive interactions with their children: “a lot of the times we feel a little bit helpless, we don’t know how to help them, we don’t know what to do with them, so it’s really empowering to be able to know how” 31 [caregiver]. Two subthemes, parent connection with their child and tailoring, illustrate different facets of parent-driven and parent-led services: Subtheme 1a. Caregiver connection with their child. Respondents in all groups talked about the importance of supporting caregivers’ connections with their children by helping them attune to the child’s communication and supporting positive social engagement. Several respondents talked about many families who begin services often have a history of negative interactions with the child (“I kept running and hitting a wall” [caregiver]; “we have this cycle of rejection and parents being – like, walking on eggshells” [provider]). These negative interactions were described as affecting caregiver wellbeing: “It feels awful to, you know, not be able to build a relationship and feel connected with a child, and I think parents feel guilty, they feel stressed, they feel disheartened” [expert]. By teaching families that “you don’t need to verbally communicate in order to communicate, communication is in everything a person does” [provider], Project ImPACT helps provide “a window into what the child is thinking” [provider] and supports connections between caregivers and their children. One caregiver reflected: “just understanding the impact of responding to him any time that he makes a bid of attention towards us has really changed our relationship and in really powerful ways.” Subtheme 1b. Individualizing. Respondents also discussed that individualizing or tailoring the intervention is an important part of family-centered services: Every family is different, every child is different, households work differently […]. By really listening you are getting to know that family’s day to day, you’re understanding their values, their parenting styles, so that you as a coach, you can make sure to fit this intervention into their family in a way that’s more meaningful. It’s not a cookie cutter, one-size-fits-all situation. [expert] 32 This individualizing supports caregiver motivation: “it was tailored to us and our needs. And about us. And yeah, I certainly think that that had an impact on our willingness and desire and motivation to carry out the plan” [caregiver]. Theme 2. Structured and systematic program Respondents in all stakeholder groups explained how the structured and systematic nature of the program filled an important need for families. Several caregivers noted the structure fit well with their personality or lifestyle: “I’m very Type A, and so having that plan, and those different, you know, setting those goals and being so specific was something I really appreciated.” The step-by-step nature of the program was described as supporting caregiver success in learning and retaining the intervention content, as well as their confidence in carrying out the techniques: “that systematic approach of teaching really helps the parents focus and home in and then they feel confident enough to continue to move forward” [provider]. Several respondents also talked about how the structured, goal-oriented curriculum is different from other types of services which are more focused on problem-solving one-off concerns rather than teaching caregivers skills that they can generalize across situations and routines. Theme 3. General knowledge and understanding about child development Stakeholders also described caregivers’ desire to learn about child development, autism, and social communication delays: “in any other program – I mean we did try other programs – we would have felt much more passive or truly ignorant, because we didn’t know, we didn’t know, and we wanted to know, and we didn’t have the right information or support” [caregiver]. Having access to this information was described as helping caregivers understand their goals and clarify expectations about how their child might respond to their use of the techniques. Some respondents described having this additional knowledge as empowering, although some also 33 acknowledged that it could be difficult too: “[it] helps them to see what they should be expecting and maybe what’s missing, because if they’re really difficult, it’s a difficult conversation. It’s so hard for parents, but it is also really eye-opening because it does help them to see where we are” [provider]. Theme 4. Child-centered service Caregivers discussed their appreciation for the child-centered nature of the Project ImPACT strategies as well: “what I loved about Project ImPACT was it was very child based and child-led, it was, you know, get in their environment, watch what they’re doing, copy them, see what they’re interested in, use that – and it was just didn’t really feel forced, it felt very natural” [caregiver]. Some caregivers noted the difference between this approach and other services available in their community, which supported their confidence in obtaining services that were a good fit for their child and family: Also feeling confident as a parent that we can help him through the – like the challenges that he has because of autism, that we can really support and advocate for his needs, and that he can learn outside of 25 hours of ABA a week, that he can learn in a different way and still learn. Because I feel like a lot of parents feel like that’s the only option. And that just doesn’t work for every family and so, that has been really cool for me and my husband. To be the ones just be there with him, helping him. Context: Fit for family Important aspects of context or family fit for a service like Project ImPACT fell into four primary themes: 1) Child factors; 2) Caregiver factors; 3) Family factors; and 4) Cultural and linguistic factors. Each theme and subthemes alongside supporting quotations are described in the sections below, and summarized in Table 3.1. Generally, providers and experts discussed 34 their role in being responsive to families’ needs with regard to fit (“the onus really is on the clinician to figure out the style of coaching that works for that parent” [expert]), and noted that in many cases, barriers can be compounded or interact with each other to affect overall fit (“it’s not solely on the basis of culture, right? Usually, it would be culture plus challenging behavior, culture plus, like, not wanting, I don’t know” [expert]). Table 3.1. Overview of themes and subthemes describing intervention fit to context. Themes Subthemes 1. Child factors 1a. Child age 1b. Goals or needs outside of social communication 1c. Child verbal ability 2. Caregiver factors 2a. Parenting style and values 2b. Caregiver mental health and wellbeing 2c. Commitment and motivation 3. Family factors 3a. Time to implement 3b. One-on-one time 3c. Challenging life circumstances 3d. Involvement of multiple caregivers and family members 4. Cultural and linguistic 4a. Cultural facilitators factors 4b. Cultural and linguistic barriers 4c. Adaptation Theme 1. Child factors Respondents described child factors that affected how well Project ImPACT fit for a given client. Child factors included subthemes relating to (a) child age, (b) goals or needs outside of social communication, and (c) child verbal ability (Table 3.2). Respondents felt that children who were younger and children who were in the early stages of developing communication skills were more appropriate for Project ImPACT, and those who had more pressing needs outside of social communication (e.g., dangerous behavior or medical needs) were less appropriate. 35 Table 3.2. Child factors that affect perceived fit for families. Theme/Subtheme Representative quotations Child factors 1a. Child age “Project impact would have been more helpful at a younger age.” [caregiver] “I think the better fits are when we get these parents in really early.” [expert] 1b. Goals or “I will say if a child has a lot of challenging behavior, or other more, needs outside of pressing behavioral needs or medical needs, I think it’s best to start social with that behavior module that we now have, to manage that first.” communication [expert] “if the parent is not having explicit social communication goals with their child then I would not think Project ImPACT would be the best option.” [expert] 1c. Child verbal “kids [who] are getting to the top of that social communication checklist, ability parents are marking ‘usually’ on most of it, then you know the strategies you know feel less applicable for them.” [provider] “I make more of a difference with a child who’s a very beginning communicator, doesn’t have any words, than with a child who’s much more verbal. I feel like it’s easier to implement, it’s easier to see changes, than it is with a child who has a lot more verbal skills.” [provider] Theme 2. Caregiver factors Subtheme 2a. Parenting style and values. Many caregivers in the sample talked about how Project ImPACT fit well with their parenting style and values. For example, one caregiver talked about how “we’ve been trying to reach her and to connect with her […] before we learned all this stuff, so, we have now a different way to do it.” Caregivers also noted that in some cases, they were already doing similar things at home: “it was exciting to see some of the things, I had already implemented were part of the program so that was encouraging.” However, respondents also talked about how in some cases, caregivers felt less comfortable participating when strategies diverged from their usual style. One caregiver described how some of the techniques did not come as naturally to her: “I think the ‘use animation’ – I’m kinda more reserved, like that one was harder for me. My husband is more 36 animated and more louder with him versus me, I’m kinda a little quieter with playing.” Similarly, one provider said that they worked with one family where one caregiver “believed in a more hands off parenting approach in general” and seemed uncomfortable with the techniques; this caregiver chose to discontinue the program, even though their spouse continued to participate. Subtheme 2b. Caregiver mental health and wellbeing. Providers and experts described that caregiver mental health concerns can make participation in Project ImPACT challenging, and sometimes referrals to additional support are needed (“I tend to add social workers on the case as well” [provider]). Respondents noted that this could be particularly relevant shortly after a child’s initial diagnosis of autism: Parent mental health can be tough. We are using the program with kids who are newly diagnosed, like fresh out of the eval. And I think that, it’s not that the program isn’t a good fit, I just feel like there’s a miss in helping parents with some basic things before they get started with the program [Expert]. Caregivers in the sample did not note any barriers to participating related to mental health. Subtheme 2c. Commitment and motivation. Many respondents explained the importance of caregiver motivation in supporting fit: “The question needs to come from them, they need to want to participate in ImPACT” [expert]. When motivation is high, “if the parents really commit to practicing and using the techniques between sessions, they’re a lot more successful from the beginning of the program to the end” [provider]. On the other hand, some respondents described “nonchalant parents” [provider] who do not seem to practice techniques between sessions or are “just going through it ‘cause they know they’re supposed to” [provider]. Caregivers in the sample also described their motivation to participate: “it’s all about him when 37 he comes home from school, we wanna play, and we wanna do things with him and take him places, and his development is hugely important to us.” Theme 3. Family factors Subtheme 3a. Time to implement. Providers and experts discussed the intensive time commitment necessary to participate in parent-mediated approaches, and how “someone who’s able to stay at home with their kid has an easier time following through and keeping those strategies than someone who doesn’t” [provider]. Although several noted cases where families with working caregivers were able to participate successfully, they also noted that it was a barrier for some families: “when you are literally working a 9-to-5 and trying to figure out where to put time in for practice, you know, it’s hard” [provider]. One provider also noted that when children are “programmed to the max […] it does become pretty challenging for [caregivers] to find that 15-20 minutes to do play practice.” While lack of time was often cited as a barrier, fitting the program in into daily routines was described as alleviating challenges with finding time to sit and play with a child. One caregiver described: “You’re incorporating it into your daily routine so it would not feel like this extra burden on our plate to add to our workday, to add to our crazy schedule – it just – it fit.” Similarly, one expert said: “the fact that it’s just integrated into their day, makes it a more realistic thing and more likely to use the strategies if they don’t necessarily need to restructure their days around them.” Subtheme 3b. One-on-one time. Respondents also discussed how it was difficult for some families to engage in one-on-one interactions with the child receiving services when there are multiple children in the home. For example, one mother of twins said: “that makes me feel really guilty if you can’t carve out that time for your one child, right? Because it’s hard for me to 38 get that time with [my child] because I’m usually home by myself with two toddlers.” Providers also noted this is a challenging to bring to families with several children: “with those families, it’s just hard to just kind of tell, ‘okay we’re really going to focus on one child right now’” [provider]. Subtheme 3c. Challenging life circumstances. Some respondents also noted that families experiencing major stressors (e.g., financial or housing insecurity) may not be able to follow through with aspects of the program when “their priorities are to keep their kids fed and clothed and safe” [expert] and “this kind of a program just isn’t at the top of their hierarchy of needs” [expert]: I had one family who was – mom lost her job, and they were evicted, and they just couldn’t keep up with the program, but it had nothing to do with the program, it had to do with – the family was in crisis. [provider] In these situations, caregivers and their providers agreed that working on social communication was not a top priority. Subtheme 3d. Involvement of multiple caregivers and family members. Respondents in all stakeholder groups talked about the role of multiple caregivers and family members – including siblings, cousins, grandparents, babysitters, nannies, nurses, and teachers – in Project ImPACT. Caregivers explained how part of their role as a primary caregiver is to foster their children’s “connection to other people too, that are important for [them] to have.” One mother explained: “I am the one that does all the things. I change 90% of the diapers, I do 99% of the baths, I feed maybe 80% of all the meals, […] so as a result that I have to not only do it myself and do it right, but also teach other people.” A challenge described by one provider was when multiple family members or caregivers had conflicting perspectives on the child’s skills and how 39 best to interact with them: “that is a hard situation when one parent is trying to implement the strategies and do what you’re doing, and then the other parent won’t do that at all” [provider]. Involving several important people in the child’s life by teaching them ImPACT techniques was broadly described as a positive and helpful experience for the child and family, even though it could sometimes make service delivery more complex: “[it] was tricky when you had six people for one family, but really what dedication is that!?” [expert]. Theme 4. Cultural and linguistic factors Subtheme 4a. Cultural facilitators. Cultural facilitators or strengths were infrequently identified in our dataset but are important to consider. For example, one mother described: I’m a Latina myself and we are very engaged in our kids’ lives in the early years and we want to be there, […] so that’s the expectation – that you need to play an active role, you need to be there, it’s important – so for me that was very easy. In her case, the congruence of cultural norms and caregiver involvement supported her motivation to participate in Project ImPACT. A provider also noted that video models can be a useful tool for engaging diverse families who may feel unsure whether families in their culture can be successful in this type of program: Every single time we get to […] the first video where it’s not just like, white mom, white dad, white child (laugh), and they’re like ‘Yes!’ and then the buy-in happens immediately. Because they’re like ‘Okay. My community can do this,’ it’s an interesting thing. Subtheme 4b. Cultural and linguistic barriers. Providers and experts spoke about cultural and linguistic barriers to participation in Project ImPACT (Table 3.3). Some barriers had to do with the intervention content (e.g., emphasis on play, following the child’s lead), and others 40 were related to coaching roles and communication among providers and clients from different cultural backgrounds or who speak different languages from each other. Table 3.3. Cultural and linguistic barriers: themes and subthemes. Theme/Subtheme Representative quotations Cultural barriers Emphasis on “I had a mom say to me ‘we don’t do that in my culture, I don’t get on the play floor with my kids. […] we don’t follow our child around and play with what they’re playing with.’” [provider] “there are some families or some backgrounds um where playing with your child is not a part of their lifestyle you know. It’s more of they play on their own or they play with other kids in the community you know.” [provider] Family roles “so the father, even though the mother was the primary caretaker for this and child, the father only wanted to interact with me and didn’t really want participation in the mom to have an active role in the communication with me, so that coaching was a little bit challenging because I couldn’t directly coach with her” [provider] “I find that other parents in, sometimes culturally, like, parents have more like deference for you as their professional, or you decide whatever you think is best” [provider] Following the “following the child’s lead was difficult because in their culture, you child’s lead know, the parent is the one in charge and the child does what the parent says and so you’re really flipping that power dynamic in their mind.” [expert] “for some of these families no, children are to obey or listen, do what they’re supposed to do when I tell them to do it, and why on earth would I get down on the floor and follow what they’re doing and have them be in control?” [provider] Linguistic barriers Communicating “you can show them certain techniques and they can imitate that, but they nuances don’t really understand why they have to use a certain technique or in which kind of situations.” [expert] Coaching “when I’m trying to communicate something through an interpreter, there through an is just so many ways that that can go wrong” [provider] interpreter “that’s extremely hard to do with real time interpretation, since it’s very, very fast.” [provider] Lack of “and maybe doing the practice plan verbally because they’re not gonna – translated if they’re not reading in English then writing it in English is not gonna materials be helpful.” [provider] “it’s been difficult at least for me with when I use it with my bilingual families, in terms of not having the book yet or the, like not all the handouts being translated” [provider] 41 Subtheme 4c. Adaptation. Although a variety of barriers were noted, respondents also discussed the role of the clinician in being flexible, open, and making necessary adaptations to support diverse families in participating in ImPACT “in a way that is respectful and individualized” [expert]: My examples might just be different, and I might need to spend some more time getting feedback from parents or seeing the parents interact and find out more about that culture, but I’ve always been able to adapt it to work and still keep that fidelity [expert]. Specific adaptations include spending additional time on the program (“we definitely have to work a lot harder it seems, it takes longer to get through the program. There’s definitely certain parts where I notice we have to review for an extra couple sessions” [provider]), emphasizing/de- emphasizing certain techniques to “meet somewhere in the middle” [expert] between cultural norms/values and Project ImPACT guidelines, emphasizing verbal over written material, and explaining content in multiple ways “so that we can make sure we have a shared understanding of what we’re talking about” [provider]. 42 CHAPTER 4: Results; Coach-parent level intervention processes Phases 1 and 2 Active ingredients, mechanisms of change, and outcomes Respondents described the adult learning process focusing on several potential active ingredients of caregiver coaching in Project ImPACT: collaborative goal setting; didactics, book/written materials, and give the rationale; demonstration/modeling; practice and feedback; plan for practice; and reflection on practice. The results of the mixed methods analysis using joint displays to develop a causal model for each of these intervention elements can be found in Tables 4.1-4.6. 43 Table 4.1. Joint display illustrating the causal model of the change process for “Collaborative goal setting,” presented alongside supporting qualitative data. Causal model Role in theory of change Codes Active ingredient Mechanism Short-term outcome Mid-term outcome Collaborative goal setting ◼ Parent motivation ◼ Parent empowerment, self- efficacy, positive emotions ◼ Practice outside of session ◼ Child skill development ◼ Supporting Quotations Caregiver It was initial enthusiasm, for sure, and it was sustained enthusiasm too – you know really try to like do our homework, and do this, and really focus on it and whatnot. And so I think that was something that kind of made us feel good. We’re also motivated to accomplish our goals. [...] it kind of made us feel good that we could actually do some of this stuff and maybe see progress for daughter. Provider I think it changes this idea that the therapist is, that the therapist knows best. So I think it puts them back in the expert role for their child. Expert I mean the more motivated the parent is to help their child achieve the goal, the more likely they are to work towards it. [...] They’re going to practice more. And what we do know from the literature is the more practice, the intensity of services, the better the outcomes for children. Narrative change process Collaborative goal setting, which includes partnering with families and validating their goals, increases parents’ motivation and sense of empowerment and self-efficacy. Working on parent-driven goals increases their motivation to participate in the program, which increases the likelihood that they implement the program with their child; this, in turn, supports the child’s development. 44 Table 4.2. Joint display illustrating the causal model of the change process for “Didactics, Book/written materials, and Give the rationale,” presented alongside supporting qualitative data. Causal model Role in theory of change Codes Active ingredient Mechanism Short-term outcome Mid-term outcome Didactics & Book/visual materials ◼ Give rationale ◼ Simplify/focus content ◼ Content knowledge & memory ◼ ◼ Parent motivation ◼ Parent empowerment, self- efficacy, positive emotions ◼ Supporting Quotations Caregiver Sometimes when you read something you don’t always get a full picture of what it’s supposed to look like, and I think when she would explain it it would give me more of a full picture like ‘oh that’s what that means, I get it now,’ you know? In the book I did like all of the different examples it would give though, all of the different types of prompts you could give, I did like that. Provider they need the teaching part to understand what they’re doing and why they’re doing it, because that is really where we get some of the buy in Provider I think that it helps them to kind of clarify so that they are more comfortable with really, you know, practicing on their own. Expert By having the key elements, this is what you’re supposed to do, and understanding why, improves their motivation. (…) [if] they understand how the strategy is going to help them achieve the goal, it increases the motivation because they can tie it together. Narrative change process Verbal and written explanations were described as helping caregivers learn and remember the intervention content by helping to simplify and focus on the important parts. Giving the rationale or explaining the purpose of the technique was viewed as especially important for motivation. Increased understanding coupled with motivation were described as increasing parents’ sense of comfort and efficacy in delivering the techniques. 45 Table 4.3. Joint display illustrating the causal model of the change process for “Demonstration/Modeling,” presented alongside supporting qualitative data. Causal model Role in theory of change Codes Active ingredient Mechanism Short term outcome Mid term outcome Demonstration/Modeling ◼ Visual learning ◼ Content knowledge & memory ◼ ◼ Attunement to child response ◼ Positive emotions ◼ Parent empowerment, self- efficacy ◼ Parent fidelity ◼ Supporting Quotations Caregiver It’s hard to imagine what that looks like in your life (laughs) until you see someone model it then you’re like ‘oh we can do that.’ Provider Once they see it in action then it makes it just more concrete for them, and it gives them a model to imitate so that they understand what it is that we’re asking them to do. Expert If you are able to watch somebody, it can make you feel more comfortable. You’re demonstrating how it should look. And you also then can show the effect that it’s having on the child. So whether it is that the child is responding as you anticipate, or whether it is that they’re not, but it allows the parents when they’re not involved in the interaction to see the correlation between one’s behavior or use of the strategy and their child’s response, which I think, again, increases motivation because they’re like, ‘Okay this is how I can get towards the goal that I want to achieve.’ Narrative change process Demonstrating or modeling the intervention provides a visual example which increases caregivers’ understanding of the techniques, which then increases their ability to use the techniques with fidelity. Getting a better understanding of the techniques as well as attuning to how their child responds (or, another child, if using a video model) increase parent comfort and sense of self-efficacy with regard to implementing the intervention. 46 Table 4.4. Joint display illustrating the causal model of the change process for “Practice and feedback,” presented alongside supporting qualitative data. Causal Model Role in theory of change Codes Active ingredient Mechanism Short-term outcome Mid-term outcome Practice and feedback ◼ Positive feedback ◼ Corrective feedback ◼ Hands on learning ◼ Content knowledge & memory ◼ Reinforcement ◼ Fidelity of implementation ◼ ◼ Attunement to child response ◼ ◼ Parent empowerment, self- efficacy, positive emotions ◼ ◼ Practice outside of session ◼ Child skill development ◼ Supporting Quotations Caregiver I think it just gave me a sense of confidence that I can do this by myself […], and you know, it’s kind of now been like a beginning muscle memory, like I kinda know how to exercise it at a later date, maybe in a different context entirely. Provider Oh my goodness it has a dramatic effect because they’re able to see (laughs) they’re able to see that these strategies work! You know they’re able to see it and once they see it again you know I don’t wanna keep using the same word but it motivates them to, you know, continue on and actually do the practice sessions Expert I think it increases their fidelity because you’re able to tell them the small change to make, to use the strategy in the accurate way. So in the moment, they’re practicing using the strategy accurately, which is gonna increase their ability to use it when you’re not there. Narrative change process Practice and feedback facilitates a “hands-on” learning process which improves caregivers’ understanding of the intervention techniques and helps them use them with high fidelity (which, in turn, supports child skill development). There are also motivational processes involved. Parents receive positive reinforcement both through direct feedback from their coach and by attuning to their child’s responses. Together, these support parent empowerment and self-efficacy, which encourages parent use of the techniques outside of coaching sessions. 47 Table 4.5. Joint display illustrating the causal model of the change process for “Plan for Practice,” presented alongside supporting qualitative data. Causal model Role in theory of change Codes Active Mid-term ingredient Mechanism Short-term outcome outcome Plan for Practice ◼ Simplify/Focus content ◼ Tailoring & adaptation ◼ Content knowledge & memory ◼ ◼ Perceived feasibility ◼ ◼ Parent empowerment, self- efficacy, motivation ◼ ◼ Practice outside of session ◼ Supporting Quotations Caregiver Having a concrete like time of day, a situation, like it made me mindful of doing it. And having like it just like that, building in the routine […], it just it was easy. Provider It’s really, really specific, so I think it’s easier for the families to remember what they’re supposed to do. And then it’s already built into their schedule anyway because the child’s already doing what’s on the practice plan. Expert I think it just increases again the likelihood that you’re going to feel successful in that interaction and I think the more confident and successful a parent feels in the interaction, the more likely they are to continue to use the strategies throughout the week and during daily routines. Narrative change process Planning for practice gives parents a simple, concrete plan that is tailored to their family’s needs and daily routines. This gives parents a concrete idea of how to carry out their practice, and also makes practice seem more feasible and achievable for them. The combination of knowledge and confidence in their ability to carry out the techniques supports practice at home during the week. 48 Table 4.6. Joint display illustrating the causal model of the change process for “Reflection on practice,” presented alongside supporting qualitative data. Causal model Role in theory of change Codes Active ingredient Mechanism Short-term outcome Mid-term outcome Reflection on practice ◼ Adaptation/individualization ◼ Attunement to child response ◼ Parent empowerment, self efficacy, positive emotions ◼ Content knowledge & memory ◼ Supporting Quotations Caregiver If something wasn’t going well I could look back on what went well the week before, and, you know, we could go back to trying that... because you forget what goes well. […] it’s always encouraging to know that something was going well and that we can (laugh) and that we were doing something, and you know we can see the differences in him too – like um the different practices that would work well for him, and we still use them. Provider We can kind of tease out like, how can we make that more – how can we make it better for you, or if you, have you be able to take what pieces are most important and bring it home. Expert I think parents can – like carving out that space for them to reflect on what it felt like, how comfortable or uncomfortable it was, what they thought they were doing well and what they wanna practice more to improve on, but also like really reflecting on like how the child responded. Narrative change process Reflecting on practice helps parents identify aspects of the intervention that went well and those that did not, and gives coaches an opportunity to help parents adapt or individualize the techniques to increase their understanding how best to implement the techniques with their child. Reflection also helps parents attune to their child’s response to the techniques and recognize strengths, which supports parent empowerment and self-efficacy. 49 Downstream outcomes Theme 1. Sustainment. All caregivers who participated in interviews, including one caregiver who was critical of select strategies in the intervention program, described continued use or intent to use the program in the long-term (“I don’t plan on dropping anything that we have learned” [caregiver]). Some respondents described how ImPACT has broad effects on parenting more generally (“it’s really kind of like a compass” [caregiver]; “it makes you look at the way you communicate in a totally different light” [provider]). Respondents in all groups talked about role of implementing in daily routines such as bath time, meals, and dressing in supporting sustainment (“because it’s in these daily routines I think it is easier for parents to keep using these techniques” [expert]; “implementation feels easy because […] you just do it in your daily routines” [provider]). This suggests that teaching the intervention in daily routines throughout the program may be particularly important for supporting long-term use. Experts and providers sometimes expressed uncertainty about the extent of sustained use, given that few had the opportunity to directly observe clients after ending services. However, in one clinic that offers sessions “for a checkup or a boost,” one individual described that long-term use varies across caregivers: “it’s amazing how many of them still using it. Do all of them use it? No. Do all of them get it? No” [expert]. Some respondents surmised that children’s response to the techniques might have a role in whether caregivers continue to implement strategies more long-term (“if they see progress they will be motivated to keep on using these techniques” [expert]). Subthemes relating to sustainment, which include (a) becoming habitual; (b) generalizing intervention skills to different situations; (c) generalizing intervention skills to different developmental stages; and (d) partial or selective sustainment, can be found in Table 4.7. 50 Table 4.7. Themes and subthemes relating to sustainment. Theme/Subtheme Representative quotations 1. Sustainment “we still use it all the time” [caregiver] “I actually think about it more as a long-term kind of parenting strategy” [expert] 1a. Becoming “I think actually we use it so much that we forget that we’re using habitual it” [caregiver] “ultimately it just becomes sort of like the way they interact with their kids” [expert] “certain concepts become more habitual with how you interact with your child” [provider] 1b. Generalizing “the more the coach succeeds in teaching the parent to generalize intervention skills to these skills and use them in a lot of different situations and different situations activities, that will help the parents to keep on using them” [expert] “they are able to really practice and embed these strategies into their daily activities, everything that they can do. […] they’re creating those learning opportunities anywhere” [provider] 1c. Generalizing “we’re teaching parents to be able to be more flexible and to adapt intervention skills to strategies based off their child’s skill level and so I think that it different developmental has longer-term impacts” [expert] stages “parents can adapt these techniques to their child growing and maturing and adopting new skills like we mentioned earlier” [provider] “it has different phases like where your child may be, so if he’s already succeeding doing one thing, you can just move up the kind of difficulty level and try the next one up and see how he does” [caregiver] c. Partial or selective “I do some of the imitation work, always do prompting, I try to sustainment imitate her whenever, and the your-turn my-turn, I would say some things I’ve carried over and that we continue to use for sure.” [caregiver] “Not everything sticks. But a couple of things stick and that’s good” [provider] Theme 2. Quality of life Subtheme 2a. Positive family interactions. Respondents in all groups discussed a general downstream outcome of more positive reciprocal interactions between caregivers and children and strengthening of the caregiver-child bond (“just understanding the impact of responding to him any time that he makes a bid of 51 attention towards us has really changed our relationship, and in really powerful ways” [caregiver]; “I definitely see more positive relationships” [provider]). Some respondents in the sample linked this to caregivers’ “knowing how to communicate with their child […and] see the way they view the world a little bit” [provider], which decreases frustration in day-to-day interactions and routines and increases the number of positive interactions: Just to kind of learn what he’s thinking, what he prefers, what he needs – and for him, and for us, I just became less frustrating. […] otherwise, he would just scream, he would just scream and meltdown, and we wouldn’t understand. And just being able to put two words together, “go outside”, “play ball”, you know, it’s just – everything became easier for everyone here at home, at school. [caregiver] Others attributed this to caregivers’ “ability to feel like they can meet their kids’ needs” [expert] which “changes their relationship with their child” [provider] and makes caregivers “more likely to interact with their child more [… and] more likely to have positive reciprocal interactions” [expert]. Subtheme 2b. Positive outcomes for caregivers. Indeed, effects on caregiver confidence, parenting self-efficacy, empowerment, and stress reduction were often described with regard to long-term outcomes on caregivers: “The fact they know what to do or feel like, ‘I know how to impart change in my child’s skills’ is a really powerful experience for families that extends past the structured intervention component” [expert]. When caregivers “learn a new skill […] it’s empowering” [provider]; it gives them an “experience of success” [expert] and “the confidence to teach [their] child” [provider]. One caregiver said: “I didn’t feel lost anymore and that kind of anxiety as a parent really went down after starting the program and it has really helped me remain hopeful and not lost.” 52 Theme 3. Advocacy. Respondents described that some caregivers become “super advocates for their children” [expert] after receiving parent training and are “wanting to do other trainings” [provider] to continue to develop skills to support their children. Several respondents talked about caregivers’ role in teaching others to work with the child effectively more long term, including with professionals (“you can tell them ‘That’s not going to work, and maybe you should try it this way’” [caregiver]; “I have some families that (laughs) are trying to unteach their ABA technicians how to prompt differently” [provider]), as well as other family members and caregivers (“the next thing they wanna do is expand it to have the siblings understand how to do things, the other caregivers, and childcare providers”). Theme 4. Goals and expectations. Experts described caregivers’ ability to “continuously set goals for their child” as a key long-term goal of the program, given that “important goals [...] will change over time.” Some caregivers described their own development in this area during the program: “it really helped us adjust our expectations and teach us as parents, how we can help him the most, and if we’re going to push him, what should we push him towards?” However, some providers acknowledged that continuing to develop goals is difficult for some families: “parents may or may not know when to push that next step, and so I think they do need a little bit of guidance going forward.” Yet, providers also noted that caregivers’ expectations about their children’s ability to do things increases during the program. As a result, caregivers are more apt to “provide that wait time and have the confidence to wait it out” or try new experiences out in the community they may not have otherwise. Integrated model of parent coaching An integrated theory of change model of parent coaching processes in Project ImPACT can be found in Figure 4.1. We identified two key domains in the change process which we 53 called learning processes and motivational processes. Together, these two processes contribute to caregiver fidelity of implementation and use of treatment techniques outside of sessions. These two short term outcomes are then thought to support the child’s development of social communication skills (mid-term outcomes). Of note, respondents consistently described having caregivers practice and receive feedback with the support of a coach as particularly important for supporting caregivers in learning and implementing Project ImPACT, consistent with broader meta-analytic findings on PMI (Wyatt Kaminski et al., 2008). Learning processes. Consistent with models of adult learning, our theory of change suggests that a combination of introducing content, modeling, practice with feedback, and reflection supports caregivers in acquiring conceptual and procedural knowledge about the intervention (Dunst et al., 2010). A positive experience with learning and subsequent skill development is also associated with positive attitudes toward learning, which feed into motivational processes described below (Dunst et al., 2010). Motivational processes. In our model, motivational processes support caregiver self- efficacy, confidence, and empowerment with regard to supporting their child’s development. Motivation has long been considered important for learning; as such, motivation features prominently in several contemporary learning theories (Cook & Artino, 2016). In our model, we identified a transactional ‘feedback loop’ whereby caregivers implement an intervention technique and observe an associated effect on the child’s behavior, which serves to support further implementation and sense of self-efficacy. Through the lens of attribution theory, caregivers experiencing this process make attributions about their role and agency in the interaction (e.g., “I can have an effect on my child’s learning”), which may lead to different psychological outcomes depending on the learner’s emotional response, and the locus, stability, 54 and controllability of the attribution (Cook & Artino, 2016; Weiner, 1985). According to social- cognitive theory, sense of self-efficacy is central to motivated action, such that self-efficacy with regard to using the intervention techniques is essential for ongoing sustainment of the practice (Bandura, 1977; Zimmerman, 2000). 55 Figure 4.1. Integrated theory of change model of parent coaching processes. 56 Phase 3 Statistical approach We identified two mediation models to examine in phase 3 pragmatically based on the availability of archival data for analysis. Leveraging the experimental design of the two studies from which we drew data, the predictor variable was group assignment for both models, comparing the self-directed group (coded 0) with the therapist-assisted group (coded 1). The resource support group was excluded from this analysis. Thus, we focused on the putative active ingredient practice and feedback at the coach-parent level, as presence of this intervention element was the primary difference between the two groups. Mediation models were developed from data visualized in the joint display illustrating the change process for practice and feedback (Table 4.4). Analyses were conducted in Mplus version 8.6 (Muthén & Muthén, 2021) using full information maximum likelihood estimation and percentile bootstrap with 1,000 resamples. In phase 1 of the study, respondents described a process whereby practice and feedback supported parents in implementing the intervention with high fidelity, which was in turn supportive of child social communication growth. Model 1 examined whether access to a coach who provided an opportunity for live practice with feedback would support growth in child social communication skills via increases in caregiver fidelity to Project ImPACT. We expected that group assignment (i.e., access to coaching) would be associated with Time 3 child social communication skills, existing as an indirect effect through Time 2 caregiver fidelity to Project ImPACT (covarying for baseline caregiver fidelity and child social communication skill); See Figure 4.2. In phase 1 of the study, respondents noted that practicing and receiving feedback from a coach helped caregivers attune to the child’s communication growth, which increased parents’ 57 self-efficacy. To examine this further, Model 2 looked at whether access to coaching (i.e., practice with feedback) supported increased parenting self-efficacy via increased caregiver attunement to child skills/communication. As with Model 1 above, Model 2 focused on practice and feedback at the coach-parent level, with the predictor variable being group assignment (self- directed vs. therapist-assisted). We expected that group assignment (i.e. access to coaching) would be associated with Time 3 parenting self-efficacy, existing as an indirect effect through Time 2 caregiver-reported child social communication (covarying for baseline caregiver-reported child social communication); See Figure 4.3. We considered our caregiver-reported child social communication measure (described below) to be a proxy for caregiver attunement to the child's social communication skills, given that it requires caregivers to notice and report on the frequency of specific social communication behaviors. Measures Caregiver fidelity of implementation was measured using the Project ImPACT Fidelity tool. Caregivers are rated from 1-5 on a series of indicators reflecting their implementation of different components of the Project ImPACT intervention, with 5 being excellent implementation. Scores across the five indicators (Focus on your child, Adjust communication, Create opportunities, Teach new skills, and Shape the interaction) are averaged to form an overall fidelity score. Fidelity was coded from 10-minute observations of caregiver-child interactions collected at pre-intervention and post-intervention in two contexts: free play, and a snack routine. Ratings were averaged across the two contexts to form an overall fidelity score. Parenting self-efficacy was measured using the Parenting Sense of Competence Scale (PSOC; Gibaud-Wallston & Wandersman, 1978). The PSOC consists of 17 items rated on a six- point Likert scale with higher sum total scores representing higher sense of self-efficacy. 58 Child social communication was measured using the Social Communication subscale of the Brief Observation of Social Communication Change (BOSCC; Grzadzinski et al., 2016). The BOSCC is a 15-item observational rating system. The BOSCC was coded from 10-minute observations of caregiver-child interactions collected at pre-intervention and follow-up in two contexts: free play, and a snack routine (Frost et al., 2019). Ratings were averaged across the two contexts to form an overall social communication score. Previous analyses of the BOSCC using this data revealed adequate psychometric properties and good interrater reliability (Frost et al., 2019). It should be noted that a higher score on the BOSCC is indicative of social communication impairment; for ease of interpretation, we have changed the sign of our coefficients in the statistical models reported below so that path models can be read intuitively. Caregiver-reported social communication was measured using the Social Communication Checklist (SCC; Wainer et al., 2017). The SCC is a 44-item caregiver-reported checklist of developmental skills in the areas of social engagement, communication, imitation, and play. As such, it reflects caregivers’ understanding of their child’s current behavior. Caregivers rate each item on a 3 point scale: “Rarely/Not yet,” scored 1, “Sometimes, but not consistently,” scored 2, or “Usually, at least 75% of the time,” scored 3. Scores are summed to obtain a total score. Previous research suggests that this measure has adequate internal consistency, test-retest reliability, and convergent validity with other measures of social communication (Wainer et al., 2017). Model 1 A path diagram with parameter estimates (standardized with respect to the outcome variable) can be found in Figure 4.2. Treatment group had a significant effect on caregiver fidelity at Time 2, such that caregivers who received coaching had significantly higher fidelity of 59 implementation to Project ImPACT. However, the direct effect of treatment group on child social communication (95% CI: [-.362, .422]) as well as the indirect effect through caregiver fidelity (95% CI: [-.353, .026]) were not significant. The model provided the following fit to the data: χ2 (4) = 1.22, p = 0.87, CFI = 1.0, TLI = 1.0, RMSEA = 0.00. Unstandardized parameter estimates can be found in Table 4.8. Bivariate correlations can be found in Appendix B. Figure 4.2. Model 1 path diagram and parameter estimates. *p < 0.05. Model 2 A path diagram with parameter estimates (standardized with respect to the outcome variable) can be found in Figure 4.3. The direct effect of treatment group on parenting self- efficacy was not significant (95% CI: [-3.725, 6.893]), nor was the indirect effect through parent- reported child social communication (95% CI: [-0.370 , 1.759]). The model provided the following fit to the data: χ2 (4) = 3.04, p = 0.55, CFI = 1.0, TLI = 1.0, RMSEA = 0.00. Unstandardized parameter estimates can be found in Table 4.8. Bivariate correlations can be found in Appendix B. 60 Figure 4.3. Model 2 path diagram and parameter estimates. *p < 0.05. Table 4.8. Unstandardized parameter estimates for coach-parent level mediation models. Model 1 Path estimates Estimate SE p-value Group → Social comm (T3) -0.35 1.50 0.81 Fidelity (T2) → Social comm (T3) 1.48 0.83 0.08 Social comm (T1) → Social comm (T3) 0.85 0.10 0.00 Fidelity (T1) → Fidelity (T2) 0.42 0.14 0.00 Group → Fidelity (T2) 0.78 0.19 0.00 Factor covariances Estimate SE p-value Social comm (T1) with Fidelity (T1) 1.316 0.542 0.015 Model 2 Path estimates Estimate SE p-value Group → Parenting self-efficacy (T3) 1.76 2.63 0.50 Social comm checklist (T2) → Parenting self-efficacy (T3) 0.11 0.10 0.24 Parenting self-efficacy (T1) → Parenting self-efficacy (T3) 0.57 0.13 <.001 Group → Social comm checklist (T2) 4.38 2.48 0.08 Social comm checklist (T1) → Social comm checklist (T2) 0.83 0.07 <.001 Factor covariances Estimate SE p-value Social comm checklist (T1) with Parenting self-efficacy (T1) -91.69 30.57 <.001 Merged Integration Our hypotheses for Models 1 and 2 were not supported by these data, as indicated by non-significant direct and indirect effects of our putative predictors on outcomes. Although 61 access to coaching was associated with time 2 parent fidelity of implementation with a large effect, fidelity did not predict child social communication at time 3 (although the association was in the expected direction). It is possible that we were underpowered to detect a small effect, or that the elapsed time was insufficient to capture treatment effects. Although we did not find an indirect effect of group on child social communication via parent fidelity of implementation, Yoder and colleagues recently found an indirect effect of group (Project ImPACT vs. control) on child communication via parent fidelity then child motor imitation using a sequential mediation model (Yoder et al., 2021b). Thus, it is possible our model omits a key variable. In addition, although our primary effect of interest for Model 2 was not significant, parent-reported social communication and parenting self-efficacy were significantly correlated at time 1, such that parents who reported more advanced social communication skills reported lower self-efficacy – a concurrent association opposite the expected direction. It is possible that a variable not included in our model (e.g., chronological age) may explain this unexpected relationship. In a qualitative analysis of parent reflection statements, researchers noted that, for parents with high and low self-efficacy alike, children’s skills and response to Project ImPACT intervention techniques affected parents’ intervention implementation (Russell & Ingersoll, 2020). Although our respondents described a process through which children’s positive response to the intervention supported parent self-efficacy, which in turn supported parent implementation, it is possible that self-efficacy does not play a role in this process. Aspects of our integrated model of parent coaching can be situated within the literature on Project ImPACT and parent coaching more broadly. Consistent with our respondents’ report, practicing and receiving feedback from a coach has been shown to improve parent learning of intervention techniques (Ingersoll et al., 2016) even in the absence of didactic instruction 62 (Shanley & Niec, 2010). In studies of parent-child interaction therapy, responsive coaching statements (responding to parents’ use of skills) but not directive coaching statements (telling the parent what to do) have been associated with parents’ use of positive parenting behaviors (Barnett et al., 2014; Heymann et al., 2022). Given that Project ImPACT coaching practices utilize both, this may be an interesting area for further study. Overall, our quantitative results were not supportive of findings from phases 1 and 2 of this study. However, given the significant limitations of our models, stemming largely from our use of secondary data, we believe that aspects of the model are still worth investigating prospectively in the future. 63 CHAPTER 5: Results; Parent-child level intervention processes Phases 1 and 2 Active ingredients, mechanisms of change, and outcomes Respondents described the child-focused intervention process for Project ImPACT by describing the change process for the following intervention components: Follow the child’s lead, Imitate the child, Use animation, Model and expand communication, Playful obstruction, Balanced turns, Communicative temptations, and Teach new skills (see Appendix A for brief descriptions of each intervention component). Consistent with our broader theory of change model of Project ImPACT, implementation of these active ingredients is also considered a “short-term outcome” of the coach-parent level of the intervention. We thus refer to resulting child outcomes as mid-term (more immediate) and downstream (more distal or generalized) outcomes, where appropriate. The results of the mixed methods analysis using joint displays to develop a causal model for each of these intervention elements can be found in Tables 5.1-5.8. Following the joint displays are the results of qualitative analysis pertaining to ‘Shape the interaction,’ the last content-based lesson in the Project ImPACT manual. 64 Table 5.1. Joint display illustrating the quantitative model of the change process for “Follow the child's lead,” presented alongside supporting qualitative data. Causal model Role in theory of change Codes Active Mid-term Downstream Mechanism ingredient outcome outcome Follow the child's lead ◼ Child motivation ◼ Child agency & validation ◼ Novelty ◼ Social attention and engagement ◼ ◼ Parent-child relationship and interaction quality ◼ Skill growth ◼ Supporting Quotations Caregiver I think it was like he’s intrigued. His interest level grew in the situation, in me, and then he was like “oh this is like this is totally fine what I'm doing.” Provider Their engagement lasts longer, a lot less frustration. They like it [laughs] they like to have that little bit of control. […] less elopement, less problem behaviors[…]. We’re taking that power struggle out of, you know, […] they are choosing what they want to play with, so we know that they like what they're doing. Expert Well I think it validates, I mean a child feels “Wow they’re talking my language.” We are not telling them what to do, we are getting to be part of their little world. Narrative change process Following the child's lead increases child social attention and engagement within an interaction in a few ways. At first the novelty might get their attention, by giving children agency and validating their interests increases their motivation and engagement with an adult. This engaged interaction sets the stage for supporting a positive interaction and relationship, as well as the development of new social communication skills over time. 65 Table 5.2. Joint display illustrating the quantitative model of the change process for “Imitate the child,” presented alongside supporting qualitative data. Causal model Role in theory of change Codes Active Mid-term Downstream Mechanism ingredient outcome outcome Imitate the child ◼ Child motivation ◼ ◼ Child agency & validation ◼ Novelty ◼ Social attention and engagement ◼ ◼ Parent-child relationship and interaction quality ◼ Skill growth (imitation and social engagement) ◼ Supporting Quotations Caregiver That was like a huge motivator for getting him to notice us in the interaction was imitation. When we would imitate him, then it was like suddenly, he was like “oh she's doing what I'm doing.” Provider I personally think because it’s novel. […] I think the novelty brings it about initially and then I think that, seeing the parent do what the child is doing gets buy-in from the child. Expert It just shows the child that they have power in this world, right? That their actions matter, that the things that they do have consequence, so it’s, there's a cause-and-effect piece that they’re uncovering and learning. Narrative change process Imitating the child initially supports child social attention and engagement through novelty. Imitation also provides a sense of agency or validation for children which is highly motivating, and supports their continued attention to and engagement with the adult. Over time, this engaged interaction supports a positive interaction and relationship, as well as the development of new social communication skills, particularly in imitation and social engagement. 66 Table 5.3. Joint display illustrating the quantitative model of the change process for “Animation,” presented alongside supporting qualitative data. Causal model Role in theory of change Codes Active Mid-term Downstream Mechanism ingredient outcome outcome Animation ◼ Novelty ◼ Emotion regulation ◼ Adult playfulness ◼ Social attention and engagement ◼ ◼ Parent-child relationship and interaction quality ◼ Skill growth (receptive language) ◼ ◼ Supporting Quotations Caregiver [he] was laughing, was very excited and running around, but was very connected with eye contact and talking […]. We connected for a longer time than he normally would be so that was good. I think he was surprised because he’s not used to seeing a lot of that. Provider I think that’s also important to recognize that kids have different sensory needs. Some children really do need to be revved up (…) and with other kids you need to be a little bit more chill. Expert Many children enjoy it. I think that’s often where we get more affect from kids, more kind of the usually positive affect like the smiling and you get their attention by using like, sound effects and those kinds of things. Narrative change process Animation, when adjusted to children's sensory needs, supports social attention and engagement by creating a playful, novel, and fun interaction. Increased social attention and engagement supports a positive interaction quality and also helps children attend to linguistic input which can support receptive language learning. 67 Table 5.4. Joint display illustrating the quantitative model of the change process for “Model and expand communication,” presented alongside supporting qualitative data. Causal model Role in theory of change Codes Active Mid-term Downstream Mechanism ingredient outcome outcome Model and expand communication ◼ Repetition/associative learning ◼ Skill growth (receptive language) ◼ ◼ ◼ Simplified speech ◼ Imitation ◼ ◼ Social engagement ◼ ◼ Skill growth (expressive language) ◼ ◼ Parent-child relationship and interaction quality ◼ Supporting Quotations Caregiver It just was like less invasive. It wasn’t requiring- it wasn’t putting as much pressure on him almost, to not include the questions, if that makes sense. He was actually more engaged when we were expecting less of him, I think. Provider The more opportunities that the child has to hear that vocabulary, the more opportunities they have to use that vocabulary. Expert We have kiddos […] start imitating what their parents are saying because it’s a simple language and their kiddo starts using some words or pairing words together Narrative change process Modeling and expanding communication supports expressive communication growth through two mechanistic pathways. Repetitive exposure to words leads to receptive communication growth via associative learning, while simplified speech increases child vocal imitation. Imitation and increased receptive vocabulary are both then thought to support expressive communication growth over time. Decreased question-asking was also described as supporting social engagement in interactions, which then impacts parent-child relationship quality over time. 68 Table 5.5. Joint display illustrating the quantitative model of the change process for “Playful obstruction,” presented alongside supporting qualitative data. Causal model Role in theory of change Codes Active Mid-term Downstream Mechanism ingredient outcome outcome Playful obstruction ◼ Motivation ◼ Social attention and engagement ◼ Initiate communication ◼ ◼ Contingent reinforcement ◼ Skill growth (initiating communication) ◼ (Un)clear expectations ◼ Confusion, frustration, disengagement ◼ ◼ Lack of sustainment ◼ Supporting Quotations Caregiver Playful obstruction was not his thing. Um he, I think he just was confused by it maybe? Caregiver When she's on the swing she has a lot of fun she thinks it's funny um and she got really good about saying go. Provider But um you know as like, as we stop the swing, it’s like okay, this is different, let me look to see what’s different, or let me kind of figure out what’s happening. And then it kind of starts that engagement piece. Expert They you know initiate to you know tell you to move or to you know give you a quick glance or to vocalize or to point or to laugh um you know whatever it might be. Narrative change process Playful obstruction can gain a child's attention. When motivated, children often initiate, giving the adult an opportunity to contingently reinforce their communication. Over time, children initiate more frequently. For some children, however, unclear expectations in the interaction can lead to frustration or confusion and disengagement from the interaction. When not perceived as effective, parents may stop using this technique. 69 Table 5.6. Joint display illustrating the quantitative model of the change process for “Balanced turns,” presented alongside supporting qualitative data. Causal model Role in theory of change Codes Active Mid-term Downstream Mechanism ingredient outcome outcome Balanced turns ◼ Consistency/repetition & reinforcement ◼ Clear expectations ◼ ◼ Emotion regulation & motivation ◼ ◼ Initiate communication ◼ Social attention and engagement ◼ Skill growth (turn taking) ◼ (Un)clear expectations ◼ Confusion, frustration, disengagement ◼ ◼ Lack of sustainment ◼ Supporting Quotations Caregiver Initially he would occupy himself with something else because he didn’t realize that it was a loop interaction, that you were going back and forth. That kind of took some time. But I would say only like a couple of weeks and then he understood that, you know, he was going to get a turn back. Provider Once they you know learn that “OK, this is gonna happen, I'm gonna have to take a turn or they’re gonna take a turn,” it really helps increase the back-and-forth interaction between a parent and child. Expert I mean sometimes they’ll yell at you or get mad, (laughs) sometimes they’ll initiate, right? Which is what we're looking for, we want that initiation of "I want that back." Narrative change process At first, many children do not understand the back-and-forth nature of balanced turns and may become upset or disengaged. Eventually, adult expectations become clear through repeated exposure, and children become motivated to initiate communication or continue to engage in a back-and-forth interaction and begin to develop skills in turn-taking. Parents may stop using this technique with children if children become upset/frustrated in response. 70 Table 5.7. Joint display illustrating the quantitative model of the change process for “Communicative temptations,” presented alongside supporting qualitative data. Causal model Role in theory of change Codes Active Mid-term Downstream Mechanism ingredient outcome outcome Communicative temptations ◼ Motivation ◼ Initiate communication ◼ ◼ Contingent reinforcement ◼ Skill growth (initiating communication) ◼ Supporting Quotations Caregiver He needed help, he can’t do it by himself, he’s very motivated by it. So, it kinda sparked that initiation you know, to communicate. Provider Even if they’re not pointing, you see them pulling the parent and reaching to what they want - and they know they’re going to get it they just have to use a certain behavior, you know whatever it may be. Expert [Over time] I think it makes it more likely kids will go on to use their language to communicate what they want in advance, so they might be more likely to ask for blocks or toys or snacks before their parents even set up that to create more spontaneous initiations. Narrative change process Communicative temptations creates a motivating situation for the child to initiate communication. Over time, these initiations are reinforced, and the child is more likely to initiate spontaneously. 71 Table 5.8. Joint display illustrating the quantitative model of the change process for “Teach new skills,” presented alongside supporting qualitative data. Causal model Role in theory of change Codes Mid-term Downstream Active ingredient Mechanism outcome outcome Prompts ◼ Consistency/repetition ◼ Scaffolding ◼ Clear expectations ◼ ◼ Skill growth (supported) ◼ Rewards ◼ Motivation ◼ Contingent reinforcement ◼ Skill growth (spontaneous) ◼ Supporting Quotations Caregiver Knowing that something was expected and then also the strategy of always following through even if it’s like a reduced version of what you asked for, I think it made it really clear that he was expected to do something. Provider the kid knows that you can’t get what you want until you complete this prompt (…). So I think with the parent following through as consistently as possible the kid then knows that ‘this is what mommy or daddy is looking for and I had- this is what I need to do to get what I want.’ Expert They’re reinforced for it and, theoretically, are more likely to use it again. And the more they use it, theoretically, the more likely they are to be able to use it with limited support. ‘Cause the expectation is there. Narrative change process Prompts provide consistent, scaffolded cues that clarify adult expectations and help children demonstrate a new skill. Knowing that a reward is coming motivates children to perform a skill, and over time, contingent reinforcement supports increased spontaneous use of those skills. 72 Shape the interaction “Shape the interaction,” the final lesson of Project ImPACT, emphasizes how to skillfully use and balance various individual intervention techniques with each other. Across interviews, respondents talked about several ways ‘shaping the interaction’ is key to understanding intervention process. Figure 5.1. Adaptation of the Project ImPACT ‘pyramid’ visual showing how earlier-taught techniques support implementation of later-taught techniques. Theme 1. Interactive effects among ImPACT techniques. Respondents talked about several ways that ImPACT techniques interact and facilitate each other. Across interviews, participants described how the techniques work better in combination, such that their effects are difficult to disentangle: “So I think, you know, for them to be effective, it is using all of them, being able to follow the sequence of them together” [expert]. In particular, respondents described how later-taught techniques rely on earlier-taught, foundational techniques to be most effective. In this way, responses aligned with the hierarchical organization of intervention techniques into a 73 pyramid with three levels as described in the Project ImPACT manual (Figure 5.1). Themes fell into two main categories. Theme 1a. ‘Bottom of the pyramid’ lays the foundation for later techniques. Providers and experts both described the importance of ‘bottom of the pyramid’ techniques (particularly following the child’s lead, imitating the child, and using animation) in facilitating “that chance to use other strategies more effectively” [provider]: “You can’t teach a new skill or create an opportunity if the child is not engaged with you and doesn’t want to be engaged with you” [provider]. For example, one caregiver described how using animation was essential in delivering balanced turns: “initially, […] I would make very short turns and then use a lot of animation to make them fun for him too, to keep him sustained.” Similarly, an expert described how following the child’s lead is essential for providing natural reinforcement when teaching new skills. Theme 1b. ‘Middle of the pyramid’ supports effectively teaching new skills. Respondents also described how the ‘middle of the pyramid’ techniques (balanced turns, playful obstruction, and communicative temptations) facilitated natural opportunities to prompt for more complex skills by capitalizing on children’s motivation: “it’s a really good technique to teach a child new skills, ‘cause you already have that initiation and the child already shows you that he wants something. So in that moment, they will be more, they will more likely follow your prompts” [expert]. Theme 2. Challenges with shaping the interaction. Although there was consensus that Project ImPACT techniques build on each other such that they are most effective when used in synchrony, several experts and providers talked about how difficult it is for caregivers to learn so many techniques and balance them appropriately: “there’s a lot of information in Project 74 ImPACT and even though everything builds on each other, […] I think sometimes parents get caught up in the specific strategy that you’re focusing on for that week” [provider]. Indeed, perhaps because caregivers learn to teach new skills at the end of the program, respondents described how caregivers can over-rely on teaching opportunities while not maintaining other techniques: “What I see parents gravitate to, however, are the prompts and rewards because you kind of get the most bang for your buck, right? And I find sometimes it’s difficult for them to come back down the pyramid because they want to stay at this prompting-all-the-time level” [expert]. Theme 3. Shaping the interaction varies across situations. Respondents described that how caregivers implement and balance the different ImPACT techniques varies across families and situations. Although a few providers described it as a passive process in which “certain concepts become more habitual” [provider] while others do not, most respondents described it as a deliberate and intentional process. For example, one provider described it as a deliberate process tailored to the child’s day-to-day needs: “I explain it to the parents [as a] tool kit. So, it’s like, some days you need a hammer, and some days you need a wrench, and you might not need those pliers very often, but when you need them it’s really handy.” Caregivers resonated with this idea as well, emphasizing that they tailored their use of ImPACT techniques based on their child’s emotion regulation: Sometimes when he was having a hard time regulating himself for the week, we would then change our strategy and go down the pyramid to sort of help him regulate himself and not so much focus on skill building and I think that was probably one of the best skills that I learned from the program. 75 Another caregiver adjusted their use of techniques to prioritize those in which their child was most successful: “it's all the things that went well and get back to those.” Theme 4. Child response variability and subsequent tailoring. Although there were many common themes and consistency with which respondents described intervention process, the fact that different children respond in different ways such that certain techniques work better for certain children was a common thread across responses. This sentiment was particularly prominent when discussion Balanced Turns and Playful Obstruction: “Some of my clients don’t like playful obstruction, they get really annoyed and they’re just they hate it. Where some of my clients don’t like balanced turns” [provider]. Several respondents explained this is because “sometimes playful obstruction or balance turns can be really difficult to implement properly” [expert] and others thought it might have to do with child characteristics such as age, developmental level, or temperament. As such, one provider emphasized: “we talk about all of them and coach through all of them because again, every child is going to respond differently to those.” Using animation was another frequently cited area of difference across children: Many children enjoy it. […] But I think you know we all know those kids too, that big loud animation is like – that’s too much for them, […] And so, I think that it's not really like getting rid of animation as much as it is just adjusting how you are animated to meet like the regulatory needs of the kid. Importantly, the theme of response variability not limited to these techniques and was in fact mentioned for all techniques in the manual: “That there’s not one technique that works better for all children, but that there are some techniques that work better for this child and other techniques that work better for another child” [expert]. 76 Downstream outcomes Several downstream outcomes for children were described over the course of the study interviews. First, across intervention techniques, respondents alluded to skill growth in various domains of social communication which unfold over time (visualized in joint displays 5.1-5.8). In addition, several downstream outcomes centered on the theme of child quality of life, including three subthemes: social connections, confidence and independence, and inclusion. Theme 1. Child quality of life. Subtheme 1a. Social connections. Respondents discussed how Project ImPACT supports children in developing positive relationships with important people in their life. This included developing a more positive relationship with caregivers learning Project ImPACT, which supports “more shared enjoyment with their parent” and “more opportunities to […] connect” [provider]. Caregivers also talked about facilitating positive relationships with others by sharing Project ImPACT techniques: “it's helping me to teach others how to interact with her, like I've spent time talking [to] like my parents […] they can at least do that to build a connection with their granddaughter.” Subtheme 1b. Confidence and independence. Caregivers noted changes to their child’s confidence and independence as they moved through Project ImPACT. One caregiver described it as, “giving her independence, and getting her to understand that she knows how to, that she can move her body and she can do things independently,” and another said, “I think it’s affected him, especially in his confidence and through the skill building and understanding that we’re noticing him and he’s not like on his own.” Subtheme 1c. Inclusion. Caregivers and providers also described downstream outcomes in terms of children’s ability to be involved in activities in school and in the community. For 77 example, one caregiver described that, before Project ImPACT, their child would not participate in circle time at daycare, whereas after the program, “now he sits in the circle, he participates, and I've seen the positive impact.” Other caregivers whose children were not yet in preschool were hopeful about how the skills learned through ImPACT would support their child in engaging in educational settings in the future. Integrated model of child change processes An integrated theory of change model of processes at the parent-child level of Project ImPACT can be found in Figure 5.2. The model is consistent with key theoretical foundations of NDBIs, including the related concepts of “pivotal skills” and “developmental cascades,” and the central role of adult responsiveness to children’s communication (described below). In the context of our qualitative approach, although we were able to elaborate on change processes for each intervention component, we were not able to ascertain which strategies seemed to be the most important in effecting change in children’s development. In fact, experts and providers often described how strategies work differently for different children and families, depending on child characteristics and caregivers’ typical interaction style. Although our integrated model represents the modal response as described by our participants, considering baseline child and family characteristics as potential moderators of treatment response is essential. Pivotal skills and the developmental cascade. Our results demonstrate that Project ImPACT targets a variety of interrelated developmental outcomes which naturally occur together during engaged social interactions. Consistent with the goal setting process laid out in the intervention manual, our analysis identified key social communication outcomes including initiating social interactions, expressive and receptive communication, and social engagement. These social communication outcomes were sequenced, such that child motivation and social 78 engagement (targeted by the first strategies learned in Project ImPACT) support receptive understanding, which in turn supports expressive communication (see Figure 5.2). Together, these early changes were described as contributing to skill development over time as communication was naturally reinforced. As such, child motivation and social engagement can be considered ‘pivotal skills’ which support children in developing other social communication skills (such as language and communication skills) via increased participation and interest in the social environment (Koegel et al., 2001). In other words, these skills support the development of other skills in a downstream, cascading effect which unfolds over time. This is consistent with the idea that development is a dynamic process shaped by several interacting inputs, in which later stages build on previous stages (Griffiths & Tabery, 2013). Adult responsiveness. Our results also speak to the central role of adult responsiveness in increasing children’s participation in reciprocal social interaction and use of spontaneous communication. For example, following the child’s lead and imitating the child are ways that adults can be responsive to children’s interests and thus support their motivation and engagement in an interaction. Similarly, responsiveness to children’s attempts to communicate (i.e., contingent reinforcement), either spontaneously or in response to adult cues, supports children in increasing initiations and subsequent broader social communication growth. These processes can be conceptualized from different theoretical lenses, consistent with the dual theoretical foundations of NDBIs. For example, behavioral theory and principles of operant conditioning were often evoked in participants’ descriptions of how communicative temptations or teach new skills strategies support children’s communication development; attempts to communicate are rewarded by adult responsiveness (i.e., contingent reinforcement), and thus more likely to occur in the future (Skinner, 1953). Results also correspond to previous 79 research which has demonstrated associations between parent verbal responsiveness and child communication from a transactional theoretical perspective, which goes further in describing the bidirectional, mutual influence between communication partners (Edmunds et al., 2019). For example, respondents talked about how engagement in a reciprocal social interaction, supported by ‘bottom of the pyramid’ intervention techniques, facilitates adult implementation of more explicit teaching opportunities. Moreover, respondents talked about how children’s response to the techniques, particularly for playful obstruction and balanced turns, affected adults’ propensity to continue to use these techniques in the future. Taken together, our results emphasize that parent mediated NDBIs are not unidirectional (parent → child) but rather a bi- directional and co-constructed process. 80 Figure 5.2. Integrated model of child change processes in Project ImPACT. 81 Phase 3 Statistical approach We identified two mediation models to examine in phase 3 pragmatically based on the availability of data that mapped onto select paths from joint displays illustrating the change process for individual intervention elements and the integrated theory of change model of child change processes in Project ImPACT. Analyses were conducted in Mplus version 8.6 (Muthén & Muthén, 2021) using full information maximum likelihood estimation and percentile bootstrap with 1,000 resamples. In phase 1 of the study, respondents described that following the child’s lead and imitating the child both increased children’s social attention and engagement in the moment, which lead to increases in communication development over time (Tables 5.1-5.2). Model 3 examined whether caregiver fidelity for the Focus on your Child domain (Follow the child’s lead, Imitate the child) supported generalized child communication skills via increases in context-dependent social engagement and social attention. We expected that caregiver fidelity for the Focus on your Child domain would be associated with Time 3 child communication skills, existing as an indirect effect through Time 2 child social attention/engagement (covarying for baseline child communication and social attention/engagement); see Figure 5.2. In phase 1 of the study, respondents also described how Playful Obstruction and Communicative Temptations increased (and provided reinforcement for) child initiations in the moment, which supported communication skill growth over time. Thus, Model 4 examined whether caregiver fidelity for Creating Opportunities (a combined rating for Playful Obstruction, Communicative Temptations, and Balanced Turns) supported generalized child communication skills via increases child initiations. We expected caregiver fidelity for Creating Opportunities 82 would be associated with Time 3 child communication skills, existing as an indirect effect through Time 2 child initiating communication (covarying for baseline child communication and initiating communication). Measures Caregiver fidelity of implementation was measured using the Project ImPACT Fidelity tool. Caregivers are rated from 1-5 on a series of indicators reflecting their implementation of different components of the Project ImPACT intervention, with 5 being excellent implementation. Here, we used data from two indicators (Focus on your child, Create opportunities). Fidelity was coded from 10-minute observations of caregiver-child interactions collected at pre-intervention and post-intervention in two contexts: free play, and a snack routine. Ratings were averaged across the two contexts to form an overall score for each of the two indicator scores. Child social attention/engagement was measured using a composite score created from a subset of items from the Brief Observation of Social Communication Change (BOSCC; Grzadzinski et al., 2016). The following items were summed: (1) eye contact, (4) vocalizations directed to others, (6) frequency and function of social overtures, (7) frequency and quality of responses, and (8) engagement in play activities/interaction. The BOSCC was coded from 10- minute observations of caregiver-child interactions collected at pre-intervention and follow-up in two contexts: free play, and a snack routine (Frost et al., 2019). Ratings were averaged across the two contexts to form an overall social attention/engagement score. It should be noted that a higher score on the BOSCC is indicative of social communication impairment; for ease of interpretation, we have changed the sign of our coefficients in the statistical models reported below so that path models can be read intuitively. 83 Generalized child communication was measured using the Vineland Adaptive Behavior Scales (Sparrow et al., 2005), a widely used standardized assessment of adaptive functioning. Specifically, we used the Communication domain standard scores to capture communication skills. Child initiations were coded from 10-minute observations of caregiver-child interactions collected at pre-intervention and follow-up in two contexts: free play, and a snack routine. Weighted coding of all child intentional communication was completed using computer software (Yoder et al., 2021b). Each instance of child communication was recorded and weighted by complexity (gestures and contingent vocalizations = 1 point; single words = 2 points; simple phrases = 3 points; complex phrases/sentences = 4 points) and classified as imitated or non- imitated. The total non-imitated weighted score was used here. Scores were averaged across the two contexts to form an overall child initiations score. Model 3 A path diagram with standardized parameter estimates can be found in Figure 5.3. The direct effect of caregiver fidelity for the Focus on your Child domain at time 2 on child communication at time 3 was not significant (95% CI: [-1.482, 2.640]), nor was the indirect effects through child social attention/engagement at time 2 (95% CI: [-.110, .895]). The model provided the following fit to the data: χ2 (2) = 3.01, p = 0.22, CFI = 0.99, TLI = 0.96, RMSEA = 0.074. Unstandardized parameter estimates can be found in Table 5.9. Bivariate correlations can be found in Appendix B. 84 Figure 5.3. Model 3 path diagram and parameter estimates. *p < 0.05. Model 4 A path diagram with standardized parameter estimates can be found in Figure 5.4. The direct effect of caregiver fidelity for the Create Opportunities domain at time 2 on child communication at time 3 was not significant (95% CI: [-1.095, 1.474]), nor was the indirect effect through child initiations at Time 2 (95% CI: [-.031, .766]). The model provided the following fit to the data: χ2 (2) = 2.07, p = 0.36, CFI = 0.99, TLI = 0.99, RMSEA = 0.019. Unstandardized parameter estimates can be found in Table 5.9. Bivariate correlations can be found in Appendix B. Figure 5.4. Model 4 path diagram and parameter estimates. *p < 0.05. 85 Table 5.9. Unstandardized parameter estimates for parent-child level mediation models. Model 1 Path estimates Estimate SE p Focus fidelity (T2) → Communication (T3) 0.38 1.31 0.72 Social attention/engage (T2) → Communication (T3) 3.07 0.92 0.00 Communication (T1) → Communication (T3) 0.59 0.10 0.00 Focus fidelity (T2) → Social attention/engage (T2) 0.11 0.07 0.14 Social attention/engage (T1) → Social attention/engage (T2) 0.66 0.07 <.001 Factor covariances Estimate SE p Social attention/engage (T1) with Focus fidelity (T2) 0.004 0.042 0.967 Social attention/engage (T1) with Communication (T1) 3.393 0.761 <.001 Focus fidelity (T2) with Communication (T1) -0.117 0.87 0.893 Model 2 Path estimates Estimate SE p Create Opportunities fidelity (T2) → Communication (T3) 0.16 0.63 0.80 Child initiations (T2) → Communication (T3) 0.03 0.01 0.01 Communication (T1) → Communication (T3) 0.54 0.11 <.001 Create Opportunities fidelity (T2) → Child initiations (T2) 8.32 5.22 0.11 Child initiations (T1) → Child initiations (T2) 1.16 0.12 <.001 Factor covariances Estimate SE p Child initiations (T1) with Create Opportunities fidelity (T2) -0.68 12.24 0.96 Child initiations (T1) with Communication (T1) 258.15 67.76 <.001 Create Opportunities fidelity (T2) with Communication (T1) 0.22 1.09 0.84 Merged Integration Our hypotheses for Models 3 and 4 were not supported by these data, as indicated by non-significant direct and indirect effects of our putative predictors on outcomes. Select individual pathways, however, were consistent with aspects of our theory of change model. For example, child social attention/engagement and initiations at time 2 was significantly associated with generalized communication skills at time 3, when accounting for child social attention/engagement and initiations at time 1. It is possible that the bi-directional and co-constructed nature of PMI can account for our findings. For example, regarding Model 3, it is possible that some children are generally 86 attentive and socially engaged within caregiver-child interactions even when a caregiver is not using the ‘Focus on your child’ techniques, allowing caregivers to spend more time implementing other techniques without adversely impacting social engagement. Similarly, for Model 4, caregivers whose children are infrequent initiators may benefit from more frequent use of Communicative Temptations in order to elicit child initiations (and provide an opportunity for natural reinforcement). On the other hand, for children who are more frequent initiators, these techniques may not confer a meaningful change in the rate of child initiations. In fact, caregivers of children who initiate frequently may actually use these techniques less often, because there are fewer natural opportunities for them to do so (i.e., times when their child is not initiating). Although following up on these possible moderators was outside of the scope and purpose of the current study, they are an interesting avenue for future research. In addition, because of how fidelity of implementation was measured, we were not able to differentiate between adult use of communicative temptations, playful obstruction, and balanced turns. Our qualitative findings suggest that adults may implement these techniques to different extents based on children’s response to them, therefore future research may consider examining these techniques separately. Previous research provides some evidence in support of the hypothesized relationships. For example, Yoder and colleagues found that the effect of Project ImPACT on child social communication was serially mediated by parent fidelity then child intentional communication (2021b), supporting the hypothesis that child initiations support generalized communication growth. Ingersoll and colleagues used a single case experimental design to compare responsive interaction (consistent with “focus on your child”), milieu teaching (consistent with “create opportunities” and “teach new skills”), and a combined approach (Ingersoll et al., 2012). Milieu 87 teaching and the combined approach both led to increases in expressive communication, suggesting that these techniques in particular may support child initiations. In sum, our quantitative results were not supportive of our findings from phases 1 and 2 of the present study. Our models had several significant limitations due to our reliance on secondary data. For example, due to the nature of the constructs involved, it was not possible to randomize our predictors or mediating variables, and we had a limited number of time points from which to draw our data. However, we believe that aspects of our integrated change model are worth investigating prospectively in the future. 88 CHAPTER 6: Discussion and conclusions This study used an exploratory sequential mixed methods design to explore potential active ingredients and mechanisms of change of Project ImPACT, an empirically supported NDBI for young children with autism or social communication delays. The study aims were to 1) Develop a comprehensive Theory of Change of Project ImPACT using stakeholder perspectives on potential active ingredients and mechanisms of change, and 2) Provide proof-of-concept of the Theory of Change model using archival data from treatment trials of Project ImPACT. Through this process, we developed two integrated models of the change process for Project ImPACT, with the first detailing the caregiver learning process as the caregiver works with a coach, and the second detailing the child learning process as the caregiver implements the intervention techniques. Although these two models were presented separately for the sake of clarity, it is important to consider how they intersect. The integrated model of child change (Figure 5.2) is akin to an inset map which zooms in to provide more detail on the relationship between fidelity of implementation and child skill development depicted on the model of parent coaching process (Figure 4.1). We also identified what needs Project ImPACT fulfills for families, which included a parent-driven and parent-led service, the structured and systematic nature of the intervention, general knowledge and understanding about child development, and a child-centered service. A variety of contextual factors, including child, caregiver, family, and cultural and linguistic factors, were all described by respondents as contributing to goodness-of- fit of the intervention for diverse families. Strengths This work presents with several strengths. We used an established framework, Theory of Change, to conduct an in-depth mixed methods exploration of the change process underlying 89 Project ImPACT. In addition, this research can be situated within the evaluation phase of the Medical Research Council framework for the developmental and evaluation of complex interventions (Skivington et al., 2021). The present study incorporated numerous core elements of complex interventions research according to the MRC framework, including considering context; developing, refining, and (re)testing program theory; engaging stakeholders; and identifying key uncertainties. Indeed, we hope to use the data from this generative process to develop new research ideas and continue to refine the intervention under study. In the future, the theory of change models developed in the present study can be leveraged to design prospective studies with the intent of capturing causal processes. A benefit of our focus on the evaluation phase of the intervention was our ability to engage stakeholders in the intervention of interest, including individuals with a high level of expertise and end-users in the community. In contrast with other recent work for developing program theory prospectively using researcher input (e.g. Edmunds et al., 2022; Kirk et al., 2019), our retrospective approach emphasized practice-based knowledge from community clinicians as well as caregiver perspectives. Engaging end-users of the intervention allowed us to identify change processes which seem important or meaningful to consumers of the intervention and to consider potential change processes which have been overlooked in research to date. In this respect, we found qualitative causation coding to be a fruitful way to generate ideas for future research, as our data pointed to several putative causal pathways and potential moderators of intervention efficacy which have yet to be tested empirically. Thus, by considering whether the Theory of Change aligns with existing theory and evidence from experimental trials, we were able to ‘identify key uncertainties.’ For example, the central role of motivational processes in caregiver learning is not addressed in the intervention manual, nor has it been evaluated as a 90 mechanism of change in subsequent child treatment outcomes. In addition, the role of these motivational processes and child response to the intervention in supporting sustainment is key to understanding mechanisms supporting long-term use of the intervention techniques. Additionally, child motivation and emotion regulation both emerged as important constructs in understanding the child learning process. Operationalizing and measuring these constructs may be difficult but is perhaps needed to capture the child learning process quantitatively. Last, our respondents described several long-term treatment outcomes pertaining to quality of life for the child and family, which to our knowledge have yet to be investigated in a research context. Limitations In terms of our original data collection for Phase 1, although it was a relatively large sample for qualitative analysis, the respondents were not necessarily representative of all consumers of Project ImPACT. For example, because we were interested exploring all components of the intervention, we spoke with caregivers who completed most or all of the Project ImPACT intervention. As such, we did not hear from families who opted to stop participating after a few sessions. In addition, most of the caregivers expressed positive experiences with their coach and a positive perception of the program overall. There are several possible explanations for this. First, although providers were asked to share study materials with all eligible families, it is possible that providers selectively offered the information to families who they perceived to be “successful.” Second, caregivers who were excited about the program may have been more likely to opt into a research study on the topic. Last, it is possible that most families who go through the program do end up having a positive experience overall. In addition, we recruited families who were able to complete an interview in English. Although community providers discussed barriers that arose for families who did not come from dominant cultural and 91 linguistic backgrounds, we did not hear about these barriers first-hand from families experiencing them. Indeed, our qualitative sample was lacking in racial-ethnic diversity and single-parent families. Future research focusing on the intervention experience for families from a variety of marginalized backgrounds (e.g., socioeconomically disadvantaged families, linguistic minorities, immigrant families, minoritized racial/ethnic groups) is essential for understanding the fit of Project ImPACT for diverse families and to avoid reproducing systemic inequities in service access (e.g., Shenouda et al., 2022) in the context of the intervention itself. This study also recruited providers and caregivers whose experience with the intervention was either in-person or using telehealth due to recruitment during the COVID-19 pandemic. It is possible that the treatment process for Project ImPACT differs depending on the modality in which it is delivered, however, emerging evidence suggests that telehealth can be used effectively to deliver the intervention. A randomized pilot study comparing self-directed and therapist-assisted training found that caregivers in both groups improved in their fidelity of implementation, but the therapist-assisted group had greater gains and reached higher fidelity by the end of the study (Ingersoll et al., 2016). In addition, a non-randomized pilot study directly compared in-person delivery with telehealth delivery and found no significant group by time interactions for child social communication measures or caregiver fidelity (Hao et al., 2020). While providing promising preliminary evidence, it is possible that caregivers who opted into telehealth services are not representative of the broader population of potential service users. There were several limitations in the quantitative phase of this study as well. Because we used archival data from studies which were not designed to test specific causal processes outlined in our Theory of Change, we did not have access to quantitative measures that could assess each hypothesized mediator and outcome in an ideal sequence or at the appropriate times 92 for an ideal longitudinal mediation analysis (Little, 2013). Similarly, because the data were not collected with this type of analysis in mind, the analyses were underpowered. Together, these limitations prevented us from examining many of the hypothesized pathways in our Theory of Change model and limits our ability to understand causal relationships. It is also possible that some key variables (e.g., moderators, covariates) were omitted from our models, and that omitted variables biased our results. Although our predictor for two of the models, group assignment, was randomized, this is supportive of causal relationships only for the path from the randomized predictor to the mediator, and not from the mediating variable to the outcome. Our other models included nonrandomized predictors and only two time points, consistent with a “half- longitudinal” mediation design (Little, 2013). Although original data collection to complete the quantitative phase of the study was not feasible within the scope of this project, prospective data collection may be needed in order to capture hypothesized change processes. It is our goal that the theory of change models developed in the present study can be leveraged to design prospective studies with longitudinal mediation analyses in mind. A strength of the design of the original studies was the use of a RCT design, which allowed us to consider the effect of live coaching in comparison to the effect of access to intervention materials alone. However, it is important to note that not all families in the self- direct group accessed all of the online lessons. Thus, although access to live coaching was the defining difference between the two groups, the two groups also likely differed in terms of their exposure to intervention content. Furthermore, exposure to the intervention content alone (or contact with the research team) may have affected motivational and learning processes for caregivers, making effects of coaching more difficult to detect. An in-depth exploration of this was outside the scope of the present study. 93 Directions for future research Parent-mediated NDBIs Historically, parent-mediated NDBIs have been studied much like therapist-delivered NDBIs, without significant consideration for how there might be unique mechanistic processes at play within a parent-mediated approach. Our analyses pointed to several unique features of parent-mediated intervention which warrant more attention in future research. For example, our respondents described a mechanistic role for self-efficacy and motivation in supporting caregiver implementation of the intervention with their child. Although parent coaching manuals discuss these factors and parenting self-efficacy has been considered as a potential outcome of PMI, to our knowledge, this mediating role has not been studied in research. However, emerging research suggests that motivational interviewing may be a helpful supplement to traditional parent- mediated NDBI (Rogers et al., 2019). In a clinical context, formal evaluation of parenting self- efficacy and motivation may help clinicians provide additional support to caregivers who are struggling in this area. In addition, our respondents described how children’s response to the intervention techniques affects whether and how caregivers continue to implement them. This represents a unique departure from therapist implementation of interventions in research contexts, in which fidelity of implementation is carefully controlled. Future research examining whether certain child characteristics predict caregiver use of specific intervention techniques may be useful to identify ways in which interventions might be individualized or tailored to individual families. Furthermore, we were not able to ascertain whether particular intervention techniques were more important for supporting child social communication development than others. While this may be in part due to our qualitative methodology, it is also the case that our respondents 94 often described how there were differences in treatment response due to differences in child characteristics as well as caregiver behavior at baseline. For example, for some caregivers, following the child’s lead in play represents a significant departure from their usual style, while others may naturally engage with children this way. Similarly, although some children find playful obstruction and balanced turns fun or silly, others find it distressing or dysregulating. Future research should consider key moderators of intervention response. Last, our research pointed to a variety of downstream outcomes pertaining to quality of life which were salient to caregivers, community providers, and experts alike. Respondents identified several downstream outcomes pertaining to quality of life for children (e.g., social connections, inclusion, confidence and independence), caregivers (e.g., confidence, empowerment), and for the family unit (e.g., positive family interactions, advocacy). In many cases, the caregiver-child dyad’s newfound ability to have fun, mutually engaging interactions was the most meaningful intervention outcome reported by caregivers. Consistent with calls from neurodiversity advocates to focus more on outcomes relating to quality of life, social support, and wellbeing (Kapp, 2018), we believe this is an important area for increased study. Mixed methods approach This study used an exploratory sequential mixed methods design can be used to develop and test theories of change for complex interventions. Our emphasis on individual intervention components and their respective mechanisms is both unique and well suited to a mixed methods approach. Qualitative causation coding and connected integration via joint displays allowed us to build detailed causal models which can be tested empirically using quantitative data. Indeed, we believe that a mixed methods approach is superior to a monomethod approach in numerous ways. Beginning with a qualitative component allowed us to gain insights from the end-users of 95 the intervention and identify potential mechanisms of change that were not originally part of the theoretical basis for the intervention. Building our model through connected integration enabled the development of causal models summarizing key constructs and their mechanistic relationships. We believe these causal models provide a useful roadmap for the design of future experimental trials capable of identifying active ingredients and mechanism of change of Project ImPACT. Other researchers interested in using similar methods to map out causal processes of a complex intervention may instead elect to design prospective studies to collect data for Phase 3. A key benefit of this approach lies in the ability to collect measures of specific constructs identified in the Theory of Change at time points that are relevant based on when hypothesized change processes are thought to occur. Depending on the complexity of the intervention and causal model, prospective studies may be able to test the Theory of Change in a more comprehensive way than a retrospective analysis. Alternatively, a series of prospective studies might instead be designed to focus on aspects of the model in more targeted way, homing in on key causal pathways of interest for a specific intervention or implementation context. A variety of experimental study designs can assess active ingredients of interventions. For example, component analysis, a type of single case experimental design, is ideal for measuring immediate, context-dependent effects of intervention components with a relatively small sample (Kratochwill et al., 2010). Group designs, such as additive and dismantling RCTs, can also evaluate one or more active ingredients by multiple treatment groups which by a single treatment element (Bell et al., 2013). Multiphase Optimization Strategy (MOST) is a phased approach for systematic study of intervention elements in order to create an optimized intervention (Collins et 96 al., 2005) through designs such as factorial experiments which enable relatively efficient tests of the efficacy of individual intervention elements and their interactions (Collins et al., 2014). Conclusion Understanding active ingredients and causal processes comprising complex interventions is important for treatment optimization. Although parent-mediated NDBIs are increasingly being studied in research contexts, their active ingredients and mechanisms of change are understudied, and their complexity presents a barrier to widespread implementation in the community. We believe that mixed methods approaches that blend qualitative analysis, model building, and visualization of mixed data are uniquely suited to improving our understanding of complex interventions. 97 REFERENCES Aldred, C., Green, J., Emsley, R., & McConachie, H. (2012). Brief report: Mediation of treatment effect in a communication intervention for pre-school children with autism. Journal of Autism and Developmental Disorders, 42(3), 447–454. Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84(2), 191. Barnett, M. L., Niec, L. N., & Acevedo-Polakovich, I. D. (2014). Assessing the key to effective coaching in parent–child interaction therapy: The therapist-parent interaction coding system. Journal of Psychopathology and Behavioral Assessment, 36(2), 211–223. Bearss, K., Burrell, T. L., Stewart, L., & Scahill, L. (2015). Parent training in autism spectrum disorder: What’s in a name? Clinical Child and Family Psychology Review, 18(2), 170– 182. Bell, E. C., Marcus, D. K., & Goodlad, J. K. (2013). Are the parts as good as the whole? A meta- analysis of component treatment studies. Journal of Consulting and Clinical Psychology, 81(4), 722–736. https://doi.org/10.1037/a0033004 Borglin, G. (2015). The value of mixed methods for researching complex interventions. In D. A. Richards & I. R. Hallberg (Eds.), Complex Interventions in Health: An Overview of Research Methods (pp. 29–45). Routledge. Bowen, G. A. (2008). Naturalistic inquiry and the saturation concept: A research note. Qualitative Research, 8(1), 137–152. https://doi.org/10.1177/1468794107085301 Brian, J., Smith, I. M., Zwaigenbaum, L., Roberts, W., & Bryson, S. E. (2016). The Social ABCs caregiver‐mediated intervention for toddlers with autism spectrum disorder: Feasibility, acceptability, and evidence of promise from a multisite study. Autism Research, 9(8), 899–912. Bruinsma, Y., Minjarez, M. B., Schreibman, L., & Stahmer, A. C. (Eds.). (2020). Naturalistic Developmental Behavioral Interventions for Autism Spectrum Disorder. Brookes Publishing. Chambers, D. A., & Norton, W. E. (2016). The Adaptome. American Journal of Preventive Medicine, 51(4), S124–S131. https://doi.org/10.1016/j.amepre.2016.05.011 Chorpita, B. F., Daleiden, E. L., & Weisz, J. R. (2005). Identifying and selecting the common elements of evidence based interventions: A distillation and matching model. Ment Health Serv Res, 7(1), 5–20. Christensen, D. L., Braun, K. V. N., Baio, J., Bilder, D., Charles, J., Constantino, J. N., Daniels, J., Durkin, M. S., Fitzgerald, R. T., Kurzius-Spencer, M., Lee, L.-C., Pettygrove, S., 98 Robinson, C., Schulz, E., Wells, C., Wingate, M. S., Zahorodny, W., & Yeargin-Allsopp, M. (2018). Prevalence and Characteristics of Autism Spectrum Disorder Among Children Aged 8 Years—Autism and Developmental Disabilities Monitoring Network, 11 Sites, United States, 2012. MMWR. Surveillance Summaries, 65(13), 1–23. https://doi.org/10.15585/mmwr.ss6513a1 Clark-Whitney, E., Klein, C. B., Hadley, P. A., Lord, C., & Kim, S. H. (2022). Caregiver Language Input Supports Sentence Diversity in Young Children With Autism Spectrum Disorder. Journal of Speech, Language, and Hearing Research, 65(4), 1465–1477. https://doi.org/10.1044/2021_JSLHR-21-00458 Collins, L. M., Dziak, J. J., Kugler, K. C., & Trail, J. B. (2014). Factorial Experiments: Efficient Tools for Evaluation of Intervention Components. American Journal of Preventive Medicine, 47(4), 498–504. https://doi.org/10.1016/j.amepre.2014.06.021 Collins, L. M., Murphy, S. A., Nair, V. N., & Strecher, V. J. (2005). A strategy for optimizing and evaluating behavioral interventions. Annals of Behavioral Medicine, 30(1), 65–73. https://doi.org/10.1207/s15324796abm3001_8 Cook, D. A., & Artino, A. R. (2016). Motivation to learn: An overview of contemporary theories. Medical Education, 50(10), 997–1014. https://doi.org/10.1111/medu.13074 Craig, P., Dieppe, P., Macintyre, S., Michie, S., Nazareth, I., & Petticrew, M. (2008). Developing and evaluating complex interventions: The new Medical Research Council guidance. BMJ, a1655. https://doi.org/10.1136/bmj.a1655 Curry, L., & Nunez-Smith, M. (2015). Mixed Methods in Health Sciences Research: A Practical Primer. https://doi.org/10.4135/9781483390659 Davidoff, F., Dixon-Woods, M., Leviton, L., & Michie, S. (2015). Demystifying theory and its use in improvement. BMJ Quality & Safety, 24(3), 228–238. https://doi.org/10.1136/bmjqs-2014-003627 De Silva, M. J., Breuer, E., Lee, L., Asher, L., Chowdhary, N., Lund, C., & Patel, V. (2014). Theory of Change: A theory-driven approach to enhance the Medical Research Council’s framework for complex interventions. Trials, 15(1), 267. https://doi.org/10.1186/1745- 6215-15-267 Dimidjian, S., & Hollon, S. D. (2010). How would we know if psychotherapy were harmful? American Psychologist, 65(1), 21–33. https://doi.org/10.1037/a0017299 Dunst, C. J., Trivette, C. M., & Hamby, W. (2010). Meta-analysis of the effectiveness of four adult learning methods and strategies. 3(1), 22. Edmunds, S. R., Frost, K. M., Sheldrick, R. C., Bravo, A., Straiton, D., Pickard, K., Grim, V., Drahota, A., Kuhn, J., Azad, G., Pomales Ramos, A., Ingersoll, B., Wainer, A., Ibanez, L. 99 V., Stone, W. L., Carter, A., & Broder-Fingert, S. (2022). A method for defining the CORE of a psychosocial intervention to guide adaptation in practice: Reciprocal imitation teaching as a case example. Autism, 136236132110644. https://doi.org/10.1177/13623613211064431 Edmunds, S. R., Kover, S. T., & Stone, W. L. (2019). The relation between parent verbal responsiveness and child communication in young children with or at risk for autism spectrum disorder: A systematic review and meta-analysis. Autism Research, 12(0), 715– 731. https://doi.org/10.1002/aur.2100 Estes, A., Vismara, L., Mercado, C., Fitzpatrick, A., Elder, L., Greenson, J., Lord, C., Munson, J., Winter, J., Young, G., Dawson, G., & Rogers, S. (2014). The Impact of Parent- Delivered Intervention on Parents of Very Young Children with Autism. Journal of Autism and Developmental Disorders, 44(2), 353–365. https://doi.org/10.1007/s10803- 013-1874-z Farmer, T., Robinson, K., Elliott, S. J., & Eyles, J. (2006). Developing and Implementing a Triangulation Protocol for Qualitative Health Research. Qualitative Health Research, 16(3), 377–394. https://doi.org/10.1177/1049732305285708 Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving Integration in Mixed Methods Designs-Principles and Practices. Health Services Research, 48(6pt2), 2134–2156. https://doi.org/10.1111/1475-6773.12117 Frost, K. M., Brian, J., Gengoux, G. W., Hardan, A., Rieth, S. R., Stahmer, A., & Ingersoll, B. (2020). Identifying and measuring the common elements of naturalistic developmental behavioral interventions for autism spectrum disorder: Development of the NDBI-Fi. Autism, 24(8), 2285–2297. https://doi.org/10.1177/1362361320944011 Frost, K. M., Koehn, G. N., Russell, K. M., & Ingersoll, B. (2019). Measuring child social communication across contexts: Similarities and differences across play and snack routines: Measuring social communication across contexts. Autism Research. https://doi.org/10.1002/aur.2077 Gale, N. K., Heath, G., Cameron, E., Rashid, S., & Redwood, S. (2013). Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Medical Research Methodology, 13(1), 117. https://doi.org/10.1186/1471-2288-13-117 Garland, A. F., Hawley, K. M., Brookman-Frazee, L., & Hurlburt, M. S. (2008). Identifying common elements of evidence-based psychosocial treatments for children’s disruptive behavior problems. J Am Acad Child Adolesc Psychiatry, 47(5), 505–514. https://doi.org/10.1097/CHI.0b013e31816765c2 Gibaud-Wallston, J., & Wandersman, L. P. (1978). Parenting sense of competence scale. Canadian Journal of Behavioural Science/Revue Canadienne Des Sciences Du Comportement. 100 Griffiths, P. E., & Tabery, J. (2013). Developmental systems theory: What does it explain, and how does it explain it? In Advances in child development and behavior (Vol. 44, pp. 65– 94). Elsevier. Grzadzinski, R., Carr, T., Colombi, C., McGuire, K., Dufek, S., Pickles, A., & Lord, C. (2016). Measuring Changes in Social Communication Behaviors: Preliminary Development of the Brief Observation of Social Communication Change (BOSCC). Journal of Autism and Developmental Disorders, 46(7), 2464–2479. https://doi.org/10.1007/s10803-016- 2782-9 Gulsrud, A. C., Hellemann, G., Shire, S., & Kasari, C. (2016). Isolating active ingredients in a parent-mediated social communication intervention for toddlers with autism spectrum disorder. Journal of Child Psychology and Psychiatry, 57(5), 606–613. https://doi.org/10.1111/jcpp.12481 Hammersley, M. (1992). Some reflections on ethnography and validity. International Journal of Qualitative Studies in Education, 5(3), 195–203. https://doi.org/10.1080/0951839920050301 Hao, Y., Franco, J. H., Sundarrajan, M., & Chen, Y. (2020). A Pilot Study Comparing Tele- therapy and In-Person Therapy: Perspectives from Parent-Mediated Intervention for Children with Autism Spectrum Disorders. Journal of Autism and Developmental Disorders. https://doi.org/10.1007/s10803-020-04439-x Hardan, A. Y., Gengoux, G. W., Berquist, K. L., Libove, R. A., Ardel, C. M., Phillips, J., Frazier, T. W., & Minjarez, M. B. (2015). A randomized controlled trial of Pivotal Response Treatment Group for parents of children with autism. Journal of Child Psychology and Psychiatry, 56(8), 884–892. https://doi.org/10.1111/jcpp.12354 Hemmler, V. L., Kenney, A. W., Langley, S. D., Callahan, C. M., Gubbins, E. J., & Holder, S. (2020). Beyond a coefficient: An interactive process for achieving inter-rater consistency in qualitative coding. Qualitative Research, 146879412097607. https://doi.org/10.1177/1468794120976072 Heymann, P., Heflin, B. H., & Bagner, D. M. (2022). Effect of therapist coaching statements on parenting skills in a brief parenting intervention for infants. Behavior Modification, 46(4), 691–705. Hume, K., Bellini, S., & Pratt, C. (2005). The Usage and Perceived Outcomes of Early Intervention and Early Childhood Programs for Young Children With Autism Spectrum Disorder. Topics in Early Childhood Special Education, 25(4), 195–207. https://doi.org/10.1177/02711214050250040101 Ingersoll, B., & Dvortcsak, A. (2010). Teaching Social Communication to Children With Autism: A Practitioner’s Guide to Parent Training and a Manual for Parents. Guilford Press. 101 Ingersoll, B., & Dvortcsak, A. (2019). Teaching Social Communication to Children with Autism and Other Developmental Delays: The Project ImPACT Guide to Coaching Parents and the Project ImPACT Manual for Parents (2nd ed.). Guilford Press. Ingersoll, B., Meyer, K., Bonter, N., & Jelinek, S. (2012). A Comparison of Developmental Social–Pragmatic and Naturalistic Behavioral Interventions on Language Use and Social Engagement in Children With Autism. Journal of Speech, Language, and Hearing Research, 55(5), 1301–1313. https://doi.org/10.1044/1092-4388(2012/10-0345) Ingersoll, B., & Wainer, A. (2013). Initial Efficacy of Project ImPACT: A Parent-Mediated Social Communication Intervention for Young Children with ASD. Journal of Autism and Developmental Disorders, 43(12), 2943–2952. https://doi.org/10.1007/s10803-013- 1840-9 Ingersoll, B., Wainer, A. L., Berger, N. I., Pickard, K., & Bonter, N. (2016). Comparison of a Self-Directed and Therapist-Assisted Telehealth Parent-Mediated Intervention for Children with ASD: A Pilot RCT. Journal of Autism and Developmental Disorders, 46(7), 2275–2284. https://doi.org/10.1007/s10803-016-2755-z Ingersoll, B., Wainer, A. L., Berger, N. I., & Walton, K. M. (2017). Efficacy of low intensity, therapist-implemented Project ImPACT for increasing social communication skills in young children with ASD. Developmental Neurorehabilitation, 20(8), 502–510. https://doi.org/10.1080/17518423.2016.1278054 Kaehler, L. A., Jacobs, M., & Jones, D. J. (2016). Distilling Common History and Practice Elements to Inform Dissemination: Hanf-Model BPT Programs as an Example. Clin Child Fam Psychol Rev, 19(3), 236–258. https://doi.org/10.1007/s10567-016-0210-5 Kaiser, A. P., Hancock, T. B., & Nietfeld, J. P. (2000). The Effects of Parent-Implemented Enhanced Milieu Teaching on the Social Communication of Children Who Have Autism. Early Education & Development, 11(4), 423–446. https://doi.org/10.1207/s15566935eed1104_4 Kallio, H., Pietilä, A.-M., Johnson, M., & Kangasniemi, M. (2016). Systematic methodological review: Developing a framework for a qualitative semi-structured interview guide. Journal of Advanced Nursing, 72(12), 2954–2965. https://doi.org/10.1111/jan.13031 Kapp, S. K. (2018). Social support, well-being, and quality of life among individuals on the autism spectrum. Pediatrics, 141(Suppl 4), S362-68. Kasari, C., Lawton, K., Shih, W., Barker, T. V., Landa, R., Lord, C., Orlich, F., King, B., Wetherby, A., & Senturk, D. (2014). Caregiver-Mediated Intervention for Low- Resourced Preschoolers With Autism: An RCT. In Pediatrics (24958585; Vol. 134, pp. e72-9). https://doi.org/10.1542/peds.2013-3229 102 Kirk, M. A., Haines, E. R., Rokoske, F. S., Powell, B. J., Weinberger, M., Hanson, L. C., & Birken, S. A. (2019). A case study of a theory-based method for identifying and reporting core functions and forms of evidence-based interventions. Translational Behavioral Medicine, ibz178. https://doi.org/10.1093/tbm/ibz178 Koegel, R. L., Koegel, L. K., & McNerney, E. K. (2001). Pivotal areas in intervention for autism. Journal of Clinical Child & Adolescent Psychology, 30(1), 19–32. https://doi.org/10.1207/S15374424JCCP3001_4 Kratochwill, T. R., Hitchcock, J., Horner, R., Levin, J. R., Odom, S., Rindskopf, D., & Shadish, W. (2010). What Works Clearinghouse: Single-case designs technical documentation. http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf Lee, S. J., Altschul, I., & Mowbray, C. T. (2008). Using Planned Adaptation to Implement Evidence-Based Programs with New Populations. American Journal of Community Psychology, 41(3–4), 290–303. https://doi.org/10.1007/s10464-008-9160-5 Leviton, L. C. (2017). Generalizing about Public Health Interventions: A Mixed-Methods Approach to External Validity. Annual Review of Public Health, 38(1), 371–391. https://doi.org/10.1146/annurev-publhealth-031816-044509 Leviton, L. C., & Trujillo, M. D. (2017). Interaction of Theory and Practice to Assess External Validity. Evaluation Review, 41(5), 436–471. https://doi.org/10.1177/0193841X15625289 Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic Inquiry. Sage Publications. Little, T. D. (2013). Mediation and Moderation. In Longitudinal structural equation modeling (pp. 286–324). Guilford press. Lord, C., Risi, S., Lambrecht, L., Cook, E. H., Leventhal, B. L., DiLavore, P. C., Pickles, A., & Rutter, M. (2000). The Autism Diagnostic Observation Schedule—Generic: A standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders, 30(3), 205–223. Lord, C., Rutter, M., DiLavore, P., Risi, S., Gotham, K., & Bishop, S. (2012). Autism diagnostic observation schedule–2nd edition (ADOS-2). Los Angeles, CA: Western Psychological Corporation. Malterud, K., Siersma, V. D., & Guassora, A. D. (2016). Sample Size in Qualitative Interview Studies: Guided by Information Power. Qualitative Health Research, 26(13), 1753–1760. https://doi.org/10.1177/1049732315617444 Mays, N., & Pope, C. (2000). Qualitative research in health care: Assessing quality in qualitative research. BMJ, 320(7226), 50–52. https://doi.org/10.1136/bmj.320.7226.50 103 Michelson, D., Davenport, C., Dretzke, J., Barlow, J., & Day, C. (2013). Do Evidence-Based Interventions Work When Tested in the “Real World?” A Systematic Review and Meta- analysis of Parent Management Training for the Treatment of Child Disruptive Behavior. Clinical Child and Family Psychology Review, 16(1), 18–34. https://doi.org/10.1007/s10567-013-0128-0 Morris, A. (2015). A Practical Introduction to In-Depth Interviewing. SAGE Publications Ltd. https://doi.org/10.4135/9781473921344 Morse, J. M. (2015). Critical Analysis of Strategies for Determining Rigor in Qualitative Inquiry. Qualitative Health Research, 25(9), 1212–1222. https://doi.org/10.1177/1049732315588501 Muthén, L. K., & Muthén. (2021). Mplus (Version 8). National Research Council. (2001). Educating children with autism. National Academies Press. Nevill, R. E., Lecavalier, L., & Stratis, E. A. (2018). Meta-analysis of parent-mediated interventions for young children with autism spectrum disorder. Autism, 22(2), 84–98. https://doi.org/10.1177/1362361316677838 Nilsen, P., & Bernhardsson, S. (2019). Context matters in implementation science: A scoping review of determinant frameworks that describe contextual determinants for implementation outcomes. BMC Health Services Research, 19(1), 189. https://doi.org/10.1186/s12913-019-4015-3 O’Reilly, M., & Parker, N. (2013). ‘Unsatisfactory Saturation’: A critical exploration of the notion of saturated sample sizes in qualitative research. Qualitative Research, 13(2), 190– 197. https://doi.org/10.1177/1468794112446106 Papa, A., & Follette, W. C. (2015). Dismantling Studies of Psychotherapy. In The Encyclopedia of Clinical Psychology (pp. 1–6). American Cancer Society. https://doi.org/10.1002/9781118625392.wbecp523 Patient-Centered Outcomes Research Institute. (2019). PCORI Methodology Standards. https://www.pcori.org/sites/default/files/PCORI-Methodology-Standards.pdf Pellecchia, M., Mandell, D. S., Beidas, R. S., Dunst, C. J., Tomczuk, L., Newman, J., Zeigler, L., & Stahmer, A. C. (2022). Parent Coaching in Early Intervention for Autism Spectrum Disorder: A Brief Report. Journal of Early Intervention, 105381512210958. https://doi.org/10.1177/10538151221095860 Pickles, A., Harris, V., Green, J., Aldred, C., McConachie, H., Slonims, V., Le Couteur, A., Hudry, K., Charman, T., & the PACT Consortium. (2015). Treatment mechanism in the MRC preschool autism communication trial: Implications for study design and parent- 104 focussed therapy for children. Journal of Child Psychology and Psychiatry, 56(2), 162– 170. https://doi.org/10.1111/jcpp.12291 Pickles, A., Le Couteur, A., Leadbitter, K., Salomone, E., Cole-Fletcher, R., Tobin, H., Gammer, I., Lowry, J., Vamvakas, G., Byford, S., Aldred, C., Slonims, V., McConachie, H., Howlin, P., Parr, J. R., Charman, T., & Green, J. (2016). Parent-mediated social communication therapy for young children with autism (PACT): Long-term follow-up of a randomised controlled trial. The Lancet, 388(10059), 2501–2509. https://doi.org/10.1016/S0140-6736(16)31229-6 Reichow, B., Servili, C., Yasamy, M. T., Barbui, C., & Saxena, S. (2013). Non-Specialist Psychosocial Interventions for Children and Adolescents with Intellectual Disability or Lower-Functioning Autism Spectrum Disorders: A Systematic Review. PLoS Medicine, 10(12), e1001572. https://doi.org/10.1371/journal.pmed.1001572 Rimestad, M. L., Lambek, R., Zacher Christiansen, H., & Hougaard, E. (2019). Short- and Long- Term Effects of Parent Training for Preschool Children With or at Risk of ADHD: A Systematic Review and Meta-Analysis. Journal of Attention Disorders, 23(5), 423–434. https://doi.org/10.1177/1087054716648775 Ritchie, J., & Lewis, J. (Eds.). (2003). Qualitative research practice: A guide for social science students and researchers (Vol. 41). Sage publications. http://choicereviews.org/review/10.5860/CHOICE.41-1319 Roberts, M. Y., Curtis, P. R., Sone, B. J., & Hampton, L. H. (2019). Association of Parent Training With Child Language Development: A Systematic Review and Meta-analysis. JAMA Pediatrics, 173(7), 671–680. https://doi.org/10.1001/jamapediatrics.2019.1197 Rogers, S. J., Estes, A., Vismara, L., Munson, J., Zierhut, C., Greenson, J., Dawson, G., Rocha, M., Sugar, C., Senturk, D., Whelan, F., & Talbott, M. (2019). Enhancing Low-Intensity Coaching in Parent Implemented Early Start Denver Model Intervention for Early Autism: A Randomized Comparison Treatment Trial. Journal of Autism and Developmental Disorders, 49(2), 632–646. https://doi.org/10.1007/s10803-018-3740-5 Russell, K. M., & Ingersoll, B. (2020). Factors related to parental therapeutic self-efficacy in a parent-mediated intervention for children with autism spectrum disorder: A mixed methods study. Autism, 136236132097423. https://doi.org/10.1177/1362361320974233 Saldaña, J. (2013). The coding manual for qualitative researchers (2nd ed). SAGE. Salomone, E., Pacione, L., Shire, S., Brown, F. L., Reichow, B., & Servili, C. (2019). Development of the WHO Caregiver Skills Training Program for Developmental Disorders or Delays. Frontiers in Psychiatry, 10, 769. https://doi.org/10.3389/fpsyt.2019.00769 105 Sandbank, M., Bottema-Beutel, K., Crowley, S., Cassidy, M., Dunham, K., Feldman, J. I., Crank, J., Albarran, S. A., Raj, S., Mahbub, P., & Woynaroski, T. G. (2020). Project AIM: Autism intervention meta-analysis for studies of young children. Psychological Bulletin, 146(1), 1–29. https://doi.org/10.1037/bul0000215 Schreibman, L., Dawson, G., Stahmer, A. C., Landa, R., Rogers, S. J., McGee, G. G., Kasari, C., Ingersoll, B., Kaiser, A. P., Bruinsma, Y., McNerney, E., Wetherby, A., & Halladay, A. (2015). Naturalistic Developmental Behavioral Interventions: Empirically Validated Treatments for Autism Spectrum Disorder. Journal of Autism and Developmental Disorders, 45(8), 2411–2428. https://doi.org/10.1007/s10803-015-2407-8 Scriven, M. (1994). The Fine Line Between Evaluation and Explanation. Research on Social Work Practice, 15(1), 75–77. Sermeus, W. (2015). Modelling process and outcomes in complex interventions. In D. A. Richards & I. R. Hallberg (Eds.), Complex Interventions in Health: An Overview of Research Methods (pp. 29–45). Routledge. Shanley, J. R., & Niec, L. N. (2010). Coaching parents to change: The impact of in vivo feedback on parents’ acquisition of skills. Journal of Clinical Child & Adolescent Psychology, 39(2), 282–287. Shenouda, J., Barrett, E., Davidow, A. L., Sidwell, K., Halperin, W., Silenzio, V. M. B., & Zahorodny, W. (2022). Disparities in Early Intervention Program Participation by Children With Autism Spectrum Disorder in a US Metropolitan Area, 2006 to 2016. JAMA Pediatrics, 176(9), 906. https://doi.org/10.1001/jamapediatrics.2022.2366 Skinner, B. F. (1953). Science and Human Behavior. Macmillan. https://www.bfskinner.org/newtestsite/wp- content/uploads/2014/02/ScienceHumanBehavior.pdf Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K. A., Craig, N., French, D. P., McIntosh, E., Petticrew, M., Rycroft-Malone, J., White, M., & Moore, L. (2021). Framework for the development and evaluation of complex interventions: Gap analysis, workshop and consultation-informed update. Health Technology Assessment, 25(57), 1–132. https://doi.org/10.3310/hta25570 Sparrow, S., Cicchetti, D., & Balla, D. (2005). Vineland adaptive behavior scales:(Vineland II), survey interview form/caregiver rating form. Livonia, MN: Pearson Assessments. Stadnick, N. A., Stahmer, A., & Brookman-Frazee, L. (2015). Preliminary Effectiveness of Project ImPACT: A Parent-Mediated Intervention for Children with Autism Spectrum Disorder Delivered in a Community Program. Journal of Autism and Developmental Disorders, 45(7), 2092–2104. https://doi.org/10.1007/s10803-015-2376-y 106 Stahmer, A. C., Rieth, S. R., Dickson, K. S., Feder, J., Burgeson, M., Searcy, K., & Brookman- Frazee, L. (2019). Project ImPACT for Toddlers: Pilot outcomes of a community adaptation of an intervention for autism risk. Autism, 136236131987808. https://doi.org/10.1177/1362361319878080 Sterman, J. D. (2006). Learning from Evidence in a Complex World. American Journal of Public Health, 96(3), 505–514. https://doi.org/10.2105/AJPH.2005.066043 Straiton, D. (2020). Parent Training for Youth with Autism Served in Community Settings: A Mixed-Methods Investigation Within a Community Mental Health System. Journal of Autism and Developmental Disorders, 12. Taplin, D. H., Clark, H., Collins, E., & Colby, D. C. (2013). Theory of Change Technical Papers: A Series of Papers to Support Development of Theories of Change Based on Practice in the Field. https://www.theoryofchange.org/wp- content/uploads/toco_library/pdf/ToC-Tech-Papers.pdf Tiede, G., & Walton, K. M. (2019). Meta-analysis of naturalistic developmental behavioral interventions for young children with autism spectrum disorder. Autism, 136236131983637. https://doi.org/10.1177/1362361319836371 Tonge, B., Brereton, A., Kiomall, M., Mackinnon, A., King, N., & Rinehart, N. (2006). Effects on parental mental health of an education and skills training program for parents of young children with autism: A randomized controlled trial. J Am Acad Child Adolesc Psychiatry, 45(5), 561–569. https://doi.org/10.1097/01.chi.0000205701.48324.26 Upton, G., & Cook, I. (2014). A Dictionary of Statistics (3rd ed.). Oxford University Press. https://doi.org/10.1093/acref/9780199679188.001.0001 Vivanti, G., Kasari, C., Green, J., Mandell, D., Maye, M., & Hudry, K. (2018). Implementing and evaluating early intervention for children with autism: Where are the gaps and what should we do?: Gaps and priorities for intervention research. Autism Research, 11(1), 16– 23. https://doi.org/10.1002/aur.1900 Wainer, A. L., Berger, N. I., & Ingersoll, B. R. (2017). Brief Report: The Preliminary Psychometric Properties of the Social Communication Checklist. Journal of Autism and Developmental Disorders, 47(4), 1231–1238. https://doi.org/10.1007/s10803-016-3026-8 Weiner, B. (1985). An attributional theory of achievement motivation and emotion. Psychological Review, 92(4), 548. Weiss, C. H. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. In New approaches to evaluating community initiatives: Concepts, methods, and contexts. (pp. 65–92). Aspen Institute. https://pdfs.semanticscholar.org/ff5a/3eea8d1d4e07a768ded6b426b425efde8f7c.pdf 107 Weiss, C. H. (1997). Theory-based evaluation: Past, present, and future. New Directions for Evaluation, 1997(76), 41–55. https://doi.org/10.1002/ev.1086 Wetherby, A. M., Woods, J., Guthrie, W., Delehanty, A., Brown, J. A., Morgan, L., Holland, R. D., Schatschneider, C., & Lord, C. (2018). Changing Developmental Trajectories of Toddlers With Autism Spectrum Disorder: Strategies for Bridging Research to Community Practice. Journal of Speech, Language, and Hearing Research, 61(11), 2615–2628. https://doi.org/10.1044/2018_JSLHR-L-RSAUT-18-0028 Wood, J. J., McLeod, B. D., Klebanoff, S., & Brookman-Frazee, L. (2015). Toward the implementation of evidence-based interventions for youth with autism spectrum disorders in schools and community agencies. Behavior Therapy, 46(1), 83–95. Wyatt Kaminski, J., Valle, L. A., Filene, J. H., & Boyle, C. L. (2008). A meta-analytic review of components associated with parent training program effectiveness. Journal of Abnormal Child Psychology, 36(4), 567–589. Yoder, P. J., Stone, W. L., & Edmunds, S. R. (2021a). For which younger siblings of children with autism spectrum disorder does parent-mediated intervention work? Autism, 25(1), 58–69. https://doi.org/10.1177/1362361320943373 Yoder, P. J., Stone, W. L., & Edmunds, S. R. (2021b). Parent utilization of ImPACT intervention strategies is a mediator of proximal then distal social communication outcomes in younger siblings of children with ASD. Autism, 25(1), 44–57. https://doi.org/10.1177/1362361320946883 Zimmerman, B. J. (2000). Self-efficacy: An essential motive to learn. Contemporary Educational Psychology, 25(1), 82–91. 108 APPENDIX A: PROJECT IMPACT INTERVENTION ELEMENTS Table A.1. Project ImPACT Intervention Elements. Intervention unit Intervention element Brief description Focus on your Follow your child’s lead Stay face to face with your child, join in a child child-led activity, avoid directions, and respond to your child’s actions Imitate your child Imitate your child’s gestures, facial expressions, body movements, vocalizations, and play with toys and objects Adjust your Use animation Be excited about the activity, exaggerate communication gestures, facial expressions, and vocal quality, adjust your animation to help your child stay regulated Model and expand Comment on what your child is seeing, communication hearing, and doing using simple language as well as gestures and visual cues, and expand on the child’s communication Create Playful obstruction Use an anticipatory cue or phrase then opportunities playfully block the child’s activity, then respond to the child’s communication Balanced turns Use an anticipatory cue or phrase then take a turn and model a play action, then respond to the child’s communication and/or give the child a turn Communicative Put items in sight but out of reach, control temptations access or give small portions, do something silly, or use items requiring adult assistance, then respond to the child’s communication Teach new skills Prompts & rewards When the child is motivated, use prompts to Using communication support the child in demonstrating a more Understanding complex social communication skills; communication Once the child demonstrates the skill, Imitation provide a natural, positive reward Play Shape the Use Project ImPACT techniques together to interaction keep your child engaged and learning; select strategies to emphasize based on your child’s motivation, mood, and the activity Note. Intervention element descriptions adapted from the Project ImPACT Manual for Parents (Ingersoll & Dvortcsak, 2019). 109 APPENDIX B: QUANTITATIVE VARIABLE CORRELATIONS Table B.1. Pearson correlations among variables used in quantitative analyses (Models 1-4). 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1. Caregiver Fidelity r 1 0.330 0.264 0.168 0.076 -0.074 -0.257 -0.163 -0.264 -0.171 -0.059 0.057 0.044 0.030 -0.028 -0.011 T1 M1 p 0.002 0.015 0.127 0.480 0.601 0.016 0.154 0.013 0.119 0.607 0.658 0.683 0.832 0.833 0.935 N 88 84 84 84 88 52 88 78 88 84 79 63 87 54 60 56 2. Caregiver Fidelity r 0.330 1 0.812 0.817 -0.096 -0.095 -0.012 -0.160 -0.014 -0.164 -0.008 0.042 0.083 0.093 0.089 0.253 M1 T2 p 0.002 0.000 0.000 0.382 0.502 0.915 0.162 0.902 0.134 0.944 0.744 0.450 0.503 0.508 0.058 N 84 85 85 85 85 52 85 78 85 85 77 63 84 54 58 57 3. Focus on your r 0.264 0.812 1 0.492 -0.076 -0.140 0.030 -0.048 0.005 -0.112 -0.069 -0.049 0.053 0.035 0.031 0.184 M3 Child Fidelity T2 p 0.015 0.000 0.000 0.489 0.322 0.783 0.674 0.963 0.306 0.551 0.702 0.632 0.803 0.819 0.170 N 84 85 85 85 85 52 85 78 85 85 77 63 84 54 58 57 4. Create r 0.168 0.817 0.492 1 -0.091 0.010 0.146 -0.068 0.151 -0.068 -0.060 0.019 0.012 0.035 -0.024 0.088 Opportunities p 0.127 0.000 0.000 0.408 0.943 0.184 0.555 0.169 0.535 0.607 0.880 0.913 0.803 0.859 0.516 Fidelity T2 M4 N 84 85 85 85 85 52 85 78 85 85 77 63 84 54 58 57 5. Parenting Self- r 0.076 -0.096 -0.076 -0.091 1 0.529 0.104 0.046 0.087 0.186 -0.330 -0.222 -0.064 0.026 -0.223 -0.271 M2 efficacy T1 p 0.480 0.382 0.489 0.408 0.000 0.330 0.687 0.417 0.089 0.003 0.077 0.545 0.849 0.085 0.041 N 88 85 85 85 92 53 89 79 89 85 80 64 91 55 61 57 6. Parenting Self- r -0.074 -0.095 -0.140 0.010 0.529 1 0.041 0.028 0.058 0.153 -0.045 0.173 -0.140 -0.049 -0.124 -0.155 efficacy T3 M2 p 0.601 0.502 0.322 0.943 0.000 0.768 0.845 0.679 0.279 0.767 0.306 0.317 0.732 0.376 0.277 N 52 52 52 52 53 53 53 52 53 52 46 37 53 51 53 51 7. Child Social r -0.257 -0.012 0.030 0.146 0.104 0.041 1 0.712 0.972 0.693 -0.611 -0.603 -0.372 -0.352 -0.629 -0.619 Communication T1 p 0.016 0.915 0.783 0.184 0.330 0.768 0.000 0.000 0.000 0.000 0.000 0.000 0.008 0.000 0.000 M1, M2 N 88 85 85 85 89 53 89 79 89 85 80 64 88 55 61 57 8. Child Social r -0.163 -0.160 -0.048 -0.068 0.046 0.028 0.712 1 0.713 0.748 -0.554 -0.673 -0.372 -0.334 -0.502 -0.668 Communication T3 p 0.154 0.162 0.674 0.555 0.687 0.845 0.000 0.000 0.000 0.000 0.000 0.001 0.013 0.000 0.000 M1 N 78 78 78 78 79 52 79 79 79 78 70 61 78 54 55 53 110 Table B.1. (cont’d) 9. Child Social r -0.264 -0.014 0.005 0.151 0.087 0.058 0.972 0.713 1 0.691 -0.642 -0.608 -0.423 -0.421 -0.665 -0.661 Attention/Engageme p 0.013 0.902 0.963 0.169 0.417 0.679 0.000 0.000 0.000 0.000 0.000 0.000 0.001 0.000 0.000 nt T1 M3 N 88 85 85 85 89 53 89 79 89 85 80 64 88 55 61 57 10. Child Social r -0.171 -0.164 -0.112 -0.068 0.186 0.153 0.693 0.748 0.691 1 -0.521 -0.628 -0.430 -0.439 -0.468 -0.669 Attention/Engageme p 0.119 0.134 0.306 0.535 0.089 0.279 0.000 0.000 0.000 0.000 0.000 0.000 0.001 0.000 0.000 nt T2 M3 N 84 85 85 85 85 52 85 78 85 85 77 63 84 54 58 57 11. Caregiver-report r -0.059 -0.008 -0.069 -0.060 -0.330 -0.045 -0.611 -0.554 -0.642 -0.521 1 0.775 0.510 0.508 0.729 0.677 Social p 0.607 0.944 0.551 0.607 0.003 0.767 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 Communication T1 M2 N 79 77 77 77 80 46 80 70 80 77 80 63 79 47 53 50 12. Caregiver-report r 0.057 0.042 -0.049 0.019 -0.222 0.173 -0.603 -0.673 -0.608 -0.628 0.775 1 0.445 0.544 0.724 0.761 Social p 0.658 0.744 0.702 0.880 0.077 0.306 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 Communication T2 M2 N 63 63 63 63 64 37 64 61 64 63 63 64 63 38 41 39 13. Child r 0.044 0.083 0.053 0.012 -0.064 -0.140 -0.372 -0.372 -0.423 -0.430 0.510 0.445 1 0.856 0.431 0.495 Communication T1 p 0.683 0.450 0.632 0.913 0.545 0.317 0.000 0.001 0.000 0.000 0.000 0.000 0.000 0.001 0.000 M3, M4 N 87 84 84 84 91 53 88 78 88 84 79 63 91 55 61 57 14. Child r 0.030 0.093 0.035 0.035 0.026 -0.049 -0.352 -0.334 -0.421 -0.439 0.508 0.544 0.856 1 0.492 0.524 Communication T3 p 0.832 0.503 0.803 0.803 0.849 0.732 0.008 0.013 0.001 0.001 0.000 0.000 0.000 0.000 0.000 M3, M4 N 54 54 54 54 55 51 55 54 55 54 47 38 55 55 54 52 15. Child Initiations r -0.028 0.089 0.031 -0.024 -0.223 -0.124 -0.629 -0.502 -0.665 -0.468 0.729 0.724 0.431 0.492 1 0.853 T1 M4 p 0.833 0.508 0.819 0.859 0.085 0.376 0.000 0.000 0.000 0.000 0.000 0.000 0.001 0.000 0.000 N 60 58 58 58 61 53 61 55 61 58 53 41 61 54 61 57 16. Child Initiations r -0.011 0.253 0.184 0.088 -0.271 -0.155 -0.619 -0.668 -0.661 -0.669 0.677 0.761 0.495 0.524 0.853 1 T2 M4 p 0.935 0.058 0.170 0.516 0.041 0.277 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 N 56 57 57 57 57 51 57 53 57 57 50 39 57 52 57 57 M1 M2 M3 M4 Note. Variable included in Model 1; Variable included in Model 2; Variable included in Model 3; Variable included in Model 4; T1 = Time 1; T2 = Time 2; T3 = Time 3. 111