INTEGRATED BEHAVIORAL HEALTH TREATMENT FOR ADHD REFERRALS IN PEDIATRIC PRIMARY CARE: CLINICAL IMPROVEMENT , ACCEPTABILITY, ADHERENCE, AND COST OF CARE By Jeffrey D. Shahidullah A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of School Psychology Doctor of Philosophy 2015 ABSTRACT INTEGRATED BEHAVIORAL HEALTH TREATMENT FOR ADHD REFERRALS IN PEDIATRIC PRIMARY CARE: CLINICAL IMPROVEMENT , ACCEPTABILITY, ADHERENCE, AND COST OF CARE By Jeffrey D. Shahidullah While Integ rated care models may offer primary care providers with the resources to meet best practice guidelines by using integrated behavioral health (IBH) services whereby embedded psychologists with specialty training in child behavioral health work closely with families and schools in providing multimodal behavioral treatment within the context of the Family - Centered Medical Home . This study used a repeated measures design at weeks 0, 1, 6, and 12 to evaluate the effectiveness of the Michigan Integrated Behaviora l Health ADHD Treatment Algorithm compared with outcomes associated with targeted externalizing behaviors from standard medical care (SMC; i.e., treatment as usual) when controlling for demographic covariates thought to be potential moderators of treatment response (i.e., age, gender, ethnicity, insurance - status). Participants consisted of 73 parents and 63 teachers ( N = 136) of children referred for ADHD concerns ( N = 73) within two university hospital - affiliated primary care clinics. Results indicated tha t (1) children receiving the IBH model of care demonstrated lower externalizing ADHD symptoms than SMC at 12 weeks post - treatment (0.36 - 0.43 standardized mean difference); (2) IBH treatment was rated significantly more acceptable than SMC by parents; and (3) IBH treatment yielded high levels of adherence by both parents and teachers. Clinical service reimbursement data between the two models of care are discussed. demonstrate how an IBH model of care for externalizing symptoms associa ted with ADHD referrals may be provided as an alternative to the current standard of care and be delivered within a brief family - centered primary care model, while adhering to evidence - based treatment guidelines. iv ACKNOWLEDGEMENTS This research was funded by grants from the Blue Cross Blue Shield of Michigan Foundation (BCBSM Foundation) Student Research Award (Grant Number 2073.SAP), the Society for the Study of School Psychology (SSSP) Dissertation Research Award, the Michigan Sta te University/College of Education Dykstra Endowed Scholarship for Program through the initiative of the program director, Dr. John S. Carlson. Funding was also received by Divis ion 54/Society of Pediatric Psychology of APA in the form of a student research award for a poster that was presented based on the results of this study. Finally, funding was received through the Hyman - Lambert Memorial Scholarship of the American Academy o f School Psychology (AASP), the Science Directorate of APA, and a Leadership Education in Neurodevelopmental Disabilities (LEND) training grant Convention in Washington, DC t o present the results of this study. v TABLE OF CONTENTS 1 12 Attention - 12 13 14 15 15 Evidence - Based Treatment of Externalizing 17 Pharmacological t 18 Pharmacological t 22 Pharmacological t 24 Behavioral t 25 Behavioral t 30 Behavioral t 31 Combined t 32 ADHD treatment: Acceptability and a 35 ADHD t 42 Systems of Pediatric Primary Care for Externalizing A 44 Standard medical c 45 48 Integrated behavioral h 49 51 Systems of c are: 52 53 Research q 53 Research q 54 Research q 55 Research q uestion 56 58 58 5 9 IBH i 59 SMC i ntervention (Treatment as u 63 63 63 64 65 66 Data c 66 Data a 68 vi 85 Research q 85 Parent r 85 Teacher r 88 Research q 91 Research q 92 Research q 92 CHAPTER 5: 95 104 107 Future p 107 Future r 110 114 1 CHAPTER 1: INTRODUCTION The integration of behavioral health services within primary care practice has been cited as an essential element within the push for population - based healthcare design as part of recent healthcare reform efforts (Brown Levey, Miller, & deGruy, 2012). The passing of the Patient Protection and Affordable Care Act (PPACA; P. L. 111 - 148; Obama, 2009) is aimed at offering greater access to and better quality of healthcare by integrating existing systems and providers to provide comprehensive care within the con text of the Family - Centered Medical Home (FCMH). The FCMH model involves a family - centered approach to providing comprehensive primary care whereby services are coordinated across a multidisciplinary health care team while emphasizing continuous communicat ion and information transfer that is typically physician - directed (Braddock, Snyder, Neubauer, & Fischer, 2012). Much research over the past 20 years has demonstrated that the integration of behavioral health services within primary care practice is effect ive in treating several behavioral health conditions such as anxiety and depression within adult populations (Blount, 2003; Cape, Whittington, Buszewicz, Wallace, & Underwood, 2010; Katon et al., 1995). However, there is limited research on the effectivene ss of this model of care within pediatric practice settings for children and adolescents (hereafter children ) with externalizing behavioral problems associated with attention - deficit/hyperactivity disorder (ADHD) clinic referrals. Core (e.g., hyp eractivity, impulsivity) and peripheral (e.g., aggression, disruptive behavior) externalizing behavior problems have long been common in child mental health centers and often occur initially in the context of ADHD clinic referrals (Patterson, 1976; William s, Klinepeter, Palmes, Pulley, & Foy, 2004). With epidemiological studies putting the childhood prevalence rate of ADHD as high as 9% (Merikangas et al., 2010), it is one of the 2 most common conditions that patients seek treatment for within primary care (L eslie, Rappo, Abelson, Jenkins, & Sewell, 2000). In fact, recent estimates suggest as many as 50 to 70% of all patients seen in primary care have referral concerns pertaining to their mental or behavioral health (Belar, 2008; Gatchel & Oordt, 2003), and mo re than 75% of office visits for ADHD occur within the primary care context (Zarin et al., 1998). Given the high comorbidity rates of ADHD with other disruptive behavior disorders such as oppositional defiant disorder (ODD; 50%) and conduct disorder (CD; 25%; American Psychiatric Association, 2013), externalizing behavioral problems associated with both core and peripheral features of ADHD (e.g., hyperactivity, impulsivity, aggression, oppositionality, conduct problems) represent an increasingly common referral concern which requires considerable attention to facilitate early detection and intervention within a evidence - based biopsychosocial framework. The role of primary care providers (PCPs; e.g., family physicians, pediatricians) has become integr al in the clinical care of externalizing behaviors associated with ADHD, in part, due to the shortage of specialty mental health providers (e.g., child psychiatrists). Given the need to treat a large volume of patients in both a time - and cost - effective ma nner, the standard of care for most pediatricians is to prescribe stimulant medication, often as the single form of therapy ( Rushton, Fant, & Clark, 2004). Unfortunately, the issue of medication overprescribing has been a contentious topic within the geogr aphical area where this research study takes place as Michigan has long seen one of the highest rates in the nation of per capital consumption of methylphenidate in child populations (Rappley, Gardiner, Jetton, & Houang, 1999). In fact, a report ( Udow - Phil lips, Ogundimu, Ehrlich, Kofke - Egger, & Stock, 2010) from the University of state report methylphenidate prescription rates that are 9% higher than the national avera ge. 3 Despite the demonstrated short term efficacy of medication in remediating core externalizing symptoms associated with ADHD (e.g., hyperactivity, impulsivity) with effect sizes in the moderate to large range (Conners, 2002; Jensen et al., 2001), much resear ch calls into question how these medications are currently being used and whether evidence - based behavioral interventions can provide patients with a better risk - to - benefit profile when employed as first - line care when considering the scope of behavioral i mpairments associated with ADHD clinic referrals (American Psychological Association [APA], 2006; Swanson & Volkow, 2009). More specifically, there is little evidence to suggest that stimulant medications are effective in remediating important peripheral externalizing symptoms of ADHD (e.g., aggression, oppositionality, conduct problems), which often affect parent - child/peer relationships and school functioning (APA, 2006; Pelham, Wheeler, & Chronis, 1998). In fact, when focusing specifically on peripheral externalizing symptoms, the 14 - month follow - up of the National Institutes of Mental Health Multimodal Treatment Study for ADHD (MTA, 1999a) found community care medication management to be significantly less effective than combined treatment on opposition al/aggressive behaviors. It is also interesting to note that behavioral treatments alone have been found to be comparable to stimulant medication on treatment effect sizes for both core (e.g., hyperactivity, impulsivity; DuPaul & Eckert, 1997; Fabiano et a l., 2009; Pelham & Fabiano, 2008) and peripheral (e.g., aggression, oppositionality; Jensen et al., 2001) externalizing symptoms. Further, the effects of several behavioral treatments are comparable to those obtained with low to moderate doses of stimulant medication (Pelham et al., 1998; Pelham & Waschbusch, 1999). ramifications for treatment decision - making considering the predictable short term side effect 4 risks and un known long term adverse effects associated with chronic stimulant use (Lerner & Pelham, Sams, & Carlson, 1992) have evaluated the additive effects of treatment as a function of dosage or intensity (i.e., dosage effects). The results of these studies indicate that low dose medication and high dose behavioral treatment have approximately equivalent effects on core externalizing symptoms as high dose medication and high dose behavioral treatment. Relatedly, results of the MTA trial (MTA, 1999a; Vitiello et al., 2001) found children receiving a combined treatment of behavioral therapy plus medication ultimately received endpoint doses that were 20% less (31.1 mg/day) than those receiving medication only (38.1 mg/day). The implication of this result is an important consideration for treatment decision - makers who may desire to minimize medication use, especially if the same therapeutic outcome can be obtained with fewer side effects. In fact, Pelham (2012) posits that physicians can effectively prescribe lower doses of stimulant medication when starting with behavioral treatment. For parents, having this treatment option available may improve their views regarding acceptabili ty and increase the likelihood for treatment adherence. A critical consideration in evaluating treatment outcomes, particularly when therapies are delivered within applied contexts such as primary care, is establishing the degree to which patient s adhere to treatment as intended by the clinician as this component of treatment integrity has long been implicated as paramount to intervention success (Gresham, 1989; Patterson & Chamberlain, 1994). As the effectiveness of treatment is reciprocally depe ndent on adherence to that regimen, researchers (e.g., Eckert & Hintze, 2000; Phelps, Brown, & Power, 2002; Witt & Elliot, 1985) have highlighted the importance of determining parent - and teacher - reported acceptability of treatment as this may be a signifi cant determinant in whether an intervention will 5 be pursued and adhered to. Parents and teachers have consistently rated behavioral treatments for ADHD as more acceptable than medication (Cohen & Thompson, 1982; Johnston, Hommersen, & Seipp, 2008; MTA, 199 9a). The results of many large - scale parent surveys and insurance claims analyses have demonstrated parental reluctance to initiate a medication trial (dosReis, Mychailyszyn, Evans - Lacko, Beltran, Riley, & Myers, 2009) and adhere to the treatment regimen o ver several months (Habel, Schaefer, Levine, Bhat, & Elliot, 2005; Toomey et al., 2012). Though the MTA (1999a) study showed that MTA - - of - the - llow - up care) was the most effective ADHD treatment, parents were clearly not satisfied with medication as the sole form of therapy and reported they would recommend behavioral treatment more often than medication or combined treatment. Parent preferences for behavioral treatment were even - based stimulant medication prescription without attention to close behavioral monitoring). The preponderance of evidence for incorporating behavi oral treatments into the clinical care for ADHD as either first - line or, at the very least, combined intervention with stimulant medication has provided empirical support in the establishment of treatment guidelines by the American Academy of Pediatrics (A AP, 2011) and APA (2006). It is important to note that these organizations do not specify different treatment recommendations for core (e.g., hyperactivity, impulsivity) and peripheral (e.g., aggression, disruptive behaviors) externalizing symptoms of ADHD . Specifically, AAP (2011) treatment recommendations for community - based pediatricians focus on (1) specifying targeted outcomes; (2) providing multimodal treatment consisting of stimulant medication with evidence - based parent - and teacher - administrated be havioral therapy; and (3) providing systematic follow - up to quantitatively evaluate treatment 6 response from the collection of both parent and teacher rating scales. Additionally, the APA (2006) Task Force Report on ADHD Treatments found from its comprehens ive review of psychopharmacological, behavioral, and combined interventions, that all three treatments are well - established as acute interventions. However, because only medication causes side effects, the recommendation was made, based on a risk - to - benefi t analysis, that behavioral treatment be used as first - line care, with adjunctive medication use if needed. Coupled with the literature suggesting behavioral therapies may present better options for patients in terms of their acceptability and adherence ra tings than the use of stimulant medication, it becomes apparent why behavioral treatments have garnered strong support as evidenced by AAP (2011) and APA (2006) guidelines. Unfortunately, due to a number of barriers, the standard of care in pediatric prim ary care, where most youth are treated, often falls short of AAP (2011) and APA (2006) guidelines ( Epstein, Langberg, Lichtenstein, Kolb, & Simon , 2013 ). In fact, the MTA (1999a) study ommunity - provided care, yielded the fewest treatment gains when compared to MTA - provided treatment using most often called into question when considering community - b ased treatment include the lack of training that pediatricians receive in child and adolescent mental health and behavioral interventions (Serby, Schmeidler, & Smith, 2002); difficulty in implementing evidence - based behavioral interventions within the time - limited context of primary care (Cummings & lack of coordination among those providers (Leslie, Stallone, Weckerly, McDaniel, & Monn, 7 2006), and also poor follow - t hrough from patients when referral - based behavioral health providers are off - site ( ). care [SMC]) provides service to many children that ma y otherwise not receive any form of treatment, the model calls into question whether the long - term benefits outweigh the risks involved with receiving medication as a monotherapy. This risk - to - benefit dilemma is made more precarious when patients are not p rovided with alternative treatment options such as Hippocratic ethical mantra, children should be provided with treatment that offers the lowest risk of side and adverse effects in relation to expected treatment gains. If parents opt for medication, they must still be presented with other evidence - based behavioral treatment options and the empirical evidence supportive of each. However, with PCPs who lack (1) knowledge of the benefits of behavioral interventions, and (2) the training needed to deliver these interventions, it appears that SMC will continue to be the prescribing of medication. Interestingly, given the economic and business - related context of integrated h ealth care practice, pediatricians may be inhibited from providing comprehensive care for behavioral health concerns even when they are trained to do so. Specifically, from a health maintenance organization (HMO) and administrative perspective, research ha s questioned the long term financial sustainability of models which require pediatricians to deliver comprehensive care for non - medical concerns (Meadows, Valleley, Haack, Thorson, & Evans, 2011). Addressing this lack of behavioral health service reimburse ment, Feldman, Lee, and Perez - Stable (2006) - contracting this type of care to managed behavioral health organizations. Consequently, adequate pediatrician reimbursement may be 8 difficult to obtain when the behavioral health plans. It is clear that new cost - effective treatment models that adhere to AAP (2011) and APA (2006) guidelines and improve the quality of clinical care for children with externa lizing ADHD concerns in pediatric primary care are needed. Currently, a number of service delivery models exist which appear to be potentially - promising alternatives to the current SMC for treatment of externalizing symptoms associated with ADHD clini c referrals. While a critical design flaw in the healthcare system is a lack of the Family - Centered Medical Home has pushed for the physician - directed integration of behavioral health services within primary care. Integrated behavioral health (IBH) models represent a distinct service delivery framework consisting of the systematic coordination of behavioral health with medical services. These models offer an effici ent means for interdisciplinary patient care involving physicians and behavioral health providers (BHPs; e.g., psychologists) that may potentiate numerous patient benefits (APA, 2013; Gatchel & Oordt, 2003; Strosahl, 1999). These benefits include, but are not limited to, increased treatment options, safer treatment options in comparison with psychotropic medication, more effective delivery of evidence - based behavioral treatments with BHPs trained specifically in these interventions, increased treatment adhe rence as the need for external referrals is eliminated and patients are - Chaffee, 2009; patie nt referrals from co - located pediatricians when specialty mental or behavioral health services are needed and, in turn, provide brief evaluation and targeted interventions as part of a coordinated and family - centered pla n . While clinical symptom reduction, treatment acceptability, and 9 adherence are generally considered to be valued outcome metrics, without also demonstrating evidence for the financial viability of these integrated care models, no practice can remain financially sustainable (Goodheart, 2010) . Several researchers (e.g., Blount, 2007; Chiles, Lambert, & Hatch, 1999; Kaplan & Groessl, 2002) have demonstrated the cost - benefit of integrating behavioral health services within medical settings. However, there is a paucity of research targeting behav ioral health integration within pediatric practice settings specifically. The empirical literature regarding behavioral treatment for core and peripheral externalizing symptoms of ADHD suggests the optimal intervention for children consists of a multimo dal approach of parent training, teacher consultation, and child intervention involving the home and school (e.g., daily report card [DRC] for behavior; APA, 2006; Pelham, 2012; Pelham, 2000). Many of the most extensively studied multimodal treatments occu r in the context of large - scale, randomized programs consisting of up to 35 group and individual parent training sessions, up to 24 face - to - face and telephone teacher contacts per school year, and child intervention provided in the context of an intensive summer treatment program with trained specialist counselors (MTA, 1999a). Though not practical for real clinic settings as indicated by a cost - effectiveness analysis of the treatments used in the MTA study (Jensen et al., 2005), these evidence - based interv entions provide a template for which treatment components may potentially be translated in treatment algorithm form for implementation in primary care using an IBH service - delivery model. One treatment model in particular that has potential for a dapted use is the Buffalo Treatment Algorithm for ADHD (Pelham, 2007). This algorithm provides a framework for sequencing treatments using low - dose behavioral intervention consisting of parent training, teacher consultation, and child intervention as first - line care, with progressively higher doses 10 (i.e., intensity/more sessions) based upon on - going assessment in the domains of impairment. These parent and teacher sessions provide skills training on principles of operant behavior change such as shaping, rei nforcement, extinction, and contingency management that are needed for the implementation of an individualized DRC . The DRC is a key treatment component used as a first - line alternative to medication, but also provides additional assessment data used to ru le out the environmental mismatch hypothesis as a cause of behavior concerns. For example, if children respond to a motivation - based intervention, they are not likely to meet ADHD diagnostic criteria conceptualizing ADHD as a neurodevelopmental deficit. Th rough this multimodal behavioral treatment, the continued use of the DRC within a combined approach then serves the dual purpose of a treatment progress monitor assessi ng behavioral response to changes in medication through targeted daily ratings. The primary purpose of this study was to evaluate an alternative to the current standard of care for core (e.g., hyperactivity, impulsivity) and peripheral (e.g., aggres sion, conduct problems) externalizing behavior problems associated with ADHD referrals in community - based pediatric primary care when controlling for demographic covariates of age, gender, ethnicity, and insurance status (i.e., commercially - vs. Medicaid - i nsured). The alternative to SMC evaluated in the present study was the psychologist - delivered Michigan Integrated Behavioral Health ADHD Treatment Algorithm (B. M. Lancaster, personal communi cation, November 17, 2012) . This algorithm was adapted from the empirically - based Buffalo Algorithm and incorporates its major components (i.e., parent/teacher management training, child home - school intervention [DRC]) in a manner to fit within the context of a brief family - centered therapeutic approach to treating ext ernalizing ADHD symptoms in 4 to 6 ± clinic sessions over a span of 2 to 3 months with 11 periodic follow - up. A secondary purpose of this study was to examine treatment preferences of patient caretakers by evaluating parent - and teacher - reported acceptability of the IBH treatment in relation to parent ratings for SMC. In addition to assessing parent adherence to SMC, this study also assessed parent and teacher adherence to the IBH treatment, as implementing this treatment with integrity may provide a means for PCPs to adhere to both AAP (2011) and APA (2006) treatment guidelines. Finally, this study quantified the comparative cost of care using 2014 Current Procedural Terminology (CPT) billing code reimbursement between the IBH model and SMC. This st udy may add to the applied literature as little research has evaluated the effectiveness of multimodal IBH treatment specifically for core (e.g., hyperactivity, impulsivity) and peripheral (e.g., aggression, oppositionality, conduct problems) externalizing behavior problems associated with ADHD clinic referrals in pediatric primary care. This is important as the majority of treatment studies tend to include inattentive - type symptoms within a global ADHD construct when the reality is that practitioners are m uch more likely to treat patients presenting with externalizing symptoms associated with ADHD and/or other comorbid disorders (i.e., ODD, CD; Hinshaw & Lee, 2003). This study may ultimately demonstrate to parents, PCPs, and HMO/administrative decision - make rs how the Michigan Algorithm may provide effective clinical care for externalizing problem behaviors associated with ADHD clinic referrals that adheres to evidence - based guidelines within the time - limited primary care context of the family - centered medica l home. 12 CHAPTER 2 : LITERATURE REVIEW The following literature review first provides a detailed overview of ADHD with specific attention paid to its diagnosis, etiology, prevalence, and outcomes pertaining to child and adolescent populations. Next, the review details the evidence - based treatments indicated for externalizing behaviors associated with ADHD in pediatric populations and offers a critique of the strengths and limitations of each withi n the context of current treatment guidelines and recommendations put forth by the AAP and APA. In addition to treatment efficacy, this critique of evidence - based treatments for externalizing behaviors also considers factors such as parent and teacher acce ptability and adherence. Finally, the review examines current systems of pediatric primary care for externalizing symptoms of ADHD, specifically in the context of their amenability to adhere to established treatment guidelines and contemporary systems - leve l reforms and initiatives such as the Patient Protection and Affordable Care Act and the family - centered medical home . A ttention - Deficit/Hyperactivity Disorder Attention - deficit/hyperactivity disorder (ADHD) begins in childhood and is characterized by a p ersistent pattern of developmentally inappropriate levels of inattention and/or hyperactivity - impulsivity resulting in functional impairment across settings (APA, 2013). Specifically, inattention is exhibited behaviorally in ADHD as wandering off task, lacking persistence, having difficulty sustaining focus, and being disorganized and is not due to defiance or lack of comprehension. Hyperactivity refers to excessive motor activity when it is not a ppropriate or excessive fidgeting, tapping, or talkativeness. Impulsivity refers to hasty actions that occur in the moment without forethought and that have high potential for harm to the 13 individual. ADHD is the most common childhood behavioral disorder an d can profoundly affect school performance, social behavior, self - esteem, and quality of life (AAP, 2011). Diagnosis. Unlike most medical conditions, medical tests or imaging studies do not exist that can be used to diagnose ADHD. Rather the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM - 5; APA, 2013), which lists operationalized diagnostic criteria using an empirically - based classification system of neurodevelopmental disorders, is used to c onfirm a diagnosis . A total of 1 8 symptoms divided into two domains (inattention and hyperactivity/impulsivity) are listed, of which at least six symptoms persisting for at least six months in one domain are required for diagnosis. To substantiate the clinical presentation during childho hyperactive - substantiate the requirement for cross - situational impairment, diagnosis must meet the crit erion - impulsive symptoms are present in two or more settings uce the quality of, social, academic, or during the course of schizophrenia or another psychotic disorder and are not better explained by APA, 2013; p. 60). Because the symptoms of ADHD overlap and/or coexist with many other conditions (e.g., other neurodevelopmental disorders, emotional or behavioral disorders, substance use disorders, medication - induced symptoms of ADHD, psychosocial or en vironmental factors) these conditions are differentiated from ADHD during the evaluation. 14 Etiology. While most findings on etiologies are correlational in nature and do not assign direct cause, there are several neurobiological and genetic factors that h ave been strongly implicated as contributors to ADHD. Since the 1960s neurological studies have identified numerous developmental pathways leading to ADHD, that all likely have some common influence on a disturbance in a final pathway in the nervous system associated with abnormalities in frontal brain regions (e.g., prefrontal cortical - striatal network; Barkley, 2003). However, more recent research has led to a general consensus that ADHD may be associated with white matter tract/connectivity abnormalities and not necessarily structural/regional abnormalities (Qiu, Ye, Li, Liu, Xie, & Wang, 2011; Weyandt, Swentosky, & Gyda Gudmundsdottir, 2013). Genetic studies using families to establish the heritability and strength of genetic factors in developing ADHD h ave demonstrated a strong familial association with approximately one - third of affected children having a first - degree relative with the disorder (AAP, 2001; Reiff & Stein, 2011). Further, twin studies have found that genetic factors can be attributed to 7 5% of the variance in the ADHD phenotype. More specifically, twin studies reflect a 50 to 80% concordance for ADHD between monozygotic twins and up to 33% concordance between dizygotic twins ( Bradley & Golden, 2001). Molecular genetic studies have identifi ed several candidate genes showing statistically significant evidence of association with ADHD including many dopamine transporter genes (e.g., dopamine receptor D4 gene; Barr et al., 2000). While these neurobiological and genetic factors are more predicti ve of ADHD, there are many environmental factors that appear to be contributory. These include neurotoxin exposure (e.g., lead), tobacco and alcohol exposure during pregnancy, pregnancy and delivery complications, low birth weight, or infections (e.g., enc ephalitis). Other factors that often act as moderators of adaptive 15 functioning and emotional health for children with ADHD include family interaction patterns in early childhood and parental psychopathology (APA, 2013). Prevalence. Recent epidemiological studies indicate up to a 9% lifetime prevalence of ADHD in children in the United States (Froehlich, Lanphear, Epstein, Barbaresi, Katusic, & Kahn, 2007; Merikangas et al., 2010; Pastor & Reuben, 2008). However, these prevalence estimates vary predictably depending on the methodology used and participant samples. Specifically, boys are approximately three times more likely to be diagnosed with ADHD than girls with the male to female ratio ranging from 4 to 1 for diagnosis of predominantly hyperactive - impuls ive type, to 2 to 1 for the predominantly inattentive type (Staller & Faraone, 2006). Other factors have also been found to influence prevalence rates. The Centers for Disease Control and Prevention (CDCP; 2010) found that parent - reported ADHD diagnoses we re highest among children covered by Medicaid. Further, Froehlich and colleagues (2007) found that poor children were more likely to meet ADHD diagnostic criteria than were wealthier children, however, wealthier children were more likely to receive treatme nt. Finally, there have also been a number of ethnicity - related diagnostic differences, as childhood diagnosis rates in the United States for Caucasians are between 36 and 56% higher than for minority populations (APA, 2013; Morgan, Staff, Hillemeier, Fark as, & Maczuga, 2013). While a definitive explanation for this disproportionality has not been made, a prominent hypothesis points to lack of health insurance coverage in minority populations compared with Caucasians (Flores, Olson, & Tomany - Korman, 2005). Outcomes. Long term outcomes for youth with externalizing symptoms associated with ADHD are moderated by the type and severity of symptoms, co - morbid conditions, developmental variables, family factors, and treatment. Despite a relative symptomatic 16 improv ement during adolescence, functional abnormalities typically persist into adulthood. ADHD, particularly the hyperactive - impulsive dimension, often occurs along with other externalizing disorders such as oppositional defiant disorder (ODD; Barkley, 2006; Be auchaine, Hinshaw, & Pang, 2010; Campbell, Shaw, & Gilliom, 2000). In fact, ODD is more than 10 times more likely to occur within individuals with ADHD diagnoses than in the general population (Angold, Costello, & Erkanli, 1999). Further, oppositional defi ant and ADHD symptoms have been shown to be developmental precursors to later conduct problems (Lahey, Van Hulle, ADHD in children is also a significant risk factor for later development of delinquency and psy chiatric problems including psychiatric hospitalization (Molina et al., 2007). ADHD is also associated with academic underachievement and impaired school functioning. Specifically, children with ADHD exhibit lower rates of academic engagement and higher ra tes of off - task behaviors in the classroom ( Vile Junod, DuPaul, Jitendra, Volpe, & Cleary, 2006). Regarding impairments in school performance, academic achievement scores of students with ADHD measure approximately 0.71 standard deviation units below their peers (Frazier, Youngstrom, Glutting, & Watkins, 2007). Additionally, students with ADHD are at higher than average risk for failing grades, grade retention, referral for special education services, and high school dropout, and are less likely to complete secondary education (Barkley, Murphy, & Fischer, 2008). Due to these substantial impairments that children with ADHD exhibit, families, schools, and health care providers invest heavily in treatment resources. Using an analytic cost - of - illness framew ork from a review of 13 published studies, Pelham, Foster, and Robb (2007) estimated the annual cost of ADHD, considering expenses for mental health care and educational costs associated with providing Section 504 services, as well as ancillary costs (e.g. , productivity - 17 related, injury - related, juvenile justice) at $14,576 per child, with a conservative estimate of the societal economic impact at $42.5 billion, with a range between $36 and $52.4 billion. It is also important to note that this financial impa ct on society may be even more profound when also considering costs associated with populations who have comorbid ODD or CD (Offord & Bennett, 1994) Evidence - Based Treatment of Externalizing ADHD Problems in Children The three evidence - based treatments for use with children demonstrating externalizing ADHD symptoms are (1) stimulant medication, (2) behavioral treatment implemented across home and school, and (3) a combination of the two (Barkley, 2006). Despite the demonstrated short - term effectiveness with each of these treatments, there remains much variability within the treatment outcome literature regarding a number of parameters, such as long - term outcomes, type and intensity of treatment, setting, and individual differences in the size and topogra phy of treatment response (Barkley, 2006; Pelham et al., 2000). The following sections review important findings from a number of seminal studies pertaining to the treatment of externalizing behaviors associated with ADHD (e.g., MTA 1999a, 1999b) as well a s other recent literature focusing on ADHD treatments delivered within primary care for externalizing symptoms. The discussion includes those studies that professional organizations (AAP, 2011; APA, 2006) include as empirical support for the development of their ADHD treatment guidelines. It is important to note that while the construct of interest in the present study is externalizing behaviors associated with ADHD, there is a paucity of research literature that differentiates treatment response between in attentive and hyperactive - impulsive type symptoms (herein response specifically in externalizing symptoms of ADHD, very few studies differentiate 18 between core a nd peripheral symptoms. Therefore, in general, the following literature review of treatment efficacy discusses ADHD as a global construct. However, when appropriate specific studies are cited and emphasized that have an acute focus on core and peripheral e xternalizing symptoms. This caveat is provided here to inform the reader of the current state of the ADHD treatment research literature and also to provide additional rationale and justification of the need for the present study which focuses exclusively o n core and peripheral externalizing symptoms associated with ADHD clinic referrals. Pharmacological t reatment. One of the most common treatments for core externalizing symptoms associated with ADHD in children is stimulant medication. These medications a ct on the central nervous system to reduce the inattentive and hyperactive - impulsive behaviors. Specifically, they are often prescribed to increase on - task behavior and decrease interruptions and fidgeting at school and improve parent - child interactions an d compliance at home (Reiff & Stein, 2011), however their efficacy has been better demonstrated in improving core rather than peripheral externalizing symptomatology (MTA, 1999a; Pelham et al., 2001). A variety of both stimulant and nonstimulant medication s exist, all containing slightly different formulations that vary in their pha rmacokinetic effects . Though some nonstimulant options are FDA - approved and have demonstrated efficacy in improving ADHD symptoms ( Mohammadi & Akhondzadeh, 2007), this class of medication remains largely understudied and is typically prescribed only when children are unresponsive to stimulants or are unable able to tolerate their associated side effects. Because these medications are not common used, only treatment response resea rch for studies evaluating the use of stimulant medications are presented hereafter. Double - blind, placebo - controlled studies have consistently found stimulants (e.g., methylphenidate, amphetamine compounds) to be significantly superior to placebo in 19 remediating externalizing ADHD symptoms. In a recent meta - analysis of the published literature over the past 30 years, Faraone and Buitelaar (2010) investigated the results of over 20 double - blind, placebo - controlled studies of amphetamine (i.e., Adderall) and methylphenidate (i.e., Concerta, Ritalin) use and found most effect size measures for treatment efficacy to be statistically significant. Though this meta - analysis included several studies which evaluated treatment response specifically in externalizi ng symptoms, they only included the core externalizing symptoms of hyperactivity - impulsivity and did not include peripheral externalizing symptoms (e.g., aggression, disruptive behaviors). In a different meta - analysis of 115 studies on stimulant medication efficacy, Crenshaw, Kavale, Forness, and Reeve (1999) reported effect sizes of .46 for treating academic impairments and .72 for externalizing behavioral symptoms of ADHD. However, when looking specifically at peripheral externalizing symptoms (e.g., aggr ession, noncompliance), treatment effect sizes were smaller (.61), though still robust. Arguably the most authoritative study pertaining to ADHD treatment in the past 20 years has been the MTA (1999a, 1999b) trial. This study is generally considered to b e the most significant attempt to establish treatment efficacy for medication and other treatments for children with AD HD . Its multisite, 14 - month randomized clinical trials assigned 579 children, aged 7 to 9.9 years, with ADHD to a medication, behavioral treatment, combined treatment, or a community care comparison group. Children in the community care group were not treated in the study, but were referred to their community providers for any services they chose to obtain independently. Over two - thirds ( n = 97 [67.4%]) of the community care participants received stimulant medication. At a 14 - month treatment follow - up, the use of MTA - provided medication (i.e., methylphenidate) decreased core externalizing ADHD symptoms, both alone and in conjunction with beh avioral treatment. Further, participants receiving MTA - provided medication 20 displayed greater symptom improvement, including peripheral externalizing symptoms such as aggression and oppositionality (Jensen et al., 2001) than those receiving community - provid ed medication. Also, MTA - medication management yielded slightly greater improvement in hyperactive/impulsive symptoms than inattentive symptoms according to both parent and teacher ratings. While most participants in the MTA - provided medication and commun ity care groups received largely the same treatment modality (i.e., stimulant medication), there were marked differences in how those medications were titrated for determining therapeutic dosage. The far more intensive and rigorous protocol for monitoring treatment response for the MTA - provided medication included longer (30 vs. 18 min.) and more frequent (8.8 vs. 2.3 visits per year) follow - up visits (Jensen et al., 2001). Symptom improvement discrepancies between the two groups confirm the critical import ance of ongoing treatment monitoring and are a major reason why this practice is included in AAP (2011) treatment guidelines. However, as will be discussed in a later section of this literature review, the MTA - provided medication procedure does not resembl e the current standard of care in community - based primary care clinics. Therefore, the treatment results using the community care comparison group provides a more valid assessment of stimulant medication usage for treating externalizing symptoms associated with ADHD clinic referrals in authentic settings. Regarding moderators ( i.e., variables that, not influenced by treatment, can influence treatment efficacy) of treatment response for children with ADHD, the MTA (1999b) study examined response patterns based on gender and socioeconomic status (SES). Moderator analyses found gender to be significant for core externalizing symptoms of ADHD as combined treatment a nd MTA - managed medication was superior to traditional care in males while only 21 combined treatment was superior to traditional care in females. Regarding SES, low - SES was significant for peripheral externalizing symptoms as this group demonstrated significa ntly superior symptom improvement from combined treatment compared with all other treatment conditions, whereas children who were not in the low - SES group did not demonstrate this difference. Examinations of the moderating influences on treatment response in low - SES families (e.g., poverty - threshold, receiving public assistance, Medicaid beneficiaries, etc.) have been cited in the literature as an important consideration given their demonstrated difficulty with participating fully in behavioral treatments ( Dumas & Wahler, 1983; Webster - Stratton, 1985). In addition to gender and SES, other treatment outcome studies have also examined the moderating influence of ethnicity. For example, Arnold and colleagues (2003) examined ethnicity as a treatment moderator in the MTA study and found that ethnic minorities (i.e., African Americans, Latinos) in general evidenced significant improvements from the combined treatment while Caucasian children did not. Despite the differential treatment response based on ethnicity fo und in these studies, Weisz, Huey, and Weersing (1998) acknowledge the general paucity of empirical treatment outcome literature that exists pertaining to ethnicity - related differences. Despite wide dissemination efforts by AAP in the promotion of its tre atment guidelines, an alarming amount of recent literature indicates poor adherence to these guidelines by PCPs who work with school - age children ( Epstein, Langberg, Lichtenstein, Kolb, & Simon , 2013; Gardner, Kelleher, Pajer, & Camp, 2004 ) . Though 92% of pediatricians report they are familiar with AAP guidelines, only 78% of these clinicians implement the recommendations within their clinical practice (Rushton et al., 2004). Regarding other specialties within primary care, Rushton and colleagues (2004) fou nd only 60% of family physicians were familiar with AAP guidelines, and only 39% incorporate the guidelines into practice. Specifically, only 53% of PCPs who 22 prescribe ADHD medication to children report they conduct follow - up visits, even though these are recommended by AAP (Rushton et al., 2004). Recent literature suggests the number of PCPs who actually conduct these critical follow - up visits to monitor treatment response is lower than what PCPs self - report. In a study using PCP chart review audits to mea sure post - treatment follow - up encounters, Epstein, Langberg, Lichtenstein, Mainwaring, and Luzader (2008) found only 27% of patients/families had telephone contact with their PCP within two weeks of medication initiation, and only 52% had in - office follow - up visits with their PCP within six weeks of medication initiation. Further, this study found only 9% of PCPs had obtained follow - treatment, including monit oring for adverse effects. Regarding the barriers to adhering to AAP treatment guidelines, survey results of 1,374 PCPs indicated that poor access to mental health services and limited insurance reimbursement for services were the two primary facto rs (Rushton et al., 2004). Epstein and colleagues (2013) posit that these logistical concerns severely limit PCPs ability to utilize rating scale data for continuous treatment monitoring and follow - up. First, PCPs may not have the appropriate knowledge of which rating scales to use for the purposes of measuring medication effects in a way that is sensitive to change depending on dosage level. Next, due to short appointment times, it is not feasible to explain the purpose of the scales, distribute the scales to parents and teachers, establish a protocol for efficient return of the scales, and score and interpret the results. This is often a time - consuming process that is not accounted for within insurance reimbursement. Therefore, PCPs often neglect this crit ical component of prescribing and do not communicate with prescribed patients unless severe adverse effects are reported. Pharmacological treatment: Limitations. Results of large - scale randomized clinical 23 trials, such as the MTA (1999a), have profoundly affected how health professionals rationalize the use of stimulants for their patients given the large and relatively clear benefits of these medications that were present at one year and, to a lesser extent, two years post treatment. However, there are ma ny limitations to stimulant usage as much research conducted over the past 25 years has indicated that stimulants do not produce long - term (i.e., two or more years) benefit once medication has been discontinued. As part of the eight - year MTA study follow - u p, Molina and colleagues (2009) demonstrated that no beneficial effects of stimulant medication were apparent after this extended latency for any of the symptom subtypes. In fact, Swanson and colleagues (2007) found that the medication treatment effects in the MTA study appeared to dissipate completely soon after the medication was discontinued. Additionally, the APA (2006) Task Force Report ultimately found that stimulant medication does not have evidence for effectiveness beyond one year. Further, medicat ion has not proven effective in addressing many important peripheral areas of functional impairment such as academic, social, and behavioral skill - building (Pelham & Smith, 2000), with the effects of medication on academic performance being minimal ( ES = 0 .12; MTA, 1999a, 2004). These findings are difficult to reconcile with those of Faraone and Buitelaar (2010), Crenshaw and colleagues (1999), and Pliszka and colleagues (2006) cited previously, thus highlighting the lack of consensus regarding treatment ef fectiveness in peripheral domains of externalizing behaviors associated with ADHD. Furthermore, data suggests between 20 and 30% of children affected with ADHD may not respond to stimulant medication at all and/or may not be able to tolerate associated si de effects (e.g., sleep disturbance, appetite suppression, mood difficulties, exacerbation of co - morbid tic disorders; Schachter, Pham, King, Langford, & Moher, 2001). These side or adverse effects may also include impairments that exist long after medicat ion is discontinued. For 24 example, in the MTA (1999a) study, those receiving medication for one year demonstrated more serious delinquent behavior by 24 and 36 months compared with children randomized to receive behavioral treatment (Molina et al., 2007), t hus demonstrating the potential for differentiated effects in peripheral externalizing symptoms of ADHD. Also, given the impairments related to sleep and appetite, research has long cited suppression in growth height and weight as being linked with stimula nt use in children (Safer & Allen, 1973). Even results from the MTA (1999a) study revealed about a two cm reduction in height gain and a two kg reduction in weight gain for children between ages 10 and 12 receiving stimulant medication compared with other treatment and control groups (Swanson & Volkow, 2009). Pharmacological treatment: Conclusions. Numerous double - blind, placebo - controlled trials have demonstrated the short - term efficacy of stimulant medication in treating externalizing problem behaviors a ssociated with ADHD symptoms in children. Research also shows that most children (70 - 80%; Schachter et al., 2001) respond to these medications with relatively immediate symptom improvement. While stimulant medication is clearly the treatment of choice for PCPs, their inability to demonstrate long - term effectiveness, coupled with their demonstrated side and adverse effects, provides cause for concern. Additionally, while outcome efficacy from stimulant use is contingent on close monitoring of treatment respo nse through systematic titration algorithms, providers within community - based settings clearly do not employ the level of on - going monitoring recommended by AAP. Finally, it is important to highlight a significant example of an action taken by a m ajor group in the field of ADHD treatment research that emphasizes an awareness of the limitations that the use of stimulant medications have as a monotherapy, particularly for treatment of peripheral externalizing symptoms associated with ADHD. Specifical ly, in developing treatment 25 ADHD Project, a consensus panel of researchers (Pliszka et al., 2006) reaffirmed from a review of the literature the well - established effica cy of stimulant medication for treating core symptomatology (i.e., inattention, hyperactivity, impulsivity). However, in what was a significant departure from previous guidelines, the panel in their 2005 revisions to the Texas Algorithm recommended the use of conjunctive behavioral intervention along with stimulant medication for the treatment of peripheral externalizing symptoms of ADHD such as aggression (p. 644). This is interesting to note, as the previous guidelines from 2004 did not include the use of behavioral intervention anywhere in the algorithm. Though on the surface this event may seem like a minor change, the significance of this event is in the recognition by prominent decision makers in the field of the critical limitations that are apparent in the use of stimulant medication as a monotherapy coupled with the awareness that alternative treatment options such as behavioral treatment may offer improved outcomes. Treatment research regarding the use of these behavioral interventions is presented in the next section. Behavioral treatment. While there are significantly fewer published behavioral intervention studies than medication studies, behavioral intervention has long been cited as an effective treatment for children with ADHD, and the extern alizing behavioral problems associated with it (e.g., hyperactivity - impulsivity, aggression, oppositionality, conduct problems) Rosenbaum, & Prince, 1976; Patterson et al., 1993). Key organizations in the fields of education (U.S. Department of Education, 2004), mental health (APA Division of Clinical Child and Adolescent Psychology, 2012; National Association of School Psychologists, 2002; National Institute of Mental H ealth, 2013), and medicine (American Academy of Pediatrics, 2011) have 26 recommended these treatments as evidence - based interventions for child externalizing symptoms associated with ADHD. Further, some researchers (e.g., Pelham, 2012; Pelham et al., 1998; P elham & Waschbusch, 1999) recommend that behavioral treatments consisting of parent and teacher training and child intervention be used as a first - line therapy, with adjunctive medication use if needed, given that the effects of several behavioral treatments are comparable to the effects obtained with low to moderate dose stimulant medication. The following sections review these behavioral treatment components, with particular emphasis paid to those that have been most extensively studied and i ncluded within empirical evaluations of multimodal treatment programs (e.g., MTA, 1999a) and algorithms (Pelham, 2007) and have the potential to be provided within the clinic setting. One widely recognized and evidence - based treatment model is the Buffalo Treatment Algorithm for ADHD (Pelham et al., 2000; Pelham, 2007), which uses multimodal behavioral treatment within the clinic context. The behavioral treatment package used in the Buffalo Algorithm is consistent with the modalities used in the Summer Trea tment Program for ADHD (STP; Pelham et al., 2000), which has been recognized by APA Divisions 53 and 37 as a model program and is listed by the Substance Abuse and Mental Health Services Administration (SAMHSA) as a well - established program for treating AD HD. This program along with the behavioral package used in the MTA (1999a) study consists of parent and teacher training, and child intervention. These components are discussed in more detail in the following sections. Poor parenting practices are a stro ng predictor of negative long - term outcomes in children with externalizing behavior problems such as those associated with ADHD clinic referral (e.g., hyperactivity, impulsivity, aggression, oppositionality, conduct problems; Chamberlain & Patterson, 1995) . For this reason, parent training should be included as an 27 integral evidence - based component of any comprehensive treatment plan (Pelham et al., 1998). Parent training is a model of service delivery in which parents are directly taught specific skills to remediate child behavior problems (Reiff & Stein, 2011). Behavioral programs targeting parental effectiveness (e.g., Living with Children ; Patterson, 1976; The Incredible Years ; Webster - Stratton & Hammond, 1997; Helping the Noncompliant Child ; McMahon & Fo rehand, 2003; Parent - Child Interaction Therapy ; McNeil & Hembree - Kigin, 2001) have been demonstrated to be effective for treating disruptive behavior problems (Barkley, 1997; Pelham et al., 1998). More specifically, programs that include psychoeducation an d specific strategies for delivering and following through on clear commands, shaping behaviors through principles of reinforcement, contingency - based programs using daily behavior charts, time - in/out protocol implementation, distraction, and planned ignor ing have shown positive effects on primary outcomes such core and peripheral externalizing ADHD symptoms (e.g., hyperactivity, impulsivity, aggression, oppositionality, conduct problems), as well as secondary outcomes such as child - parent interactions, and parental stress (Anastopoulos, Shelton, DuPaul, & Guevremont, 1993; McCleary & Ridley, 1999; Patterson et al., 1993; Pisterman, McGrath, Firestone, Goodman, Webster, & Mallory, 1989; Reiff & Stein, 2011). Though parent - focused programs are effective when used in isolation, research suggests that these treatment effects may be enhanced, and include greater generalization across home, school, and peer domains, when combined with teacher and child skills training (MTA, 1999a). Despite substantial evidence for the effectiveness of parent behavior management training programs (e.g., Patterson et al., 1993), research has clearly highlighted a lack of generalization of those improved outcomes into other settings such as the classroom (Webster - Stratton & Hammond , 1997). Given this outcome, several lines of research have evaluated and 28 demonstrated the effectiveness of training teachers on strategies to improve the behavior of children with externalizing problem behaviors associated with ADHD. Many principles of be havioral therapy included often in parent training sessions have shown useful for teachers to implement within the classroom (e.g., labeled praises, reflections, behavioral descriptions, establishing instructional control, response cost, contingent attenti on; McIntosh, Rizza, & Bliss, 2000). Initial teacher training typically includes recommendations on limiting classroom distractions and providing accountability by using daily classroom incentive programs for on - task behavior, work completion, and followin g directions and also include consultation regarding the implementation and use of home/school interventions (e.g., DRC). Corkum, McKinnon, and Mullane (2005) evaluated a 10 - week behaviorally - based training program for teachers of children with ADHD sympto ms, and found that children displayed greater improvement compared with those whose teachers did not participate in the training. However, this study did not differentiate by ADHD sub - type, rather a global construct was used. The training topics in the tea cher program included psychoeducation on ADHD, effective communication, and using behavioral management techniques including ignoring, natural consequences, time - out, and token systems. Another effectiveness study used the popular Parent - Child Interaction Therapy, and adapted its components for use in the preschool classroom setting (i.e., Teacher - Child Interaction Therapy). Lyon and colleagues (2009) found that this adapted classroom intervention skills (e.g., positive attention skills, consistent discipline) which have shown to be effective in the parent version of the treatment for externalizing symptoms associated with ADHD, ODD, and CD. Most research pertaining to the effectiveness of teacher trainings for treating externalizing core and peripheral symptoms associated with ADHD has examined programs included as one component of a comprehensive 29 behavioral package (e.g., MTA, 1999a). For example, the efficacious behavioral treatment component of the MTA (1999a) study consisted of up to 24 sessions of biweekly teacher behavior management consultation occurring through face - to - face visits and telephone contacts (Wells et al., 2000). Though these types of treatments have shown to be effective, the re search literature is overwhelmingly consistent in its demonstration that maximum improvements in the generalization and maintenance of child behavioral improvements stem when teacher training is used in combination with parent training and directed child i ntervention. One particular child intervention that is commonly used and evaluated within these multimodal packages is reviewed in the next section. inappropriate res ponses to environmental stimuli through behavior modification techniques (e.g., shaping, extinction, contingency management, token reinforcement). Given the multiple impairments that children with core and peripheral externalizing symptoms of ADHD exhibit both at home and school, interventions have been developed that address behaviors in both settings. DRCs for behavior are commonly used and effective instruments for treating externalizing problem behaviors associated with ADHD, ODD, and CD ( Christ, Riley - Tillman, & Chafouleas, 2009; Evans & Youngstrom, 2006; Fabiano et al., 2010; Fabiano, Vujnovic, Naylor, Pariseau, & Robins, 2009; Pelham et al., 2000; Riley - Tillman, Chafouleas, & Briesch, 2007). Nearly four decades ago, cted an experimental evaluation of children with externalizing symptoms of ADHD (at the time referred to as Hyperkinetic Syndrome ) using a DRC intervention. Pre - and post - measures included teacher nd the Problem Behavior Rating (PBR) scale. On the PBR, despite the two groups not differing significantly at baseline, 30 significantly ( p < .005) improved scores were found in the treatment group ( p < .012) after DRC intervention. On the TRS, despite equiva lency of groups at baseline, the treated group had significantly ( p < .005) improved scores compared with the control group ( p < .066). While DRCs are effective when used in isolation (Fabiano et al., 2010), much research has looked at DRCs as a component of multimodal behavioral treatment packages (e.g., Owens et al., 2008; Pfiffner et al., 2007; Wells et al., 2000). In fact, a DRC intervention was included as part of the intensive behavioral treatment package in the MTA (1999a) study. DRCs also evidence u tility and empirical support for monitoring behavioral progress overtime and in response to varying treatment conditions (e.g., medication/no medication, etc.). Pelham and colleagues (2001) used DRCs to measure the effectiveness of behavioral versus combin ed treatment on ADHD symptoms in children attending a summer treatment program. While systematic direct observation has long been considered the gold standard for monitoring student behavior, Chafouleas and colleagues (2005) found a significant correlation between the use of a DRC with systematic direct observation on ratings of on - task and disruptive classroom behavior. Behavioral treatment: Limitations. Despite the empirical support that behavioral treatments have garnered for treating core (e.g., hyper activity, impulsivity) and peripheral (e.g., aggression, oppositionality, conduct problems) externalizing problem behaviors associated with ADHD clinic referrals, there are limitations to their use. Compared with the effects of medication, behavioral treat ments did not evidence the same rapidity or magnitude of effects on ADHD symptoms (MTA, 1999a). However, it should again be noted that several limitations were present in the design of the MTA trial which likely predisposed a favorable outcome for medicati ons in comparison to be havioral treatment . Regardless, several studies have found that the therapeutic effects of behavioral interventions are limited to the window in which the 31 treatment occurs (Pelham et al., 2000). Even the APA (2006) Task Force Report did not find behavioral treatments to be effective one year after treatment discontinuation. Finally, evidence 2007) indicate that their intensive multimodal therapies must be implemented simultaneously across settings in order to conjointly affect functional do mains given the lack of generalization that is generated when behavioral treatments are used in isolation. Behavioral treatment: Conclusions. A review of the research including multisite longitudinal studies and extant meta - analyses indicates that behavio ral interventions can reasonably be offered as first - line care for core (e.g., hyperactivity, impulsivity) and peripheral (e.g., aggression, oppositionality, conduct problems) externalizing symptoms associated with ADHD clinic referrals with effect sizes i n the moderate to large range (APA, 2006). In fact, the effects of many behavioral treatments for externalizing problem behaviors associated with ADHD are comparable to the effects obtained with low to moderate dose stimulant medication (Pelham et al., 199 8; Pelham & Waschbusch, 1999). One important distinction within the outcome data between these two approaches is the focus within medication research on reducing the core symptoms of ADHD (i.e., inattention, hyperactivity, impulsivity), while behavioral tr eatment studies have focused more extensively on improving key peripheral domains of functional impairment (e.g., aggression, oppositionality, conduct problems) which are thought to mediate secondary outcomes (e.g., parent - child/peer relationships, academi c/school functioning; APA, 2006; Fabiano et al., 2010). Because ADHD is a chronic disorder, it is important that children develop skills that can be employed over a lifetime. If left untreated, these impairments may continue to debilitate functioning in la ter childhood, adolescence, and beyond. Guidelines of the APA (2006) Task Force Report state that the decision regarding which 32 treatment to use must be guided by the balance between benefits and harms. Since there is short - term effectiveness for both me dication and behavioral treatment, the report recommends that behavioral treatment be used as first - line care due to its absence of side effects. However, given - effectiveness analysis of behavioral treatments u sed in the MTA, implementing these interventions to the intensity established in the research literature (e.g., weekly group and individual training sessions lasting for several months in the MTA) and recommended by professional organizations (AAP, 2011; A PA, 2006) is not feasible within the fast - paced and insurance driven context of most primary care settings, where youth are typically treated. While some research (e.g., McMenamy, Sheldrick, & Perrin, 2011) has examined the feasibility of incorporating com prehensive behavioral treatment programs into applied healthcare settings, given the constraints of time, personnel, and funding, it is clear that adapted forms of these treatments are needed. Combined treatment. Due to the limitations in using either me dication or behavioral treatment as a single modality, much research (e.g., Klein & Abikoff, 1997; Majewicz - Hefley & Carlson, 2007; MTA, 1999a; Pelham et al., 1988; Pelham & Waschbusch, 1999) suggests using a combination of both medication and behavioral t reatment may be most effective. An important consideration in employing combined treatment is how the intensity level of each modality (e.g., medication dosage, frequency of behavioral intervention) influences the cumulative treatment effect. Much of the l iterature (e.g., Carlson, Pelham, Milich, & Dixon, 1992; Klein & Abikoff, 1997; Pelham et al., 1988; Pelham, Milich, & Walker, 1986; Pelham, Schnedler, Bologna, & Contreras, 1980) has demonstrated the additive effects (i.e., synergistic/interactive effects ) of combining medication with behavioral treatment in yielding larger treatment gains than either treatment used alone. Further, much of this literature has evaluated the relative effectiveness of 33 c outcomes be obtained with lower intensities of combined treatments than when either is used alone? In a meta - analysis of combined treatment efficacy studies for children with differing subtypes of ADHD, Majewicz - Hefley and Carlson (2007) found large effect sizes for the core features of ADHD: inattention (1.27), hyperactivity (1.27), and impulsivity (0.91). These effect sizes for core features are higher than those reported from meta - analytic medication monotherapy efficacy studies (e.g., Conner, Fle tcher, & Swanson, 1999; Crenshaw et al., 1999; Faraone & Biederman, 2002). The MTA (1999a) study found that while teachers reported greater improvements were made in hyperactive/impulsive symptoms, parents reported greater improvements on inattentive sympt oms in children receiving the combined treatment. This study also found the effects of the combined treatment and medication alone to be comparable in treating both sub - types of ADHD symptoms. However, Swanson and colleagues (2001) found the combined treat ment in the MTA study yielded a higher percentage (68%) of participants who Though not a statistically significant difference, it is left up to individual interpret ation whether this difference is meaningful. Another study conducted by Klein and Abikoff (1997) randomly assigned 89 children to eight weeks of behavioral therapy, medication, or their combination. Behavior ratings from teachers (Conners Teache and 97%, respectively, for children receiving combined treatment; considerably higher than the 50 - 7 9% improvement rates reported for the behavioral treatment and medication groups. However, this study did not differentiate treatment response by ADHD sub - type. The behavior 34 therapy used by Klein and Abikoff (1997) involved function - based treatment plannin g for both home and school consisting of parent and teacher training on the use of operant methods (e.g., reinforcers, punishment, time out, consequences, rewards). In a case study evaluating the isolated and interactive effects between the use of common c lassroom contingencies for behavior (contingent teacher reprimands, brief time - out, no contingency) and methylphenidate at different dosage levels on disruptive and off - task behaviors in children with ADHD, Northrup and colleagues (1999) found the effects of medication are moderated by the behavioral contingency conditions in place. Again, this study combined symptoms of both sub - types together within a global ADHD construct. Numerous other studies (e.g., Blum, Mauk, McComas, & Mace, 1996; Schell, Pelham, B effects between combinations of medication and behavioral treatment. Sams, & Carlson, 1992), ha ve examined the combined effects of treatment as a function of dosage or intensity (i.e., dosage effects). The results of these studies indicate that a low dose medication and high dose behavioral treatment have approximately equivalent effects as high dos e medication and high dose behavioral treatment. With the exception of the MTA (Vitiello et al., 2001) study which showed that children receiving combined treatment ultimately received endpoint doses that were 20% less (31.1 mg/day) than those receiving me dication only (38.1 mg/day), no research was found which has examined whether high dose behavioral treatment can be effectively improved upon by adding medication or whether comparable therapeutic effects can be obtained with a lower medication dosage if b ehavioral treatment is implemented either previously or conjunctively. This is an important consideration with health and safety implications for children, families, and treatment providers who may desire to minimize 35 medication use, especially if the same therapeutic outcome can be obtained with fewer side effects. In fact, Pelham (2012) posits that physicians can effectively prescribe lower doses of stimulant medication when starting with behavioral treatment. For parents, having this treatment option avai lable may improve parental views regarding acceptability and increase the likelihood for treatment adherence, both variables in which are key considerations in evaluating treatment effectiveness. ADHD treatment: Acceptability and adherence. Seve ral researchers (e.g., Eckert & Hintze, 2000; Kazdin, 1980; Phelps, Brown, & Power, 2002; Witt & Elliot, 1985) have proposed that treatment acceptability is a significant determinant in treatment adherence . Further, treatments with high acceptability corre late with greater patient outcomes, compliance and motivation, treatment satisfaction, and lower drop - out rates (Cross - Calvert & Johnston, 1990). Given the growing trend toward third - party reimbursement for mental health care, there has been a long - standin g societal demand for evaluation of mental health services to ensure that treatments are cost - effective (Cross - Calvert & Johnson, 1990). From the perspective of health care providers, it stands to reason that for treatments to be cost - effective, they must demonstrate both high acceptability and adherence (Kazdin, 1981). The following sections review literature pertaining to parent and teacher preferences for various externalizing ADHD (including symptoms associated with aggression, ODD, and CD) treatments a nd their likelihood for implementing treatment as intended by the therapist. By delineating the specific factors that influence treatment preferences, externalizing ADHD intervention models may more readily include those components with high - acceptability and exclude those components with low - acceptability, ultimately facilitating increased adherence and effectiveness. 36 Acceptability is defined as the overall evaluation by nonprofessionals, lay persons, clients, and other potential consumers of treatment reg arding whether the treatment procedures are fair, appropriate, reasonable, unobtrusive, and in line with conventional notions about what treatment should be for a particular problem or client (Kazdin, 1980). Using treatments with a high - degree of acceptabi lity is an important consideration for health care providers given the sequential and reciprocal relationship between perceived treatment acceptability and treatment use, treatment integrity, and treatment effectiveness (Witt & Elliot, 1985). Research eval uating acceptability for externalizing ADHD treatments tends to be overlooked as this information is not included typically within treatment outcome studies (Springer & Reddy, 2004). The studies that do report this data are predominantly focused on assessi ng parental views toward treatment, treatment acceptability of teachers, as they are often tasked with the responsibility of providing intervention in the class room. If a teacher disagrees with the therapist - recommended treatment, they may evidence reluctance or difficulty in implementing treatment as intended (Eckert & Hintze, 2000; Wilson & Jennings, 1996). R esearch examining parent acceptability of behaviora l and medication treatments for externalizing ADHD has consistently found behavioral treatments as more acceptable (Cohen & Thompson, 1982; Johnston, Hommersen, & Seipp, 2008; Summers & Caplan, 1987; Wilson & Jennings, 1996). However, it is important to no te that studies evaluating parent acceptability have revealed patterns indicating that higher acceptability ratings of proposed treatments tend to be correlated positively with a greater severity of behavior problems (Gage & Wilson, 2000; Kazdin, 1980). Fo r example, Gage and Wilson (2000) found that while parents of children with ADHD rated medication as a more acceptable treatment than did parents of children without 37 ADHD, those parents of children without ADHD rated the behavioral treatments higher than p arents of children with ADHD. Another survey of parents ( N = 96) of children with ADHD (did not differentiate by subtype) found behavioral treatment alone was rated significantly higher in acceptability than behavioral treatment combined with medication (Wilson & Jennings, 1996). In fact, no parent reported medic ation alone would be their preferred treatment. Regarding the various behavioral treatments used in this study (self - management therapy, parent training, parent - child interaction training), behavior therapies involving both the child and parent were rated higher than therapies involving either the child or parent alone. In a study of mothers ( N = 109) randomly assigned to read either descriptions of behavioral parent training or stimulant medication as treatments for a child with ADHD in a case vignette, mo thers rated behavioral parent training as more acceptable than medication (Johnston, Hommersen, & Seipp, 2008). Similarly, parents in the MTA (1999a) study reported they would recommend behavioral treatment more often than medication and combined treatment . In fact, Pelham and colleagues (submitted for publication) found that 34% of parents assigned to the medication treatment in the MTA study reported being disappointed in their assignment, compared to 7% of parents assigned to behavioral treatment, and 4% of parents assigned to combined treatment. In another survey of parents ( N = 55) of children treated for ADHD, behavior therapy was rated more acceptable than medication (Krain, Kendall, & Power, 2005). This study did not distinguish whether children prese nted with predominately inattentive - type, hyperactive - impulsive - type, or combined - type symptoms. In addition to assessing parent acceptability ratings for various treatments, the Krain and colleagues (2005) study also examined the role that ethnicity plays in treatment preferences as - Caucasian 38 parents. These results are similar to another study which found that ethnic group differences may indicate preferences for tre atment as ethnic minority (e.g., African American, Latino) parents were found to rate medication treatments less positively than Caucasians (Pham, Carlson, & Kosciulek, 2009). Using the Treatment Evaluation Inventory (TEI; score range = 0 to 115; Kazdin, 1980), Gage (2002) found that parents ( n = 126) and teachers ( n = 45) differed in how acceptable they viewed various treatments (e.g., behavioral treatment, medication, combined treatment) for ADHD in children (did not differentiate by subtype). While teac hers ( M = 79.60; SD = 11.29) rated the combined treatment as more acceptable than parents ( M = 70.07; SD = 19.44), parents ( M = 84.38; SD = 14.50) rated the behavioral intervention as more acceptable than teachers ( M = 80.27; SD = 10.82). While the rationa le for these results may be difficult to determine, it is important to note, however, that both parents and teachers rated the behavioral and combined treatments as more acceptable than medication use alone. In a study assessing ratings only from teachers ( N = 147) regarding the acceptability of two behavior interventions (daily report card, response cost procedure) and medication for use in elementary and middle school students diagnosed with ADHD (did not differentiate subtype), results indicated the dai ly report card procedure was significantly more acceptable than the response cost or medication (Power, Hess, & Bennett, 1995). Further, teachers rated medication as more acceptable when used in combination with behavioral intervention. This study, along w ith Witt and Robbins (1985), evaluated teacher acceptability ratings of ADHD treatment (did not differentiate subtype) as a function of teaching experience. For example, Power and colleagues (1995) found a moderately negative relationship between elementar y and middle 39 experience and their acceptability ratings of behavior al intervention. Most teacher acceptability research on ADHD treatment is focused on the use of DRCs. Fortunately, this literature base indicates positive findings regarding acceptability due to the time - effectiveness and ease of use ( Chafouleas , Riley - Tillman, & Sassu, 2006; Waschbusch et al., 2011). Chafouleas, Riley - Tillman, and Sassu (2006) found over 60% of a teacher sample had used a home - school note program to some degree. In another study in which elementary school teachers ( N = 79) were asked to rate the acceptability of various evidence - based treatments (daily report card, medication) for use on a vignette case study of a child with ADHD (did not differentiate subtype) , the daily report card received the highest mean ratings ( M = 46.5; S D = 8.4) and was rated significantly more acceptable than all other treatments, including stimulant medication ( M = 37.78; SD = 9.53; Girio & Owens, 2009). The acceptability measure used in this study was the Intervention Rating Profile - 10 (IRP - 10; Power e t al., 1995) which includes a score range of 10 to 60). Research has demonstrated that treatment adherence has a direct influence on the effectiveness of behavioral, cognitive - behavioral, and psychoeducational interventions (Schmidt & Woolaway - Bickel, 2000 the treatment process, as well as their utilization of the treatment techniques as intended by the therapist (Springer & Reddy, 2010). While 100% treatment adherence across a populatio n of implementers will likely not be attained, much of the research literature has provided indicators for what level of treatment adherence might be expected by parents and teachers and which factors predict higher levels of adherence. Research has consistently found that acceptability of a treatment correlates positively 40 with adherence to that treatment. In fact, in exploring predictors of treatment adherence in children and families receiving therapy for disruptive behaviors, Kazdin, Holland, and C rowley (1997) found parental beliefs regarding the relevance of the treatment for addressing a particular problem evidenced the largest magnitude in effect size differences for distinguishing who completed ( ES = 1.70) the training program and who dropped o ut prematurely ( ES = 1.07). Given parental views regarding the acceptability of medication as a treatment for ADHD previously described, it is not surprising that many parents demonstrate reluctance to begin a medication trial for their children, and those that do start a trial often discontinue the treatment prematurely. In a parent survey of children with ADHD (did not differentiate by subtype) who recently initiated a medication trial, 42% of parents reported their child discontinued medicati on within one month of initiation, 33% within 2 to 3 months, 21% within 4 to 6 months, and only 4% continued the regimen longer than 6 months (Toomey et al., 2012). Parents cited medication side effects (62%) and inadequate effectiveness (34%) as primary r easons for discontinuation. This study also inquired about parental attitudes regarding ADHD medication usage, and found over half of participants reported to worry about long - term medication effects (55%) and believed ADHD medications were overprescribed (58%). In another parental attitudes study of children ( n = 48) diagnosed with ADHD from outpatient primary care and mental health clinics affiliated with a large university hospital, only 29% of parents believed medication was necessary for treatment (dos Reis et al., 2009). In addition to parent surveys, several large insurance claims analyses have been published recently which provide strong indirect indicators of parental reluctance for long - term medication use. For example, Winterstein and coll eagues (2008) found using a Medicaid 41 insurance claims analysis of over 40,000 children with ADHD (did not differentiate by subtype) that most (51%) children who received a medication prescription were not taking the medication one year later. In another st udy investigating stimulant medication utilization trends in a sample of 11,698 children prescribed with at least one stimulant, roughly one fourth (24%) of those children never filled a second prescription (Habel, Schaefer, Levine, Bhat, & Elliot, 2005). Several studies have explored demographic differences in treatment adherence. For example, in an MTA study exploring the moderating effects of ethnicity on the use of behavioral parent management training strategies for ADHD (did not differentiate by subt ype), Jones and colleagues (2010) found ethnicity to be a moderating factor for treatment adherence, though not for treatment effects. In a similar study exploring the role of treatment acceptability in the initiation of treatment for ADHD (did not differe ntiate by subtype), Krain, Kendall, and Power (2005) found that of non - Caucasian parents and that Caucasian families were more likely to pursue a recommendation for pharmacologic al treatment than non - Caucasian families. A paucity of studies exist which specifically assess teacher adherence to classroom - based treatments for ADHD. Many of the teacher - implemented intervention studies for behavior problems that exist have found that intervention implementation is poor and adherence tends to decline after the initial phases of intervention (Noell et al., 2005). However, these adherence engag ement in the intervention process (Power et al., 2009). For example, Power and colleagues (2009) found that teacher engagement (i.e., teacher support, cooperation, and involvement in the intervention process) in treatment was positively correlated with par ent ratings of the family - school relationship in two efficacious family - school interventions for ADHD (e.g., Family - 42 School Success [FSS]; Coping with ADHD through Relationships and Education [CARE]) provided within a tertiary care pediatric hospital. Given the high acceptability reported by teachers on a DRC intervention, it is no surprise that Vujnovic (2009) found in a study assessing the effectiveness of a classroom - based behavioral intervention using a DRC for students ( N = 33) with ADHD (did not differentiate by subtype), that teachers implemented adequate levels of integrity (80%) on the intervention which was found to be effective. Also, in a teacher survey ( N = 1000; Chafouleas et al., 2006) tracking the relationship between teacher - reported a cceptability and the reported usage of DRC intervention for behavior concerns, teachers indicated high levels of adherence to the intervention as it is typically designed to be implemented. Nearly half (43%) of teachers reported conducting DRC ratings on a daily basis, with almost a third (27%) reporting that they conduct ratings more than once daily (e.g., morning and afternoon ratings). Given that results of this study indicate that DRCs are used frequently already by teachers (64%), it appears that the D RC is a practical intervention for implementation in the school setting for students externalizing problem behaviors associated with ADHD. ADHD treatment: Conclusions. Many important conclusions can be gleaned from the literature with implications for how current treatment methods align with recommendations from practice guidelines of professional organizations, which treatments have empirical support, which treatments are likely to demonstrate high acceptability and adherence, and how current treatment me thods might be improved. For treating school - age children with ADHD, AAP - approved medications for ADHD (strength of evidence: strong) and/or evidence - based parent - and/or teacher - adm inistered behavior therapy as treatment for ADHD, preferably both (strength of 43 combined interventions, APA (2006) concluded that all three treatments are well - established as acute interventions. However, because only medication causes side effects, based on a risk - to - benefit analysis, the report recommended that behavioral treatment be used as first - line care, with adjunctive use of medication if needed. If children are unresponsive to behavioral treatment, the preponderance of evidence clearly supports the use of combined treatment for both effectiveness and safety, especially when considering the dosage effects reviewed herein that may allow for lower stimulant dosages when behavioral treatment is used previously and concurrently. While the AAP and APA consensus guidelines do not offer specific guidelines for treatment of externaliz ing symptoms of ADHD, given the emphasis they provide on implementing behavioral treatments in the context of a more globally - defined construct of ADHD, it stands to reason there would only be more of an impetus to deliver behavioral treatment, particularl y when treating core and peripheral externalizing symptoms associated with ADHD. This impetus is only made more pronounced when considering the behavioral parent training literature pertaining to treatment of symptoms associated with other externalizing di sorders of ODD and CD that commonly co - occur with ADHD (Patterson et al., 1993). In sum, coupled with the evidenced parental reluctance to initiate and adhere to medication long - term, it is clear that systems of ADHD care that include behavioral health tre atment options may provide a viable alternative to the current standard of medical care for this common child referral concern. However, when conducting any type of treatment response study it is important to take into consideration the potential for moder ating effects that have previously been demonstrated for age (e.g., Patterson et al., 1993), gender (e.g., MTA, 1999b), ethnicity (Arnold et al., 2003), 44 and socio - economic status (MTA, 1999b). The following sections review the literature pertaining to vari ous systems of care for the delivery of evidence - based treatment for externalizing problem behaviors associated with ADHD. Systems of Pediatric Primary Care for Externalizing ADHD Problems Most patients seeking help for mental or behavioral health problem s are seen in the context of community - based primary care (Reiger et al., 1993), with estimates suggesting as many as 50 to 70% of all patients seen in primary care have referral concerns pertaining to their mental or behavioral health (Belar, 2008; Gatche l & Oordt, 2003). Specifically , ADHD is one of the most common behavioral health condition for which patients seek treatment ( Leslie, Rappo, Abelson, Jenkins, & Sewell, 2000), with more than 75% of these office visits occurring within primary care (Zarin e t al., 1998). Given the high comorbidity with other disruptive behavior disorders such as ODD and CD (Barkley, 2006), core and peripheral externalizing behavioral problems associated with ADHD (e.g., hyperactivity, impulsivity, aggression, oppositionality, conduct problems) represent an increasingly common referral concern which requires considerable attention to facilitate early detection and intervention within an evidence - based biopsychosocial framework. Due to healthcare reimbursement practice s that emphasize the PCPs role in treating not only medical, but also mental and behavioral health concerns, and that deemphasize the practice of referring out to specialty providers, the primary care sector has traditionally been referred to treated ADHD and other externalizing symptoms associated with the disorder, much literature has questioned whether they possess the training to address these referrals con cerns competently. 45 2010) and their training in child and adolescent mental health and behavioral interventions (e.g., Serby, Schmeidler, & Smith, 2002). Given t he prevalence of externalizing ADHD symptoms in child populations, the lack of specialty service providers available in many communities (Leslie, Stallone, Weckerly, McDaniel, & Monn, 2006) and the demonstrated difficulties in obtaining HMO - reimbursement f or external referrals, it is no surprise that the standard of care in pediatric primary care falls short of the treatment guidelines put forth by AAP (2011) and APA (2006). The following sections review the literature highlighting the shortcomings of SMC f or externalizing ADHD symptom treatment and the literature pertaining to the use of IBH models as alternatives to the current standard of care. Standard medical care for ADHD. Researchers (e.g., Hoagwood, Jensen, Fe il, Vitiello, & Bhatara, 2000; MTA, 1999a) have acknowledged that SMC for externalizing ADHD symptom treatment includes physician overreliance on medication, often as the single form of therapy. If medication usage is not indicated due to contraindications or patient preferences, then physicians typically refer out to a specialty provider. When services are referred out it may be more costly for patients as HMOs may be reluctant to pay for external providers and have typically built in incentives for physic - specialty mental health providers to refer out to as the shortage of child and adolescent psychiatrists, particularly in poor, urban or rural settings, results in long appointmen t wait - times (i.e., 3 to 6 months; Kim, 2003). Also, roughly only a quarter of patients follow through with be seen more quickly in the context of primary care whic h has led to the majority of stimulant medications being prescribed by PCPs (e.g., family physicians; pediatricians), and not psychiatrists. Research investigating the clinical care of externalizing ADHD symptoms 46 indicates that this practice standard does specifying treatment goals, using evidenced - age and developmental level, and providing systematic follow - up using parent and teacher rating scales for quanti tatively assessing treatment response. Further, PCPs, particularly family physicians, report a lack of adherence to AAP guidelines for evaluation of ADHD (Rushton, Fant, & Clark, 2004), thus contributing to a potential overuse of medication in populations which may not even meet diagnostic criteria for the disorder. In a study evaluating the importance of adherence to AAP (2001) guidelines for physician treatment practices, Epstein and colleagues (2007) assessed outcomes of 377 children with ADHD bein g treated by either (1) pediatricians randomly assigned to a group receiving training on AAP (2001) guidelines through a consultation service promoting the use of titration trials and periodic monitoring during medication maintenance, or (2) pediatricians randomly demonstrated significant behavioral improvement compared with the treatment as usual group and were more likely to be receiving medication at 12 months (73 vs. 41%). However, this study did not evaluate ADHD behavioral improvement using separate subscales for inattentive and externalizing behaviors, but rather combined all ADHD symptoms into a global construct. The MTA (1999a) study included, in addition to a group receiving carefully prescribed and titrated medication (i.e., MTA - medicated group), a group that received medication through community care (i.e., treatment as usual). Patients receiving treatment as usual received less frequent and shorter off ice visits with their providers compared with patients in the MTA - medicated group. Consequently, community - treated patients were less likely to receive medication titration trials to determine optimum therapeutic dosage. Despite both groups 47 receiving the s ame medication (i.e., methylphenidate), the prescribing practices were considerably different, which significantly affected treatment outcomes. Though medication management is the dominant treatment modality for ADHD within community - based settings (Hoagwo treatment practices are not commensurate with treatment as it is intended to be provided within evidence - based guidelines (e.g., AAP, 2011). Not only does it appe ar that SMC for ADHD often fails to establish the lowest therapeutic dosage through the use of systematic titration algorithms, but also that physicians rarely conduct follow - up evaluations of treatment response using home and school data. In a feasibility that teacher rating data were rarely collected and did not appear to be valued by physicians for stimulant titration purposes (Pliszka et al., 2003). This practice clearly do es not adhere to Recommendation 5 of AAP (2011) treatment guidelines stating that clinicians should establish a plan for periodic monitoring of treatment response through teacher data (e.g., teacher periodic behavior ratings/narratives, telephone conversat ions) obtained through active and direct communication with the school. The MTA (1999a) study clearly showed the value of teacher information as a supplement to parent information in stimulant titration (Greenhill et al., 2001; Vitiello et al., 2001). This et al., 2007; MTA, 1999a), for children with ADHD referral concerns in community - based practice falls short of consensus guidelines (AAP, 2011; APA, 2006). Finally, from an HMO and adm inistrative perspective, despite the increased prevalence of pediatric primary care appointments which include behavioral health clinic referrals (Belar, 2008), research has questioned the long term financial sustainability of models which require 48 pediatri cians to deliver comprehensive care for non - medical concerns. In a study evaluating appointment length and HMO reimbursement, Meadows, Valleley, Haack, Thorson, and Evans (2011) concluded that financial disincentives are placed on p ediatricians when addressing behavioral health concerns. Specifically, this study found that while behavioral health issues minutes for behavioral health appoi ntments) and number of codes billed (1 code for behavioral health vs. up to 10 codes for medical visits), providers are inadequately reimbursed for these services. Addressing this lack of behavioral health service reimbursement, Feldman, Lee, and Perez - Sta - contracting this type of care to managed behavioral health organizations. Consequently, adequate pediatrician within these behavioral health plans. SMC: Conclusions. While SMC provides service for many children that may otherwise not receive any form of treatment, the model calls into question whether the long - term benefits outweigh the risks involv ed with receiving stimulant medication as monotherapy. This risk - to - benefit dilemma is made more precarious when patients are not provided with other treatment no side and adverse effect risk in relation to expected treatment gains. If families opt for medication use, they must still be presented with other evidence - ba sed behavioral treatment options and the empirical evidence supportive of each. However, without healthcare providers with training in delivering these behavioral interventions and with knowledge of their effectiveness as well as 49 current HMO disincentives health concerns, it appears that SMC will continue to be stimulant medication monotherapy. It is clear that new treatment models that adhere to AAP (2011) and APA (2006) guidelines and im prove the quality of care received by children with externalizing behavioral problems associated with ADHD referrals concerns in primary care are needed. Inte grated behavioral health for ADHD . acknowledge that a critical design flaw in the current healthcare system is the lack of behavioral health services in medical settings. With recent systems - level healthcare reforms (e.g., Patient Protection and Affordable Care Act [PPACA; P. L. 111 - 148] and the evolution of the Family - Centered Medical Home, there has been a push for the physician - directed integration of mental and behavioral health services within primary care. Integrated primary care (IPC) or, more specificall y, integrated behavioral health (IBH) models may offer an efficient means for interdisciplinary patient care involving physicians and behavioral health providers (i.e., BHPs; e.g., psychologists) that may potentiate numerous patient benefits (APA, 2013; Ga tchel & Oordt, 2003; Strosahl, 2005). The integration of behavioral health services with medical care allows for increased treatment options, safer treatment options in comparison with psychotropic medication, more effective delivery of evidence - based trea tments with BHPs trained specifically in these behavioral therapies, and increased treatment adherence as the need for external referrals are - 2012). While in tegrated care as a delivery model has been used in various forms since the 1960s when Kaiser Permanente experimented with this service approach in their clinics, numerous models of integrated care have been developed and implemented more recently ( Blount, 1998; 50 Gatchel & Oordt, 2003; Strosahl, 2001). These models differ in the degree and method to which providers collaborate among each other, interact with patients, and delegate clinical care decisio n - making responsibilities . Typically, the physician mainta ins primary responsibility for treatment decision - making and uses BHPs within a consultative or short - term treatment role. The BHP, in the context of an IBH model, receives patient referrals from the PCP when specialty mental or behavioral health services are indicated and, in turn, provides brief evaluation and targeted interventions that are carried out over mu ltiple sessions by the BHP . The focus of clinical care within IBH is problem - focused, solution - oriented, and family - centered treatment, whereby patients receive evidence - based skills training needed to address behavioral health concerns over a relatively rapid course of care of only a few appointments ( Bryan, Morrow, & Appolonio, 2009). These IBH models offer increased coordination of patient car e, which in turn have demonstrated to improve patient and provider satisfaction, adherence to treatment regimens, and clinical outcomes for many patients and problem types (Blount, 2003; Katon et al., 1995). While most systematic evaluations of IBH models stem from the adult treatment literature, these findings provide direction for efforts to integrate care for behavioral health concerns within pediatric primary care settings. As further support for integrating behavioral health services into primary care practice, several studies examining the relative cost - benefit of IBH models have demonstrated their cost - effectiveness in treating a variety of conditions such as depression (Katon et al., 1995; Lave, Frank, Schulberg, & Kamlet, 1998, Pyne, Rost, Zha ng, Williams, Smith, & Fortney, 2003; Schoenbaum et al., 2001; Simon et al., 2001) and anxiety (Katon, Roy - Bryne, Russo, & Cowley, 2002; Chaffee, 2009). For example, in a meta - analytic review of 91 published medical cost - offset studies evaluating the impac t of psychological interventions (e.g., psychiatric consultation, 51 brief psychotherapy, biofeedback/relaxation training, behavioral health psychoeducation) within medical services, Chiles, Lambert, and Hatch (1999) found cost - savings of about 20% when behav ioral treatments were used. While a paucity of studies included in this meta - analysis targeted pediatric practice settings specifically, those that did indicated that cost outcomes were comparable to treatments targeting adult populations. Also, while seve ral studies have examined the relative cost - effectiveness of distinct ADHD treatments (i.e., behavioral vs. medication vs. combined therapy; e.g., Foster et al., 2007; Gilmore & Milne, 2001; Jensen et al., 2005) using a payer perspective (i.e., accounting for all direct billing costs regardless of whether they were paid for by a patient, an insurer, or any other third party and not accounting for any discounts and/or negotiated fees), no studies have been found which examine the financial cost - benefit deriv ed from IBH pediatric primary care models for externalizing problem behaviors associated with ADHD clinic referrals. IBH: Conclusions. It appears that the IBH model whereby behavioral health services ar e integrated within primary care practice provide patients with more treatment options that abide by consensus guidelines of AAP (2011) and APA (2006). Further, IBH models have clearly demonstrated to provide a financial cost - benefit compared to SMC, altho ugh a paucity of research exists regarding the treatment of externalizing problem behaviors associated with ADHD clinic referrals in pediatric primary care specifically. Given the need for translating efficacious and highly - acceptable treatment models desc - based settings in a format that is cost - and time - effective for treatment providers, IBH service - delivery appears to provide the appropriate forum to do so. 52 Systems of care: Conclusions. While much research has evaluated the efficacy of behavioral treatment, medication, and combined approaches in carefully controlled settings, there is a paucity of research evaluating the effectiveness of thes e treatments in real clinic settings, particularly community - based pediatric primary care. In these settings, those participants receiving therapy more closely resemble actual patients who possess clinical diagnoses with co - morbid conditions, pay out of po cket or through HMO - reimbursement for treatment, do not participate solely for research participation incentives, and receive treatment from clinicians who maintain full clinical caseloads. These types of studies are important for obtaining outcome data th at is generalizable to actual clinical practice. Therefore, an important consideration in evaluating empirically - supported multimodal treatments (e.g., MTA, 1999a, pediatric primary care when considering their accessibility, feasibility, and potential for financial viability. The issue of scalability is particularly important for systems of care for ADHD and associated externalizing behavioral problems given the volu me of children affected. While SMC of medication monotherapy offers the advantage of access to large populations of children given its short term cost - and time - effectiveness, there is little research on IBH treatments that are effective, efficient, and de signed and implemented with consideration for patient caretaker acceptability and adherence ratings. These considerations are critical to the development of cost - effective practice models which are amenable to larger healthcare reform initiatives emphasizi ng increased integration and coordination among services and providers within the context of the family - centered medical home. 53 Research Questions and Hypotheses Research question 1. Based on parent and teacher report, does IBH treatment (i.e., Michigan Algorithm) provide greater symptom score improvement in externalizing behaviors (e.g., hyperactivity, impulsivity, aggression, oppositionality, conduct problems) associated with ADHD referral concerns in comparison with SMC when controlling for the demographic covariates of age, gender, ethnicity, and insurance type? When used in isolation, both behavioral treatment and medication have demonstrated short - term eff ectiveness in treating the core symptoms of ADHD ( DuPaul & Eckert, 1997; Fabiano et al., 2009; MTA, 1999a; Pelham, Wheeler, & Chronis, 1998). However, extensive literature (e. g., Klein & Abikoff, 1997; MTA, 1999a; Pelham & Waschbusch, 1999; Pelham et al., 1988) has demonstrated that combined approaches using both behavioral intervention and medication are most effective, particularly when focusing on peripheral externalizing symptoms such aggression, oppositionality, and conduct problems (Pelham et al., 20 01). Due to medication side effect concerns, APA (2006) recommends that ADHD treatment consist of a combined intervention approach, but with behavioral treatment used as first - line care with adjunctive medication, if needed. Several treatment programs for children with externalizing problem behaviors associated with ADHD referral concerns (e.g., Chronis et al., 2004; Pelham, 2007) evaluating the relative effectiveness of behavioral treatment, medication, or their combined use have found that most children c an be treated effectively and maintain those gains both at school and home with only the use of multimodal behavioral treatment consisting of parent training, teacher consultation, and child intervention (e.g., DRC). Given that the IBH treatment used in th e present study (i.e., Michigan Algorithm) includes the core behavioral components (e.g., parent training, teacher consultation, DRC with contingency - based incentives) used by Chronis and 54 colleagues (2004) and the Buffalo Algorithm described by Pelham (200 7), it is hypothesized that children receiving IBH treatment of either behavioral treatment alone or behavioral treatment combined with medication (if behavioral treatment alone does not yield symptom improvement after 6 weeks/2 - 3 clinic sessions) will display improved symptom outcomes compared with children who receive SMC as measured by an Externalizing/ADHD symptom severity scale (BASC - 12) based on parent and teacher ra tings when controlling for key demographic covariates. The demographic covariates of age, gender, ethnicity, and insurance - status are controlled for in this research questions as previous literature had identified these variables as being potential moderat ors of treatment response. Also, by nature of the procedure used to analyze these results, pretreatment (baseline) scores will also be controlled for. Research question 2. How do parents and teachers involved in the implementation of the IBH treatment, spe cifically the DRC for behavior component, rate its degree of acceptability? Do parents in the IBH treatment group rate their services as significantly more acceptable when compared to parents in the SMC treatment group when controlling for the demographic covariates of age, gender, ethnicity, and insurance type across the two groups? Parents consistently rate behavioral treatment as more acceptable than medication for ADHD (Cohen & Thompson, 1982; Krain, Kendall, & Power, 2005; Wilson & Jennings, 1996). Reg arding medication use as a treatment for ADHD concerns, research has demonstrated parental reluctance to initiate medication treatment (e.g., Liu, Robin, Brenner, & Eastman, 1991) and adhere to medication treatment regimens (dosReis et al., 2009; Habel, Sc haefer, Levine, Bhat, & Elliot, 2005; Sherman & Hertzig, 1991; Toomey et al., 2012; Winterstein et al., 2008). Given these results, it is hypothesized that parents will rate the IBH treatment as highly - 55 acceptable (i.e., group mean of 55 or higher on Treatm ent Acceptability Survey; Kratochwill et al., 2003). It is also hypothesized that parents will rate the IBH treatment as significantly more acceptable when compared to ratings from parents who were in the SMC group, given prior literature suggesting parent al preferences for behavioral treatments over SMC (MTA, 1999a). Regarding teacher acceptability ratings of the IBH treatment, and more specifically their experiences with the consultation they receive on the implementation and use of the DRC intervention, numerous studies have demonstrated that teachers view the use of the DRC as a highly - acceptable treatment for ADHD ( Chafouleas, Riley - Tillman, & Sassu, 2006; Waschbusch et al., 2011 ). Therefore, it is hypothesized that teachers will rate the IBH treatment as highly - acceptable (i.e., group mean of 55 or higher on Treatment Acceptability Survey; Kratochwill et al., 2003). It is noted that teacher ratings were not obtained from the SMC group. Research question 3. Do parents and teachers involved in the implem entation of the DRC component of the IBH treatment evidence high levels of adherence (based on pre - specified number of intervention components implemented successfully)? Are there group differences in adherence between parents and teachers? Which treatment s are provided as part of SMC and Research literature has consistently found relatively high levels of acceptability for behavioral ADHD treatments including behaviorally - based parent training (e.g., Johnston et al., 2008) and teacher - implemented DRC intervention (e.g., Chafouleas et al., 2006). Given the positive correlations found between treatment acceptability and treatment adherence (Kazdin et al., 1997), it is hypothesized that bo th parents and teachers involved in the implementation of the IBH treatment in the present study will evidence a high degree of treatment adherence (i.e., 56 also hyp othesized that there will not be significant group differences in adherence between parents and teachers given the hypothesized expectation for both groups to evidence high levels of adherence. Regarding which treatments are provided as part of SMC, over t wo - thirds (67.4%) of patients treated within the community - care group as part of the MTA (1999a) study received psychotropic medication (principally the stimulant class). Similarly, it is hypothesized in the present study that two - thirds or more of childre n treated within the SMC group will receive psychotropic medication as their physician - recommended treatment. Also, given previous research showing that 33% of families opt to discontinue medication treatment for ADHD within the first 2 to 3 months of init iation (Toomey et al., 2012), it is hypothesized that one - third or more of patients receiving psychotropic medication as part of SMC will not be adherent to this - treatment initiation). Research que stion 4. (most typical) patient within each of the treatment models (IBH v. SMC) based on procedural billing codes used over the three month course of care tracked in the study? It is hypot hesized that IBH treatment will evidence a higher cumulative financial cost over the course of care tracked in the present study compared to SMC treatment based on the modal patient in the study. The use of mode as a measure of central tendency was used ba sed on a preliminary analysis of data showing that the modal patient attended the recommended number of clinic sessions (at least four) of the Michigan Algorithm. Using the mean or median patient would have provided data that was not reflective of the actu al cost of care for those patients who used the treatment model as intended. Specifically, the hypothesis for research question #4 was based on the characteristics of the treatment models used in the present study which includes apparent differences pertai ning to initial diagnostic evaluation and post - treatment follow - up. 57 Specifically, the Michigan Algorithm uses an evaluation component which adheres to guidelines of the AAP (2011) consisting of multimethod, multisource, and multisetting data. This practice stands in contrast with existing literature demonstrating reluctance on the part of PCPs to include cross - setting and cross - informant data in the evaluation phase as well as the lack of follow - up care as part of SMC (e.g., Pliszka et al., 2003). It is ant icipated that these additional procedures will be accounted for within procedural billing codes evidenced within the first three months of care that are tracked in the study. 58 CHAPTER 3 : METHOD Participants Participants were 136 parents or legal guardi ans and teachers of child and adolescent patients ( N = 73) aged 6 through 17 years. Child and adolescent patients presented with both core (e.g., hyperactivity, impulsivity) and peripheral (e.g., aggression, oppositionality, conduct problems) externalizing problem behaviors associated with ADHD referral concerns in 1 of 2 university - hospital affiliated and community - based pediatric primary care practices in the Southeast Michigan area between the six month period of 09/01/13 and 02/01/14. Patients who were previously diagnosed with a disruptive behavior disorder (e.g., ADHD, ODD) as indicated in their electronic medical record were excluded from the study. Sixty - nine participants (35 parents; 34 teachers) completed survey information pertaining to child pati ents ( n = 35) receiving treatment within a clinic using an integrated behavioral health (IBH) model of care for externalizing ADHD concerns (i.e., Michigan Algorithm), while 67 participants (38 parents; 29 teachers) completed information pertaining to chil d patients ( n = 38) receiving treatment within a clinic using a standard medical care (SMC) model (i.e., treatment as usual). In the IBH group, 71% ( n = 25) of patients were children (ages 6 to 12) while 29% ( n = 10) were adolescents (ages 13 to 17) and the mean age was 10.4 years; 60% (n = 21) were male; 94% ( n = 33) were Caucasian while 3% ( n = 1) were Hispanic American, 3% ( n = 1) were Asian American, and none were African American; and 57% ( n = 20) were comm ercially insured while 43% ( n = 15) received Medicaid. In the SMC group, 63% ( n = 24) were children while 37% ( n = 14) were adolescents and the mean age was 11.1 years; 71% (n = 27) were male; 91% ( n = 35) were Caucasian while 3% ( n = 1) were Hispanic Amer ican, 3% ( n = 1) were Asian American, 59 and 3% ( n = 1) were African American; and 87% ( n = 33) were commercially insured while 13% ( n = 5) received Medicaid . Preliminary analyses were performed to determine whether the distribution of age, gender, e thnicity, and insurance - status were significantly different in the IBH and SMC groups. Chi - square tests were used for the categorical variables, and t - tests were used for continuous variables. The mean ages of participants did not significantly differ in t he IBH and SMC groups, t (70.92) = - 1.17, p = .25. The proportion of Caucasian and non - Caucasian participants did not 2 = 68, p = 1. The proportion of male and female participants did not significantly differ 2 (1) = 0.56, p = .45. The proportion of participants with commercial insurance or Medicaid differed significantly 2 (1) = 6.65, p = .01. The percentage of participants who had Medicaid was higher in the IBH group (43%) than in the SMC group (13%). Overall, demographic data of the cities/communities in which these clinics are located were remarkably similar in regard to town population (IBH clinic = 9,489; SMC clinic = 7,444), median household income (IBH clinic = $43, 958); SMC clinic = $47,668), and racial/ethnic make - up of citizens (IBH clinic = 94.8% While/Caucasian; SMC Clinic = 96% While/Caucasian) according to recent U.S. Census data. Though insurance - status was the only demographic characteristic that differ ed significantly between the treatment groups, the effect of all the demographic characteristics on the T score trajectories was controlled in the study. Study Interventions IBH i ntervention (Michigan Algorithm). Based on a review of empirically supported treatments for and adapted from established treatment 60 guidelines (AAP, 2011; APA, 2006) and algorithms (e.g., Pelham, 2007), the Michigan Integrated Behavioral Health ADHD Treatment Algorithm used in the present study is descr ibed in the following sections. The context in which the Michigan Algorithm was implemented was a community - based, university hospital - affiliated pediatric primary care practice consisting of five pediatricians in the southeast region of the state of Michigan. The co - located service - delivery model occurred within a time - limited context of 4 to 6 ± cli nic sessions of 50 minutes each with periodic follow - up if needed. The multimodal behavioral treatment consisted of three components: (1) behavioral parent training, (2) behavioral teacher consultation, and (3) child intervention within the home and school via a daily report card. Patients were seen by the behavioral health provider, a licensed psychologist, after being referred by an attending or resident physician for ADHD or other externalizing behavioral concerns. In the Michigan Algorithm, referrals fo r externalizing Initial behavioral treatment facilitated by the BHP in the first session was relati vely simple and tied directly to the domain of functional impairment (e.g., parent training, teacher consultation, child intervention). Additional treatment was applied on an as - needed basis, based on progress monitoring results, by intensifying the existi ng treatments (e.g., providing training and/or consultation on additional behavior management strategies). If after 4 to 6 weeks, the child remained unresponsive to treatment, they were referred back to their PCP for a medication evaluation. If medication was prescribed, participants continued to receive the behavioral package within a combined approach. The benefit of continued usage of the DRC component in conjunction with medication was its dual utility as a progress monitoring tool in assessing 61 behavior al response to changes in medication through targeted daily ratings. Parent training consisted of instruction on behaviorally - based parenting skills to address noncompliant behavior, avoid conflict cycles, recognize the function (e.g., escape from avers ive task; gain parental attention) of behavior, and provide fair and consistent rules. Specific strategies consisted of developing a time - in/out protocol (Barkley, 2006); using deferred time - out when children are unresponsive to traditional time - out (Warza k & Floress, 2009); compliance training; task - based grounding; providing contingencies effectively; and giving effective commands. Training also addressed aspects of the parent - child relationship and incorporates components of Parent - Child Interaction Ther apy (PCIT; McNeil & Hembree - Kigin, 2011), such as nurturance within the child - directed interaction stage, and limit - setting within the parent - directed interaction stage. This two - stage model provides reinforcement for appropriate behavior and non - attention - based consequences for inappropriate behavior . These trainings typically begin during the intake session and occur throughout duration of treatment. Parents are provided one or two new strategies to implement at home for 2 to 3 weeks and are instructed to schedule a follow - up session to discuss progress, troubleshoot, and/or add new strategies as needed. Teachers received consultation focused on behaviorally - based classroom management skills and directions on the implementation of the DRC. Teacher consu ltations were conducted by four clinical assistants (two predoctoral interns in clinical and school psychology, and two predoctoral externs in clinical and school psychology). Phone consults occurred on an as - needed basis if a child was unresponsive to tre atment or if questions regarding the DRC arose. Consultation typically addressed developing operationalized, objective, and measureable target goals; establishing a predictable environment; providing feedback and consequences consistently; praising appropr iate behaviors and ignoring mildly inappropriate behaviors that are 62 not reinforced by attention; maintaining a 6 to 1 positive to negative comment ratio; providing effective accommodations that are individualized to need; developing data collection protoco ls; providing immediate recognition for goal attainment; and communicating daily outcomes to parents. A DRC was implemented to modify child behavior by setting daily behavior goals, tracking progress toward those goals, and linking school performance to h ome - based contingencies. Examples of target behaviors that have been monitored by DRCs include school - based functional outcomes such as work completion, following directions, aggressive behavior, and on - /off - task behavior (e.g., Dougherty & Dougherty, 1977 ; Jurbergs, Palcic, & Kelley, 2007; McCain & Kelley, 1993; Seay, Fee, & Holloway, 2003). Goals were developed collaboratively by teachers and parents with consultative support from the BHP and/or trained clinical assistants. From the literature review, fiv e critical components of DRCs were apparent and incorporated into the intervention used in the present study: (1) operational definition of target behavior goals; (2) praising the child for efforts to achieve those goals; (3) daily teacher ratings of the c progress via the DRC; (5) providing home - based contingencies for meeting/not meeting target behavior goals. These components facilitated the use of conti ngency management , an empirically - supported method of treating externalizing behaviors associated with ADHD (DuPaul & Eckert, 1997). If a classroom teacher already had a DRC protocol in place, the BHP and/or clinical assistants worked with parents and the teacher to modify or adapt the existing program if necessary, and provide training on the use of home - based contingencies for meeting daily behavior goals. 63 SMC intervention (Treatment as u sual). provided was a community - based, university hospital - affiliated pediatric primary care practice consisting of seven pediatricians in the southeast region of the state of Michigan. PCPs in the present study provided their treatment as usual without influence from the BHP or researchers. An End of Study Treatment Follow - Up Form (Appendix N) was mailed to parents in the SMC group within the Week 12 mailer to determine which specific interventions were provided as part of standard medical care. This data is discussed in the results section . Outcome Measures Externalizing ADHD Symptom Severity S urvey. The Behavior Assessment System for Children, Second Edition, Progress Monitor, Externalizing and ADHD Problems Form, Child and Adolescent Version (BASC - 2 ADHD PM; Reynolds & Kamphaus, 2009) was completed by parents (Appendix H) and teachers (Appendix I) to measure the severity of externalizing ADHD symptoms. This instrument was selected given its focus on both core (e.g., hyperactivity, impulsivity) and peripheral (e.g., aggre ssion, conduct problems) externalizing symptoms associated with ADHD clinic referrals. Given that the majority of ADHD symptom scales yield total symptom composite scores, the use of this instrument helps address the paucity of treatment literature focusin g exclusively on externalizing symptoms of ADHD. The instrument was also selected due to its ability for frequent repeated administrations (i.e., daily, weekly, or monthly basis) to measure behavioral response to treatment change over time. There are separ ate forms for parents and teachers, each of which consist of 20 - items that use a 4 - point frequency scale (N, never ; S, sometimes ; O, often ; and A, almost always ). Scores are entered into the BASC - 2 Progress Monitor ASSIST version 1.0 scoring software that converts raw scores to clinical T scores ( M = 50; SD = 10). T scores of 70 and above are considered to be in the Clinically 64 Significant range, and T scores of 60 to 69 are considered to be in the At - Risk range. The instrument is a well - established measure that has good discriminative construct validity necessary to detect valid changes in externalizing ADHD symptoms over time, thus reducing the & Conners, 2010). Also , it has high estimates for internal consistency (coefficient alpha = .91 parent; .95 teacher); test - retest (.88 parent; .91 teacher); and inter - rater (.83 parent; .72 teacher) reliability. Regarding validity, the instrument is strongly correlated ( r = .93 parent; .94 teacher) with the BASC - 2 broadband measure. Treatment Acceptability S urvey. The Treatment Evaluation Questionnaire - Acceptability Scale was used to evaluate acceptability (i.e., overall evaluation of the effectiveness, appropriateness, and sat isfaction with the procedures used) of the treatments used in the study. All parent participants in the study completed a shortened version of the acceptability scale on the Treatment Evaluation Questionnaire, parent version (TEQ - P; Kratochwill, Elliott, L oitz, Sladeczak, & Carlson, 2003; Appendix J) as part of the Week 6 post - treatment assessment. To assess treatment acceptability from teachers in the IBH group, the TEQ - T was used (Appendix K). Both the TEQ - P and TEQ - T were originally adapted from the Trea tment Evaluation Inventory (TEI; Kazdin, 1980), a 16 - item, 7 - point Likert - scale. Despite being an unpublished measure, the TEI has traditionally been used as a measure of acceptability for evaluating a variety of behavioral interventions and medications fo r children. Items on the TEI were selected based on a factor analysis obtained from ratings pertaining to the appropriateness of various treatment options for externalizing behaviors. Regarding internal consistency, Spirrison, Noland, and Savoie (1992) fou nd ratings for six treatments for child 65 - P/T used in the present study consist of 11 statements asking raters the degree to which they agree using a 6 - point Likert - strongly Possible scores on the TEQ - P/T acceptability scales range from 11 to 66. Historically, a score of 55 or higher has been indicative of high treatment acceptability (Kratochwill et al., 2003). Only teachers of children in the IBH group received a survey to complete in order to keep teachers of children in the SMC group blind to treatment status which may have involved medication. Treatment Adherence S urvey. Parent and teacher treatment adherence measures include separate forms for the IBH and SMC group. Teachers in SMC group did not complete the Treatment Adherence Survey. Because parents and teachers in the IBH group implement behavior management recommendat ions (e.g., DRC), these responsibilities are inquired about through treatment adherence forms focused on DRC implementation (see Appendix L and Appendix M). These forms were developed by the researchers and include a checklist of intervention components th at can be used to indirectly assess the degree to which intervention procedures were implemented as intended. Treatment adherence was calculated by obtaining the percentage ratio of the number of intervention components implemented correctly out of the tot al For parents in the SMC group, an end - of - study survey (see Appendix N) inquiring both qualitatively and quantitatively about treatment received, date initiated, and current use of treatment provided the researchers with information regarding which interventions constituted 66 Procedures All procedures detailed in the following sections were approved by the Institutional Review Boards at the Un iversity of Michigan Medical School and Michigan State University. Data collection. A waiver of informed consent to research was obtained allowing researchers to examine web - based medical records in each of the two clinics to determine who was seen as a n ew patient for an initial intake appointment for externalizing behavior problems associated with ADHD referral concerns. Next, those potential participants, without exclusionary criteria, were mailed a postal packet containing a Parent Informed Consent to Research Form, an Authorization for Release and Exchange of Records with the School Form, and an Externalizing/ADHD Symptom participants received in the mailer, as well as all subsequent mailers, a 10 dollar bill as a noncontingent participation incentive. This dollar amount was selected based on similar survey studies (e.g., Doody et al., 2003; Ulrich et al., 2005) within clinic populations that yielded high surv ey return rates. If they elected to participate, they were instructed to sign and return the Informed Consent to Research and the Release of Records Forms to the researchers via Self Addressed Stamped Envelope (SASE) provided within the mailer. They were a lso instructed, pending agreement to participate in the study, to complete and return the Symptom Severity Survey. When mailers were returned to the researchers, the Symptom Severity Survey was separated from the packet and assigned a code number so that n ames were not attached to the form. Also, if parents returned a completed Release of Records Form, a mailer was sent to the Severity Survey, and a 10 dollar bill. If they e lected to participate, they were instructed to sign 67 and return the Informed Consent Form. They were also instructed, pending agreement to participate in the study, to complete and return the Symptom Severity Survey. When mailers were returned to the resear chers, the Symptom Severity Survey was separated from the packet and assigned a code number so that names were not attached to the form. All participants who agreed to participate in the study by returning a signed Informed Consent Form and a completed Symptom Severity Survey were also mailed a follow - up Symptom Severity Survey at Weeks 1, 6, and 12 post - treatment. Within the Week 6 mailer, parents and teachers in the IBH group and parents in the SMC group were also mailed the Treatment Acceptability For m. Teachers in the SMC group were not mailed the Treatment status, unlike the teachers in the IBH group who were expected to play an active part in their s treatment via their implementation of the daily report card (DRC) intervention. Also, within the Week 6 mailer, parents and teachers in the IBH group received the Treatment Adherence Form, which inquired about their integrity in delivering the DRC interv ention. The Week 6 data collection period was selected to specifically evaluate clinical response, treatment adherence and acceptability before medication was possibly introduced into the treatment regimen for those in the IBH group. Teachers in the SMC gr oup, again, were not asked to complete a Treatment Adherence Form, due to the assumption that they were blind to their Adherence Form, they were asked to complete as part of the Week 12 mailer an End of Study Treatment Form which inquired about all treatments received by their pediatrician over the course of care and whether they were still be implemented at Week 12. Upon return, all forms were separated from the maile rs and assigned a code number so that no names were attached to 68 the form. Also, medical records were reviewed to determine the CPT codes that were billed over the course of care for ADHD referral concerns included in the research study. Data analysis. Research question 1 asked whether the IBH treatment reduced participants' externalizing ADHD symptoms more than SMC. Multilevel modeling (MLM) was used to answer this question. Conceptually, MLM of the repeated measures or longitudinal data in this study c an be viewed as a two - step procedure (Singer & Willett, 2003). Although multilevel models are not estimated in a two - step process, describing them in this way helps with understanding how multilevel models represent response to treatment. In the first step , a curve is fitted for each participant that describes the change in their symptoms from pre - treatment to 12 weeks post - treatment. The ratings from all four measurement occasions (i.e., Weeks 0, 1, 6, 12) are used to form these curves. These curves can be linear or non - linear and are characterized mathematically by slopes, which describe the rate of change in symptoms over time for each participant. In the second step of the MLM procedure, factors that explain differences between participants in the rate o f change in their symptoms can be investigated. One factor that is hypothesized to explain significant differences between participants in their slopes is treatment type (i.e., IBH v. SMC). Specifically, participants receiving the IBH treatment are expecte d to experience a greater rate of receiving SMC. In sum, MLM characterizes treatment response as a slope. This conceptualization of treatment response is st ressed because it is different from the way repeated measures analysis of variance (RM - ANOVA) characterizes treatment response, which has been a traditional method used to analyze longitudinal data. RM - ANOVA conceptualizes treatment response as mean 69 differ ences between time points (e.g., the average difference between T scores from pre - treatment to 12 weeks post - treatment). Modeling treatment response as a slope instead of a difference between two means is better in the context of this study because four da ta points are used to derive the slope, making it a more reliable indicator of treatment response ( Singer & Willet, 2003) . Treatment response was measured using behavioral ratings from two sources, parents and teachers. Separate multilevel models were developed for the T scores based on parent and teacher ratings. The reason for modeling these data separately is that the concordance between parent and teacher ratings of children's behavior is relatively poor, thus indicting that differences in rat ings in part reflect true differences in their behavior across settings (Malhi, Singhi, & Sidhu, 2008). Therefore, the parent and teacher ratings are considered to measure different constructs and as a result are analyzed independently in this study. Also, see the Missing Data section for the process used in the present study to handle missing data within each of the groups. Due to the quasi - experimental design of the study and the fact that participants were not randomly assigned to the treatments, it i s possible that the participants in the IBH and SMC groups differed in characteristics that influenced their externalizing ADHD symptomatology. These characteristics are referred to as confounding variables . Therefore, even if the IBH treatment reduced ext ernalizing ADHD symptoms more than SMC, differences in confounding variables could account for this effect instead of differences in the treatments. Models developed in this study controlled for the effects of four major demographic variables (age, gender, ethnicity, and insurance - status) on pre - treatment symptomology and treatment response before evaluating the effect of the treatments. Even if the effects of these confounding variables were not statistically significant, they were retained in the model. T he reason for retaining these 70 effects is that even if the effect of a variable on an outcome is not significant, it still may be meaningful when considering the analogous contrast between p - values and effect sizes ( Cohen, 1988) . In addition to the four dem ographic variables reviewed above, steps were taken initially to control for the effect of symptom severity. Specifically, in the model building procedures, we tested whether the IBH and SMC groups had significantly different intercepts (Week 0 T scores). These tests indicated that Week 0 T scores did not significantly differ between the groups, removing the need to control for pre - treatment status. Based on initial data exploration and visualization, the form of the decline of externalizing ADHD symptoms over time appeared to be non - linear. The T scores generally decline over time. However, the rate of this decline appears to decrease over time. For example, the decline in T scores between pre - treatment and Week 1 post - treatment is generally greater than t he decline in T scores between Weeks 1 to 6 post - treatment. Therefore, the T scores appear to decline rapidly initially and then this decline tapers off. One type of function that can describe this pattern of change is a second - order polynomial (Singer & Willett, 2003). As a result, the full multilevel model used to describe the data was: Y ij = 0 + 1 ( Week ij ) + 2 ( Week 2 ij ) + 3 ( Treatment i ) + 4 ( Treatment i )( Week ij ) + 5 ( Treatment i )( Week 2 ij ) + 6 ( Insurance i ) + 7 ( Insurance i )( Week ij ) + 8 ( Insurance i )( Week 2 ij ) + 9 ( Age i ) + 10 ( Age i )( Week ij ) + 11 ( Age i )( Week 2 ij ) + 12 ( Gender i ) + 13 ( Gender i )( Week ij ) + 14 ( Gender i )( Week 2 ij ) + 15 ( Ethnicity i ) + 16 ( Ethnicity i )( Week ij ) + 17 ( Ethnicity i )( Week 2 ij ) + u 0 i + u 1 i ( Week i ) + u 2 i ( Week 2 i ) + j This equation models the non - linear trajectories of participants' externalizing ADHD symptom T scores from Weeks 0 to 12. Additionally, it models how these trajectories differ between participants receiving IBH treatment and SMC while controlling for the e ffect of 71 demographic variables (age, gender, ethnicity and insurance - status) on the trajectories. This model was estimated separately for the T scores based on parent and teacher ratings. Y ij represents the T score for participant i ( i = 1 - 73) at measurement occasion j ( j = 1 - 4), which is the dependent variable predicted by the model. 1 and 2 represent the effect of time (i.e., number of weeks passed since treatment initiation) on the T scores when the values of the other variables in represented the average or most common value of the variable. As a result, 1 and 2 represent the effect of time on the T scores for participants who had the following characteristics: 10 years and 8 months of age, male, white, commercially - insured, and received SMC. Participants with The effect of time on T scores is modeled as a second - order polynom ial function which has two components, a linear component, 1 ( Week ij ), and a quadratic component, 2 ( Week 2 ij ). Modeling the effect of time on T scores in this way allowed the T score trajectories to be non - linear. The parameter for the linear component, 1 , represents the instantaneous or starting rate of change in T scores (Singer & Willett, 2003). It was hypothesized that 1 would be negative and significant because the treatments were expected to reduce T scores. The quadratic parameter, 2 , represents t he rate of acceleration or deceleration in the instantaneous rate of change in T scores. If 2 is negative and significant, it indicates that the simple slope is decreasing over time (Singer & Willett, 2003). As a result, it was hypothesized that 2 would be negative and significant because based on the pattern of change in T scores over time, the rate of decline in T scores appears to decelerate. The 4 and the 5 parameters represent the effect of the IBH treatment on the linear component and quadratic component of the T scores trajectories respectively. It was 72 hypothesized that 4 would be negative and significant, thus indicating that the instantaneous rate of d ecline in T scores was greater for participants in the IBH treatment. It was hypothesized that 5 would be positive and significant, indicating that the deceleration of the instantaneous rate of decline in T scores was lower for participants in the IBH tre atment. The direction and significance of these parameters is of substantive interest as they answer research question 1. However, because participants were not randomly assigned to treatment groups, differences in demographic characteristics of the partic ipants could account for differences in the T score trajectories between the treatment groups. To address this problem, the effects of measured demographic characteristics (age, gender, ethnicity, insurance status) on the linear and quadratic components of participants' T score trajectories were included in the model to control for them. The parameters 7 , 8 , 10 , 11 , 13 , 14 , 16 , and 17 represent these effects. Models of change trajectories are normally described by an intercept and slope, and only t he slope (i.e., the effect of time) has been described so far. The intercept represents the starting points of the trajectories. In this case, the starting points of the trajectories are the T scores at Week 0. 0 represents the pre - treatment T score for t 3 represents the difference in the pre - treatment T scores between participants in the IBH and SMC treatment groups. It was hoped that this parameter was non - significant, indicating that pre - treatment T scores were comparable in the IBH and SMC groups. The parameters 6 , 9 , 12 , and 15 represent differences in the pre - treatment T scores based on the demographic variables measured in this study. One of the requirements of general linear models is that the values of the dependent variables are independent. Each value of the dependent variable must come from a separate source. However, the data in this study violate this assumption because externalizing ADHD 73 symptomology is measured on multiple occasions for the same participant. In other words, the T scores across measurement occasions are clustered or nested within participants. One consequence of nested data is that the model errors are correlated within persons (i.e., autocorrelation), and the variance of these errors may differ across measurement occasions (i.e., heteroscedasticity), violating assumptions of general linear models. Multilevel models deal with error autocorrelation and heteroscedasticity by including what are called random effects (Singer & Willet, 2003). The parameters u 0 j , u 1 j , and u 2 j capture the autocorrelation and heteroscedasticity in the errors of the model. Conceptually, random effects can be interpreted in the following manner. Factors that could explain interindividual variability in participants T score trajectories have been identified: age, gender, ethnicity, and insurance - status. The effects of these factors in the model are referred to as fixed effects . However, these factors would not be expected to completely exp lain interindividual variability in the T score trajectories. For example, two participants who shared the same characteristics measured in this study would still not be expected to have the exact same T score trajectory because of unmeasured factors that were not included in the model. Random effects model the interindividual variability in T score trajectories that originates from sources beyond the factors included in the model. In other words, they allow each participant to have a unique T score traject ory. Specifically, the parameter u 0 j allows each participant to have a unique pre - treatment T score, the parameter u 1 j allows each participant to have a unique linear component to their T score trajectory, and the parameter u 2 j allows each participant to h ave a unique quadratic component to their T score trajectory For normal regression models, the statistical significance of multiple parameters in a single model can be tested simultaneously. However, for multilevel regression models, the 74 degrees of freedom for the t - tests examining the significance of the parameters are not well - defined (Mirman, 2014). Therefore, the statistical significance of the variables in the full multilevel model described previously cannot be tested in the typical way. An alternative method for testing the significance of multilevel model parameters is to start with the simplest model and add a parameter to the model successively. If adding a parameter to a model significantly improves the overall fit of the model, then the parameter can be interpreted as statistically significant. For normal regression models, R 2 indicates the overall fit of the model. The statistics used to indicate the overall fit of the multilevel regression models are discussed in the model esti mation subsection. The statistical significance of 4 and 5 , the parameters that represented the effect of the IBH treatment on the T score trajectories, was of substantive interest. However, the statistical significance of other fixed effects also informed the interpretation of the model. For example, the statistical significance of 2 would indicate whether a second - order polynomial b etter described the T score trajectories than a linear pattern. This model building process tested the statistical significance of more than just 4 and 5 . . Each model constructed in the model building process is labeled (e.g., unconditional means model ) and the models are referred to by these labels. This model building process was performed for the T scores based on parent and teacher ratings separately. The model building procedure was characterized by two stages. The main purpose of the first stage w as to determine the form of T score trajectories and also if they were better described by a linear or second - order polynomial function. Models in this first stage are referred to as unconditional change models because the shape of the trajectories does no t depend on participant - level characteristics (O'Connell et al., 2013). The main purpose of the second stage was to determine the effect of participant - level characteristics on the intercepts and slopes of the trajectories, which is why 75 models in this stag e are referred to as conditional change models. Some of these participant - level characteristics (age, gender, ethnicity, and insurance status) were not of substantial interest and were only included in the model to control for their effect. As a result, th e parameters representing the effects of these participant - level characteristics are subsequently referred to as nuisance parameters . The effects of these nuisance parameters were added to the model before adding the parameters of substantial interest to t he model (the effects of treatment on the trajectories). In this way, differences in T score trajectories between treatment groups were only tested after accounting for the effect of demographic characteristics on the trajectories. The baseline model was the unconditional means model. The unconditional means model notably does not include an effect of week on T scores. Therefore, this model implies that T scores do not change over time. The predicted T score for the participants is their mean score across the four measurement occasions, which is represented by the intercept included in the unconditional means model (O'Connell et al., 2013). Unconditional change model A added a linear effect of week on T scores, allowing T scores to change linearly over tim e. If unconditional change model A fit better than the unconditional means model, then this result indicated that the linear change in T scores over time was statistically significant. Unconditional change model B added a quadratic effect of week on T scor es, allowing T scores to change in a non - linear way over time. If unconditional change model B fit better than unconditional change model A, then this result suggested that a second - order polynomial function that is non - linear better described the T score trajectories than a linear function (O'Connell et al., 2013). If unconditional change model B did not fit better than unconditional change model A, then the T score trajectories were subsequently modeled as linear. Conditional change models A throug h C tested the effect of the nuisance variables (i.e., 76 demographic characteristics) on the T score trajectories. Conditional change model A added the effect of the nuisance variables on the intercept of the trajectories. All of the effects of the nuisance variables on the intercept were entered simultaneously. If conditional change model A fit better than unconditional change model B, then this result implied that pre - treatment T scores differed significantly based on demographic characteristics. The reason for entering multiple nuisance parameters in a single block is that their statistical significance was not of substantial interest. If conditional change model A did fit better than unconditional change model B, post - hoc testing was performed to determine which demographic characteristics significantly affected pre - treatment T scores. This post - hoc testing consisted of adding the nuisance variables one at a time to unconditional change model B to determine which nuisance variables significantly affected pr e - treatment T scores. Conditional change model B added the effect of the nuisance variables on the linear component of the T score trajectories. If conditional change model B fit better than conditional change model A, then this result indicated that the l inear component of the trajectories differed significantly based on demographic characteristics. Conditional change model C added the effect of the nuisance variables on the quadratic component of the trajectories. If conditional change model C fit better than conditional change model B, then this result suggested that the quadratic component of the trajectories differed significantly based on demographic characteristics. Conditional change models D through F tested the effect of treatment on the T score trajectories after the effect of the nuisance variables had been controlled. Conditional change model D added the effect of treatment on the intercept of the trajectories. If conditional change model D fit better than unconditional change model C, th en this result implied that pre - treatment T scores differed significantly between the IBH and SMC treatment groups. Conditional change 77 model E added the effect of the treatment on the linear component of the trajectories. If conditional change model E fit better than conditional change model D, then this result indicated that the instantaneous rate of change in T scores differed significantly between the IBH and SMC groups. Conditional change model F added the effect of treatment on the quadratic component of the trajectories. If conditional change model F fit better than conditional change model E, then this result suggested that the acceleration or deceleration of the instantaneous rate of change in T scores differed significantly between the IBH and SMC g roups. Even if the parameters 4 and the 5 , which represent the effects of the IBH treatment on the T score trajectories, are statistically significant, this result does not necessarily indicate that these effects are meaningful. To assess the meaningfulness of the effects, Cohen's d was used. Cohen's d is an effect size metric that reflects the standardi zed mean difference on a measure between two groups. The standardized mean difference in T scores at 12 weeks post - treatment between the IBH and SMC groups was calculated as an indicator of effect size. To calculate Cohen's d , the following equation was us ed: (12( 1 + 4 ) + 12 2 ( 2 + 5 1 + 12 2 2 ) b The 12( 1 + 4 ) + 12 2 ( 2 + 5 ) component of the equation represents the model - predicted difference in T scores between Weeks 0 and 12 for participants in the IBH group. The 12 1 + 12 2 2 component of the equation represents the model - predicted difference in T scores between Weeks 0 and 12 for participants in the SMC group. The second component was subtracted from the first component to estimate the raw mean difference in model - predicted T scores at 12 weeks post - treatment between the IBH and SMC groups. To standardize this raw mean difference, it was divided by the standard deviation of T scores between persons, b . 78 The standard deviation of T scores between participants was estimated u sing information from the unconditional means model. The unconditional means model partitions variance in T scores into two components: the variance in T scores between persons and the variance in T scores within persons (Singer & Willett, 2003). The squar e root of the variance in T scores between participants was used as the estimate of the standard deviation of T scores between participants. 5 was omitted from the equation if it was not statistically significant because it would indicate that the IBH and SMC groups did not significantly differ in the quadratic component of their trajectories, and 2 was omitted from the equation if it was not s tatistically significant because it would indicate that a second - order polynomial did not better describe the trajectories than a linear pattern. As a starting point for interpreting Cohen's d , guidelines developed by Cohen (1998) proposed that d = 0.20 re presented a small, d = 0.50 a medium, and d = 0.80 a large effect. Full information maximum likelihood (FIML) was used to estimate the models. One advantage of using FIML estimation with multilevel models of longitudinal change is that it does not omit ca ses with missing data at any time point, unlike RM - ANOVA (Singer & Willett, 2003). Another advantage of using FIML estimation is that the fit of models that are nested in their effects can be compared. Two models are nested in their effects if one model co ntains all the effects of the other model and at least one more effect. Because the models in the model building procedure were nested in their effects, FIML estimation allowed for judgment of whether adding an effect to a model significantly improved its fit. Models estimated by FIML produce the deviance statistic, which is an indicator of model fit (Singer & Willett, 2003). Smaller deviance values indicate better fitting models. The difference in deviance statistics between two nested models is distri buted as a 2 distribution with 79 degrees of freedom equal to the difference in the number of parameters between the models. If the probability of the difference in deviance statistics between nested models was less than 0.05, then the model with the additio nal parameter(s) was interpreted as fitting better than the original model. Models estimated by FIML also produce the Akaike Information Criteria (AIC) and the Bayesian Information Criterion (BIC), which are additional indicators of the fit of a model (Sin ger & Willett, 2003). Smaller AIC and BIC values indicate better fitting models. Relative to the deviance statistic, the advantage of the AIC and BIC is that they account for model complexity and/or sample size. Adding variables to a model always decreases the deviance statistic, albeit this decrease may not be significant. Because the AIC and BIC account for model adding trivial variables to a model increases AI C and BIC, indicating worse model fit. The disadvantage of the AIC and BIC is that the statistical significance of the difference in these fit statistics between models cannot be tested. Therefore, the deviance statistic was used as the primary indicator o f model fit and the AIC and BIC statistics were used as supplementary indicators of model fit. Research question 2 asked whether parents rated the IBH treatment as more acceptable than SMC. A regression model that controlled for the effects of demographic variables on treatment acceptability was used to answer this question. The dependent variable in the model was the acceptability rating. The independent variables in the model that predicted acceptability ratings were treatment and the four nuisance demog raphic variables (age, gender, ethnicity, insurance - status). The equation representing this model was: Y i = 0 + 1 ( Treatment i ) + 2 ( Insurance i ) + 3 ( Age i ) + 4 ( Gender i ) + 5 ( Ethnicity i ) + ij 80 1 represents the raw difference in the mean acceptability ratings between participants in the IBH and SMC groups when the value of the other variables is 0. Because the other variables were dummy coded or centered in a way that 0 represented the average or most common value of the variable as discussed previously, 1 can be interpreted as the raw difference in mean 1 would indicate that IBH ratings were higher than SMC ratings. Therefore, the statistical significance of this parameter was of substantial interest because it directly addresses research question 2. To assess whether this raw difference in ratings was meaningful, Cohen's d was calculated by dividing 1 pooled standard deviation of the parent acceptability ratings. Research question 3 explor ed how parents and teachers adhered to the IBH treatment and whether there were group differences between them. Adherence only to the DRC was evaluated due to the idiographic nature of dosing within the IBH algorithm which is tailored to unique child, fami ly, and school needs. A dependent samples t - test was performed to determine whether parents and teachers demonstrated different levels of adherence to implementing the treatment. The demographic covariates of age, gender, ethnicity, and insurance - status we re controlled. Treatment adherence ratings were also analyzed using descriptive statistics. Mean parent and teacher scores on the Treatment Adherence Form are reported using a ratio of the percentage of intervention tasks completed out of the eight possi ble intervention tasks. It was determined that 80% (at least 6 of 8) treatment integrity to the possible intervention tasks is considered are reasonably ab le achieve, yet yield effective results on child behavior. These pilot study 81 results were obtained by tracking parent and teacher usage of the DRC within the six months prior to the present study. Also, for parents in the SMC group receiving treatment as u sual by their pediatrician, data are reported from ratings on the End of Study Survey. This data reports what specific treatments were facilitated by their pediatrician, including both behavioral and psychopharmacological, and also which of those treatment s were still being implemented after the 12 week course of treatment evaluated within the present study. Financial cost of care for treatment provid ed within each group was assessed using the total amount of commercial reimbursement based on 2014 CPT billing codes for the State of Michigan (excluding Detroit) indicated over the course of the study (Weeks 0 to 12). The American Medical Association (AMA) Code Manager websit e (see https://ocm.ama - assn.org/OCM/CPTRelativeValueSearch.do ) was used. Medicaid billing rates are also available on this website for a subscription cost. Specifically, descriptive sta tistics were used to report the number of clinic sessions attended, type of procedural billing codes used, and cumulative financial cost of care for the total duration of 12 weeks of treatment evaluated within the study and also a per session cost for the modal or most typical participant presenting within each treatment group. It was determined that the modal participant in the study would provide the best indicator of what the Michigan Algorithm would cost from a payer perspective for those that attend in the range of all recommended sessions. Conversely, it was found from a preliminary analysis of the cost of care data that the cumulative financial cost of care for the mean participant within each treatment group did not reflect participants who actuall y received the treatments as they were intended, particularly the Michigan Algorithm consisting of 4 to 6 ± clinic sessions, due to a minority of participants (N = 14) not returning to clinic after one or two sessions which skewed 82 the results. While presen ting data for the median participant would have provided a more accurate portrayal of a typical participant who received the Michigan Algorithm as intended, it was determined that there were slight variations in the use of procedural billing codes (e.g., c ode type, units used, minutes in session) and that the codes used for the median participant was not representative of the majority of participants in this treatment group. Data was fully present on all the demographic variables measured in the study giv en that this data was collected via online medical records. Data was also completely present for parent and teacher ratings at pre - treatment given that this data was a requisite for subsequent study participation. Data was missing for parent and teacher sy mptom ratings at Weeks 1, 6, and 12, treatment acceptability ratings, and treatment adherence ratings. In general, the method of treating missing data depends on the missing data mechanism. Therefore, the missing data mechanism needs to be determined. The missing data mechanism describes the process that caused data to become missing. There are three types of missing data mechanisms: missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR). MCAR means that the probabil ity of missing data on a dependent variable does not depend on the value of the dependent variable itself or other measured variables (Enders, 2010). For example, if parents did not submit an ADHD symptom rating scale because their child displayed very hig h ADHD symptoms, then this process would not be consistent with a MCAR mechanism because the likelihood of missing a parent rating depends on the unreported value of the T score. In the less restrictive MAR mechanism, the probability of missing data on a d ependent variable can depend on other measured variables in the model, but it still should not depend on the value of the dependent variable itself (Enders, 2010). For example, if parents of low socioeconomic status (SES; i.e., Medicaid - insured in the pres ent study) are less likely to 83 submit an ADHD rating scale, and SES is included in the model, then this process would be consistent with a MAR mechanism. MNAR means that the probability of missing data on a dependent variable depends on the value of the dep endent variable itself, even after controlling for other variables (Enders, 2010). For example, if parents did not submit an ADHD symptom rating scale because their child displayed very high ADHD symptoms, then this process is consistent with a MNAR mechan ism. Missing data mechanisms cannot be determined conclusively, and the determination of a missing data mechanism needs to be justified theoretically. Because the parents and teachers of the participants were informed that the confidentiality of thei r children's data would be protected, it is not believed that parents would not submit a rating scale because their children displayed very high externalizing ADHD symptoms. Parents and teachers were offered an incentive (ten dollar bill) to submit the rat ing scales, so even if their child was not responding to the treatment, it is credible that this incentive motivated parents and teachers to continue submit rating scales regardless of treatment response. Parents of high SES may have not submitted rating s cales because the incentive was not sufficient, but an indicator of SES (insurance - type) was included in the model, making a MAR mechanism plausible. Little's (1988) MCAR test is a statistical method for examining the assumption that the data are MC AR. Specifically, Little's MCAR test investigates whether the probability of missing data on a variable depends on other variables in the data set. Although it cannot conclusively verify that data are MCAR, it can offer evidence that MCAR is a reasonable a ssumption. Little's MCAR test suggested that the data in this study were MCAR ( 2 (329) = 341.58, p = .31). Multilevel models estimated using full information maximum likelihood (FIML) produce unbiased parameter estimates for missing data when it is MCAR o r MAR (O'Connell et al., 84 2013). Because MCAR or MAR was plausible for the ADHD rating scale data, the analyses in research question 1 handled missing data by using FIML estimation which uses all available data. For the regression models in research questio n 2, missing data was handled by using listwise deletion. Listwise deletion produces accurate parameter estimates for missing data when it is MCAR (Enders, 2010). Because MCAR was plausible, the analyses in research question 2 handled missing data by using listwise deletion. 85 CHAPTER 4 : RESULTS Research q uestion 1 Parent ratings. The first step in the model comparison procedure was to determine whether T scores changed over time, regardless of differences in treatment type or demographic covariates. Unconditional change model A fit significantly better than the unconditional means model, 2 (3) = 67.42, p < .001. This result indicated that T scores did cha nge significantly over time. In unconditional change model A, the parameter for the variable representing the number of weeks that the treatment had been implemented was negative, indicating that T scores declined over time. However, unconditional change m odel A was limited in that it described this decline as linear. Unconditional change model B modeled this decline as a non - linear second - order polynomial function. Unconditional change model B fit significantly better than unconditional change model A, 2 ( 4) = 21.49, p < .001. This result suggested that a second - order polynomial function better described the change in T scores over time than a linear function. In unconditional change model B, the parameter for the linear week variable was positive and the p arameter for the quadratic week variable was negative, thus indicating that T scores declined over time, but the rate of this decline slowed over time. The next set of model comparisons investigated whether demographic variables (i.e., age, gender, ethnic ity, insurance - status) significantly affected pre - treatment T scores or the T score trajectories over time. Conditional change model A did not fit significantly better than unconditional change model B, 2 (4) = 5.64, p = .23. This result implied that the demographic variables did not significantly affect pre - treatment T scores. Conditional change model C did not fit significantly better than conditional change model B, 2 (4) = 3.38, p = .5. This result indicated 86 that dem ographic variables did not significantly affect the linear component of the T score trajectories. Conditional change model D did not fit significantly better than conditional change model C, 2 (4) = 0.11, p = 1. This result suggested that demographic varia bles did not significantly affect the quadratic component of the T score trajectories. In sum, this set of model comparisons indicated that demographic variables did not significantly affect pre - treatment T scores or the T score trajectories. Given this st - experimental design with lack of treatment group randomization, these results are reassuring in that even if the treatment groups truly differed on these demographic variables, these differences would not fully explain differences in demonstrat ed outcomes between the treatment groups. The last set of model comparisons tested whether treatment type significantly affected pre - treatment T scores or the T score trajectories over time, after the effects of the nuisance variables (i.e., age, gender, ethnicity, and insurance - status) had been controlled. Conditional change model D did not fit significantly better than conditional change model C, 2 (1) = 0.93, p = .34. This result implied that IBH and SMC groups did not have significantly different pre - t reatment T scores, offering further evidence that the IBH and SMC treatment groups were similar at pre - treatment. However, it should be noted that though the effect of treatment on the intercept ( 3 ) added in conditional change model D was not significant, it was retained in conditional change model E, thus essentially controlling for differences in starting point scores between groups. The reason for retaining this effect is that conditional change model E includes the interaction between the treatment and week variables. To model the interaction or multiplicative term between these two variables in the model, their associated parameters needed to be included in the model independently. Conditional change model E fit significantly better than conditional ch ange model D, 2 (1) = 10.6, p = .001. This result indicated that participants in 87 the IBH and SMC groups differed significantly in the linear component of their T score trajectories. The parameter for the variable representing the interaction between time a nd treatment was negative, indicating that the instantaneous rate of decline in T scores was significantly faster for the IBH than SMC group. Conditional change model F did not fit significantly better than conditional change model E, 2 (1) = 3.12, p = .08 . This result suggested that participants in the IBH and SMC groups did not differ significantly in the quadratic component of their T score trajectories. Because conditional change model F did not fit significantly better than conditional change model E, conditional change model E is regarded as the best model representing the T scores trajectories based on the parent data. For this reason, the parameter estimates for conditi onal change model E are reported . In sum, while the SMC group experienced a significantly faster rate of symptom reduction from Week 0 to Week 1, the rate of decline in T scores over the 12 week course of care evaluated in the study was clearly in favor of the IBH group. Specifically, mean T score ratings at Week 12 for the SMC group fell in the At - risk range of scores, while ratings for the IBH group fell in the Average range. To assess whether the overall difference in the instantaneous rate of decline in T scores between the IBH and SM C groups produced a meaningful or practical difference in T scores at 12 weeks post - treatment, the standardized mean difference (Cohen's d ) of the model - implied T scores between the IBH and SMC groups at 12 weeks post - treatment was computed. The value of t his mean difference was 0.36, which exceeds the criteria for a small effect size based on guidelines developed by Cohen (1998). This effect size means that at 12 weeks post - treatment, the average participant in the study would be expected to have a lower T score in the IBH treatment than SMC by about three tenths of a standard deviation. 88 T he average participant in the IBH group would be expected to start at a T score of roughly 63, which corresponds to a percentile rank of 90, and end at a T score of rough ly 56, which corresponds to a percentile rank of 73. In contrast, the average participant in the SMC group would be expected to start at a T score of roughly 62, which corresponds to a percentile rank of 88, and end at a T score or roughly 59, which corres ponds to a percentile rank of 82. Teacher ratings. The first step in the model comparison procedure was to determine whether T scores changed over time, regardless of differences in treatment type or demographic characteristics. Unconditional change model A fit significantly better than the unconditional means model, 2 (3) = 55.83, p < .001. This result indicated that T scores did change significantly over time. In unconditional change model A, the parameter for the variable representing the number of week s that the treatment had been implemented was negative, indicating that T scores declined over time. However, unconditional change model A was limited in that it described this decline as linear. Unconditional change model B modeled this decline as a non - l inear second - order polynomial function. Unconditional change model B fit significantly better than unconditional change model A, 2 (4) = 17.15, p = .002. This result suggested that a second - order polynomial function better described the change in T scores over time than a linear function. In unconditional change model B, the parameter for the linear week variable was positive and the parameter for the quadratic week variable was negative. This result indicated that T scores declined over time but the rate o f this decline slowed over time. The next set of model comparisons investigated whether demographic variables significantly affected pre - treatment T scores or the T score trajectories over time. Conditional change model A did fit significantly better than unconditional change model B, 2 (4) = 16.72, p = .002. This result implied that the demographic variables did significantly affect pre - treatment T 89 scores. Post - hoc testing indicated that gender significantly affected pre - treatment scores, 2 ( 1) = 7.69, p = .006. The parameter estimate for the gender variable was - 4.24, indicating that females' pre - treatment scores were roughly four T score points lower than those of males. The other demographic variables did not significantly affect pre - treatm ent scores. Conditional change model C did not fit significantly better than conditional change model B, 2 (4) = 1.3, p = .86. This result indicated that demographic variables did not significantly affect the linear component of the T score trajectories. Conditional change model D did not fit significantly better than conditional change model C, 2 (4) = 5.03, p = .28. This result suggested that demographic variables did not significantly affect the quadratic component of the T score trajectories. In sum, t his batch of model comparisons indicated that only gender affected pre - treatment scores, but age, ethnicity, and insurance - status did not significantly affect pre - treatment T scores. None of the demographic variables significantly affected the T score traj ectories. Though gender significantly affected pre - treatment T scores, the preliminary analyses indicated that the proportion of males and females in the IBH and SMC treatment groups did not differ. Therefore, the effect of gender on pre - treatment T scores was not a concern. The last set of model comparisons tested whether treatment type significantly affected pre - treatment T scores or the T score trajectories over time, after the effects of the nuisance variables had been controlled. Conditional cha nge model D did not fit significantly better than conditional change model C, 2 (1) = 1.45, p = .34. This result implied that the IBH and SMC groups did not have significantly different pre - treatment T scores, offering evidence that the IBH and SMC groups were comparable at pre - treatment. Conditional change model E fit significantly better than conditional change model D, 2 (1) = 21.88, p < .001. This result indicated that participants in the IBH and SMC groups differed significantly in the linear componen t of their T 90 score trajectories. The parameter for the variable representing the interaction between time and treatment was negative, indicating that the instantaneous rate of decline in T scores was significantly faster for participants in the IBH group t han the SMC group. Conditional change model F did not fit significantly better than conditional change model E, 2 (1) = 0.95, p = .33. This result suggested that participants in the IBH and SMC groups did not significantly differ in the quadratic component of their T score trajectories. Because conditional change model F did not fit significantly better than conditional change model E, conditional change model E was regarded as the best model representing the T score trajectories based on the teacher data. Therefore, the parameter estimates for conditional c hange model E are reported. . In sum, the pattern of change in the T score trajectories based on teacher ratings was similar to the pattern of change in the T score trajectories based on parent ratings. T he rate of decline in T scores over the 12 week course of the study was significantly faster for participants in the IBH group. In fact, mean T scores ratings fell from the At - risk range at Week 0 to the Average range at Week 12 for the IBH group. However, as was the case in the parent data, while the IBH group did experience a significantly faster rate of decline in T scores over the course of the study, it is also important to note that symptom reduction from Week 0 to Week 1 was clearly in favo r of the S MC group . To assess whether this difference in the rate of decline in T scores between the IBH and SMC groups produced a practical difference in the model - implied T scores at Week 12, the standardized mean difference (Cohen's d ) of T scores between the IBH and SMC groups at 12 weeks post - treatment was computed. The value of this mean difference was 0.43, which exceeds the criteria for a small effect size based on guidelines developed by Cohen (1998). This effect size means that at 12 weeks post - treatment, the average participant in this study would be 91 expected to have a lower T score in the IBH treatment than the SMC treatment by about four tenths of a standard deviation. T he average participant in the IBH group would be expect ed to start at a T score of roughly 60, which corresponds to a percentile rank of 84, and end at a T score or roughly 55, which corresponds to a percentile rank of 69. In contrast, the average participant in the SMC group would be expected to start at a T score of roughly 58, which corresponds to a percentile rank of 79, and end at a T score or roughly 57, which corresponds to a percentile rank of 76. Research q uestion 2 A cceptability ratings did not differ significantly depending on participants' age, gen der, ethnicity, and insurance status ( p = 0.211 - 0.781). Treatment was the only factor that significantly affected acceptability ratings, b = 15.01, t (53) = 7, p < .001. Because the parameter estimate of the treatment factor was positive, it indicated that parents of the average participant rated the IBH treatment as 15.01 points more acceptable than SMC. When this raw difference is standardized ( d = 1.39), it indicates that this difference is large. The group mean for parents in the IBH group on the 11 - it em/6 - point likert - scale (total scale range = 11 to 66) Treatment Acceptability Survey was 56.3 ( SD = 8.9; Range = 48 to 66) and nearly three quarters (71%; n = 20 of 28) of these participants rated the IBH treatment as having a high degree of acceptability (i.e., score of 55 or higher). The group mean for teachers in the IBH group was 59.20 ( SD = 11.4; Range = 35 to 66) and 82% ( n = 22 of 27) of these participants rated the IBH treatment as having a high degree of acceptability. The group mean for parents i n the SMC group was 44.20 ( SD = 12.10; Range = 26 to 59) and only 28% ( n = 8 of 29) of these participants rated the SMC treatment as having a high degree of acceptability (scores greater than 55). 92 Research q uestion 3 Parents and teachers in the IBH group did not demonstrate different levels of treatment implementation adherence, t (22) = - 0.45, p = 0.66. Surveys indicated that 88% (23 of 27) of ce to the DRC component of the model. Parents on average successfully implemented 6.88 out of the 8 total (86%) treatment components. Teachers on average successfully implemented 7.12 out of the 8 total (89%) treatment components. As part of the explorat ory analysis component of this research question, it also examined what pediatrician - facilitated treatments were delivered as part of SMC in the present study through the End of Study Survey included within the Week 12 mailer for parents. Results of the 28 participants who completed the survey indicated that 25 (89%) received medication, four of medication, 23 received stimulant formulations (e.g., amphetamine, methylphe nidate) compared to two who received non - stimulants (e.g., guanfacine, atomoxetine). Regarding the delivery of and adherence to behavioral treatments in the SMC group, the following results were found: parenting skills training ( n = 5 received; n = 2 imple mented at end); family therapy ( n = 3 received; n = 0 implemented at end); social skills training ( n = 1 received; n = 0 implemented at end); individual counseling ( n = 4 received; n = 2 implemented at end); teacher training/consultation ( n = 5 received; n = 2 implemented at end); home - school intervention ( n = 4 received; n = 2 implemented at end). Research q uestion 4 The cumulative financial cost of care for the mean participant within each treatment group was $493.86 ( M = 2.4 clinic sessions) for IBH and $221.06 ( M = 1.9 clinic sessions) for 93 SMC. However, it was determined that those values did not reflect participants who actually received the treatments as they were intended, particularly the Michigan Algorithm consist ing of 4 to 6 ± clinic sessions, due to a small number of participants not returning to clinic after one or two sessions which skewed the results. While presenting data for the median participant would have provided a more accurate portrayal of a typical p articipant who received the Michigan Algorithm as intended, it was determined that there were slight variations in the use of procedural billing codes (e.g., code type, units used, minutes in session) and that the codes used for the median participant was not representative of the majority of participants in this treatment group. Therefore, it was determined that the modal participant in the study would provide the audience a better indicator of what the Michigan Algorithm would cost from a payer perspectiv e for those that attend in the range of all recommended sessions. The data indicated that the modal participant in the study did indeed attend at least four sessions. Using a modal or typical patient within each group, results indicated a cumulative c ost of care of $784.02 for IBH and $232.53 for SMC. For the IBH group, the modal number of sessions that occurred over the course of the study with the BHP was four ( M = 3.7; Range = 1 to 5), compared to three ( M = 2.9; Range = 1 to 4) for the SMC group. A mode of five ( M = 4.2; Range = 1 to 7) procedural codes were billed for each patient in the IBH group compared to three ( M = 2.8; Range = 1 to 4) for the SMC group. Accounting for the difference in number of clinic sessions, the mean cost for each session was $196.05 for the IBH group compared to $77.51 for the SMC group. Though this is a sizeable cost difference, it is apparent that the discrepancy is attributed to 1) the four hours of billing for assessment derived from carrying out a comprehensive ADHD evaluation, which included school observations, that adheres to AAP 94 guidelines and 2) longer appointment times found in the IBH (45 to 60 minutes) group compared to SMC (25 minutes). 95 CHAPTER 5 : DISCUSSION This study evaluated the clinical effectiveness , acceptability, adherence, and cost of care within an integrated behavioral health treatment model for both core (e.g., hyperactivity, impulsivity) and peripheral (e.g., aggression, conduct problems) externalizing behavior problems associated with ADHD cl inic referrals in community - based pediatric primary care. Specifically, the Michigan Algorithm (B. M. Lancaster, personal communication, November 17, 2012) was adapted from the Buffalo Algorithm (Pelham, 2007) for the purpose of delivering the core interve ntion components (i.e., parent management training, teacher classroom management training, and child home - school intervention) for use within a time - limited context (i.e., 4 to 6 ± clinic sessions over a 2 to 3 month period). Similar to the Buffalo Algorit hm, the model of care used in the present study adheres to the treatment recommendations of the AAP and APA in its use of evidence - based multimodal behavioral therapy involving the home and school as part of first line care with the initiation of medicatio n on an as - needed basis if gains are not made from the behavioral package after approximately 3 to 4 clinic sessions. The first hypothesis that IBH treatment provides greater improvement in externalizing symptoms (e.g., hyperactivity, impulsivity, aggress ion, conduct problems) associated with ADHD clinic referrals compared to SMC over the course of the study was supported. From the model, the average participant in the IBH group experienced lower T scores by 0.36 SDs based on parent ratings and 0.43 SDs ba sed on teacher ratings relative to SMC participants. While the treatments evaluated, these change scores derived over the 12 weeks of the study, in general, repre sent improvements in externalizing symptoms and in most cases a symptom severity shift 96 T - T score of 60 to 69). Overall, these results demonstrate that childr en can be treated effectively within a time - limited context with the use of behavioral treatment alone or in combination with medication. These findings are consistent with those described by Pelham (2007) and which served as the basis for the formulation of the Buffalo Algorithm, and in the present study the Michigan Algorithm, by which, the core components used in the multimodal behavioral packages of the MTA (1999a) and STP (Pelham 2000) were adapted for use within time - limited community - based clinic set - month follow - up produced effect sizes of comparable magnitude to those of the more intensive behavioral packages within the MTA and STP. However, unlike the MTA and STP, the Mic higan Algorithm may offer the feasibility for implementation in community - based settings that the former packages do not offer without significant financial support and logistical capacity. Findings in support of the first hypothesis also serve to reinfor ce the justification by APA (2006) for the use of a combined intervention approach using behavioral treatment as first line care with adjunctive medication if needed, due to side effect concerns. In fact, only 22% of participants in the IBH group eventuall y received a medication compared with 89% in the SMC group. Though not examined specifically in the present study, prior research (e.g., MTA, 1999a; Vitiello et al., 2001) has found that lower stimulant medication dosage amounts can be prescribed effective ly when evidence - based behavioral treatments are implemented concurrently. 97 put forth as a result of their risk - to - benefit considerations within their treatment r ecommendations. Though not a formal research question in the present study, when looking closer at the rate of core and peripheral externalizing symptom change trajectories across groups, it is important, though perhaps not surprising, to note that SMC, o r the de facto medication group, yielded greater symptom change in the first week for both parent and teacher ratings. This finding is certainly logical when considering the relatively immediate pharmacodynamics effects on the central nervous system repeat edly demonstrated to take effect as a result of stimulant usage. However, by 6 weeks post - treatment, differences in symptom scores in SMC and IBH treatment were negligible as there was less than a one T score point differential between the two groups. By 1 2 weeks post - treatment the relative difference between the two groups clearly favored IBH as children in this treatment group continued to evidence a moderate, yet steady decline in symptom scores, while the SMC group appeared to trend slightly back to bas eline. This finding was particularly surprising to note as this treatment response does not fit what would typically be expected from a new medication trial and the typical dose - response curve whereby medications are systematically titrated over the first few weeks. While the expectation would be for behavioral response to increase in correlation with the dosage increase to the point where the maximum therapeutic response is obtained with the lowest possible dose, the data in the present study do not reflec t this typical dose - response curve. Thus, two hypotheses are generated; the first being that systematic dosing titrations were generally not implemented for the purposes of gradually increasing dosage to slowly generate a therapeutic response (though medic ation dosage amounts were provided within participants medical records, due to the fact that each PCP would likely use different titration procedures and a variety of stimulant 98 medications were used, this data was not tracked in the present study), and the second being that the lack of adherence to medication treatments indicated throughout the study was a significant confounding variable which resulted in the group regression toward baseline. Taken as a whole, the IBH model of care yielded a gradual ye t steady improvement in symptom reduction. This gradual symptom reduction appears to correspond to the gradual increase in intensity of the treatment algorithm, whereby additional treatment components were added after each clinic session representing the i mportant ethical and clinical mantra of facilitating the least intrusive method to yield the safest therapeutic response. By nature of the MLM procedure used in the present study, valid predictions cannot be made regarding symptom reduction past the Week 1 2 data collection time point. However for the SMC group, it appears that the symptom score trajectory would likely continue its current trend for reversion toward baseline unless a significant change in treatment dosage/intensity occurred. The results per taining to this first research question uniquely contributes to the literature given its exclusive focus on both core (e.g., hyperactive, impulsive) and peripheral (e.g., aggression, conduct problems) externalizing behavioral problems associated with ADHD clinic referrals. While most studies evaluating treatments for symptoms associated with ADHD tend to lump both inattentive and hyperactive - impulsive type symptoms into a global composite, the reality in child mental health centers is that clinic referrals are more likely to involve complaints regarding core and peripheral symptoms of externalizing behavioral problems (Hinshaw & Lee, 2003). In fact, noncompliant, oppositional and aggressive behavior problems have long made up more than half of all clinic ref errals within these settings with the figure increasing to nearly 75% of referrals when including concerns for symptoms of ADHD as well (Patterson, Dishion, & Chamberlain, 1993). Results from this study may be helpful to providers who provide time - 99 limited treatment to patients presenting specifically with complaints of externalizing problem behaviors associated with ADHD, including concerns for ODD or CD, without inattentive - type symptoms. The second hypothesis that IBH treatment would be rated as more a cceptable than SMC by parents and teachers, and also that parents and teachers would rate the IBH treatment as highly acceptable was supported. Of note given the quasi - experimental nature of the study design was that these ratings did not differ significan tly depending on the demographic covariates of age, gender, ethnicity, and insurance status that from prior literature were found to be potential moderators of treatment response. These results are consistent with other recent research demonstrating both p arent (Johnston, Hommersen, & Seipp, 2008; Krain, Kendall, & Power, 2005; MTA, 1999a) and teacher (Chafouleas, Riley - Tillman, & Sassu, 2006; Girio & Owens, 2009) preferences for behavioral or combined treatments over medication for externalizing symptoms a ssociated with ADHD. For the SMC group, when examining individual items on the Treatment Acceptability Form, the item that received the lowest mean rating (3.1; SD = 0.9) was jority of ratings fell between the likert scale scores of Slightly Disagree and Disagree . The item that received the next lowest rating ( M = 3.5; SD group as evidenced by the End of Study Survey, this parental bias against medication stands in sta rk juxtaposition with the former item regarding the side effects that resulted from the treatment. Together, these two findings seemingly encapsulate the APAs (2006) treatment - s ADHD treatments. 100 The third hypothesis that IBH treatment would yield high levels of adherence by parents and teachers was supported and in line with previous research (e.g., Chafouleas et al., 2006; Vujnovic, 2009). This finding also confirms previous r esearch demonstrating positive Kazdin et al., 1997). As expected based on a review of previous literature, more than two - thirds of children treated within the SMC group received psychotropic medication as their physician - recommended treatment. In fact, 89% of children in the SMC group actually received psychotropic medication. The hypothesis that two - thirds of those patients receiving medication would not be ad indicated that 16% of children in the SMC group were no longer implementing their medication regimen. However, it should be noted that the prior research in which this hypothesis was based (Toomey et al., 2012) consisted of a much larger sample size than that used in the present study. Regardless, the 16% of children who began medication treatment in the study and who eventually stopped taking their medication for whatever reason, likely confounded the results of any attempt to establish a true treatment response to medication provided as standard medical care for externalizing behaviors associated with ADHD. Given the programmatic evaluation scope of this 6 - month pilot study, the methodological procedures implemented did not allow for analysis of how treatment adherence influenced treatment response as the End of Study Survey did not differentiate when participants stopped taking medication (only if they were no longer receiving me dication at the 12 - week follow - up), nor the dosing titration algorithm used by each of the seven respective PCPs within the SMC clinic. However, this type of data will be of critical importance to include when the present pilot study is scaled out in a lar ger and more comprehensive program evaluation study in the future. 101 Evaluation of the treatments received within SMC provides information that is of particular interest when interpreting results of the research questions of the present study. Notably, 89% (25 of 28 who completed End of Study Surveys) of patients in the SMC group received medication as their pediatrician - delivered treatment. Of which, at least 80% received medication as a monotherapy without any conjunctive behavioral treatment. Further, onl y a quarter ( n = 2) of those that received behavioral treatments ( n = 8) indicated that those treatments were still being implemented at the end of the study. These findings corroborate other reports (e.g., Hoagwood, Jensen, Feil, Vitiello, & Bhatara, 2000 ; Hoagwood, Kelleher, Feil, & Comer, 2000, MTA, 1999a) of what typically consists of the current standard of care for externalizing symptoms (e.g., hyperactivity, impulsivity, aggression, conduct problems) associated with pediatric ADHD within community - ba sed settings. As reflected in the procedural billing codes evidenced within the SMC group, the relatively brief clinic appointments (i.e., 25 minutes face time with a pediatrician) and lack of face - to - face follow - ups (i.e., less than three on average) demo nstrate the apparent barriers pertaining to time and availability inherent in SMC. Regardless of the potential of lack of training in the use of behavioral treatments that medical personnel receive for behavioral health concerns, it is clear that the logis tical make - up of the primary care model places confinements on what options PCPs may have available in order to instill the patient autonomy in treatment decision making and coordination of care amongst numerous child serving systems needed for adherence t o best practice guidelines. The fourth hypothesis that IBH treatment would evidence a higher cumulative cost of care based on CPT billing code reimbursement over the 12 week course of care tracked in the study was supported. However, while the IBH model o f care was seemingly more expensive for the modal patient (IBH = $784.02 v. SMC = $232.53), at least initially, based on the amount of 102 codes billed, results highlight a number of interesting findings. First, the typical patient in the IBH group was more li kely to be seen for follow - up during the first three months of treatment. In fact, the mode of four clinic sessions over the course of treatment aligns with the low end of the IBH initiative for time - limited patient care of 4 to 6 ± clinic sessions. Second , despite the relatively brief nature of treatment common within the primary care context, IBH patients still that occurred for psychological testing, interpre tation, and reporting of evaluation results to the patient and their family. The three hours of face time was more than double than that received via billing codes within the SMC group (75 minutes). Finally, an assessment of the type and amount of billing codes used reveals that patients in the IBH group were likely to receive a more extensive ADHD evaluation given the frequency of testing and diagnostic evaluation codes used in the first two sessions. B eyond a cursory interpretation of this initial data, the exploratory nature of the research question and data collection procedure was intended to provide a starting point and rationale for the need for follow - up analyses to be conducted by behavioral health advocates to justify the initial added cost of behavioral health integration via cost - effectiveness (i.e., comparing monetary value of resources used with clinical health effects such as symptom improvement), cost - utility (i.e., comparing monetary value of resources used with health - related quality of life effects), cost - benefit (i.e., comparing monetary value of resources used with resources saved and created and includes a quality of life index), or cost - offset (i.e., comparing monetary value of resources used with health care costs reduced by the evaluations are particularly important lines of research to pursue given the results in the present study that IBH treatment (despite increased costs and resources required initially) may be more 103 sustaining ove r the long term based on the Week 12 scores and the overall trend in parent and teacher ratings of externalizing symptoms (e.g., hyperactivity, impulsivity, aggression, conduct problems) commonly associated with ADHD. Given the chronic nature of the extern alizing symptoms associated with ADHD and the necessity to approach treatment through a life - course associated with the disorder throughout the lifespan parallel the need for evaluation of the financial costs associated with treatment to also take a life - course perspective. Likewise, the seemingly high short - term cost of behavioral health treatment in the present study should be evaluated in the context of (1) the cos t of SMC over the course of 1, 5, and 10 years, (2) the finding by Pelham, Foster, and Robb (2007) which estimated the annual cost of ADHD at $14,576 per child, and (3) the costs associated with providing no treatment at all for ADHD, which is a relevant f actor considering the demonstrated issues with access to care in SMC settings. Taken as a whole, this study was important for several reasons. First, while much research has evaluated the efficacy of multimodal treatment models for ADHD and other external izing behavioral problems in carefully controlled settings, there is a paucity of research evaluating the effectiveness of these treatments in real clinic settings. In these settings, those participants receiving therapy more closely resemble actual patien ts who possess clinical diagnoses with co - morbid conditions, pay out of pocket or through HMO - reimbursement for treatment, do not participate solely for research participation incentives, and receive treatment from clinicians who maintain full clinical cas eloads. These types of studies are important for obtaining outcome data that is generalizable to actual clinical practice. Therefore, an important consideration in evaluating empirically - supported multimodal treatments (e.g., MTA, 1999a, 104 Buffalo Algorithm, study, is their scalability for use in the context of applied settings when considering their accessibility, feasibility, and financial viability. The issue of scalability is particula rly important for ADHD systems of care given the volume of children who are affected and present within primary care. While SMC of medication monotherapy offers the advantage of access to large populations of children given its short term cost - and time - ef fectiveness, there is little research on IBH treatments that are effective, efficient, and designed and implemented with consideration for acceptability and adherence ratings from patient caretakers. These considerations are critical to the development of cost - effective practice models which are amenable to larger healthcare reform initiatives emphasizing increased integration and coordination among services and providers within the context of the Family - Centered Medical Home. While clinically - and cost - eff ective IBH practice models have been described for some time in adult settings, the research questions included in this study and discussed herein may provide early evidence for behavioral health integration in child settings, such as pediatric primary car e. Limitations The results of this study should be interpreted cautiously given several methodological limitations inherent. First, randomization to treatment condition was not undertaken or possible in this community - based study, meaning that indi viduals in the two groups may not be the same. In fact, because the participant population was a convenience sample and not a random sample, it is possible that the families in the study who sought treatment in the IBH clinic as first line care may be inhe rently different from other families who sought treatment from their local PCP. One example supporting this difference is the discrepancy in Medicaid rates between groups. Also, the present study was limited by the absence of a control group due to the eth ical implications of 105 withholding treatment for up to three months to children who were referred for externalizing ADHD concerns to their healthcare provider. T o minimize the risks posed to internal and external validity inherent in the non - randomized and q uasi - experimental design of the present study, several methodological strategies were employed to control for key demographic covariates (i.e., age, gender, ethnicity, and insurance status) that previous literature has indicated to be potential mediators o f treatment outcomes. However, there may be other prominent group differences in those who sought services in the IBH clinic versus those who sought services in the SMC clinic. These variables (e.g., differences in treatment preferences for ADHD, differenc es in perceptions of mental health etiology/treatment) that were not observed limit the generalizability of the results. While the non - randomization and the quasi - experimental design are the most prominent limitations of the study, there are sever al other limitations that must be considered when interpreting the results. First, while researchers could exclude participants if they had previously been diagnosed with a disruptive behavior disorder as indicated in their electronic medical record, resea rchers had no way of knowing if patients had previously been diagnosed at a different clinic. Second, given the homogenous sample of primarily Caucasian participants, caution should be used when attempting to generalize these results to more heterogeneous populations. Another limitation of the study was its evaluation period of only 12 weeks, a short time frame for treatment effects compared to many other studies examining treatment of externalizing behavioral symptoms associated with ADHD, given the chroni c nature of the disorder. While this treatment time frame provides data regarding whether treatment effects are lasting, no conclusions can be drawn about the effectiveness of the treatment past this time. Also, due to the heterogeneous nature of both the Michigan Algorithm which was idiographic in nature 106 delivered in the S MC group included the use of up to seven different treatments, the results of this study may be limited in their generalizability. For the SMC group, while the treatments received were tracked, the intensity and dosage of those treatments (e.g., medication dosage, frequency of behavioral interventions) were not evaluated specifically. For the IBH gro up, specific treatments that each participant received were not tracked in a reliable manner. Another limitation in the study was that there was no tracking of differential treatment response in the modeling procedure for those patients in the IBH group wh o received medication after 4 to 6 weeks as combined approach and those who continued with behavioral treatments only. Additionally, in the present study, blind to medication treatment status to prev ent this knowledge from influencing the ratings. potentially influenced their ratings of behavior. Relatedly, clinical improvements in this study were based on parent and teacher rating scales which are subjective in nature. There are several factors (e.g., biases such as leniency or severity of raters, halo effect, expectancy effe ct, and central tendency effects) which may contribute to error variance related to rater - subjectivity. Also, despite post hoc analyses of missing data in the present study confirming that parent and teacher data was missing at random, it cannot be certain whether those who did not respond to surveys were inherently different in some regard than those who did respond. Also, though this broad sense, provides a novel addition to the treatment literature for ADHD care within applied community based settings, this study however was limited in its lack of specificity in evaluating 107 differential treatment response in core versus peripheral externalizing symptoms and also di d not look at changes in the core ADHD symptoms of inattention within the study. Finally, regarding the exploratory research question examining cost of care in the two treatment groups, two prominent limitations exist which must be considered when inter preting the data. First, the cumulative cost of care for the modal patient in the IBH group only considers reimbursement rates for CPT codes billed by the behavioral health provider. It is does not include reimbursement fees for the initial PCP visit in wh ich the ADHD - related concerns were raised. Also, the cumulative cost of care does not include reimbursement stemming from billing by the PCP once a patient was referred for a medication evaluation while continuing to be treated by the behavioral health pro vider. Relatedly, the reimbursement rates in both treatment groups do not account for the cost of any medications received by a patient. Lastly, the cost of care data do not account for any prior services received either at the clinics followed in the stud y or any other clinic. Implications Future practice . The integration and evaluation of evidence - based behavioral health treatments in pediatric primary care for both core (e.g., hyperactivity, impulsivity) and peripheral (e.g., aggression, oppositionality, conduct problems) externalizing behavior problems i n ADHD clinic referrals represents an important service delivery development. Recent changes in health care legislation have created a new frontier for mental health service delivery aiming to afford patients with an increased accessibility to and continui ty of high quality clinical care that was not possible under the previous standard of care often characterized by service and provider fragmentation and a lack of treatment options provided to the patient. The Patient Protection and Affordable Care Act (PP ACA) emphasizes comprehensive health care delivery within the 108 Family - Centered Medical Home (FCMH) via multidisciplinary integration and collaboration ( Braddock, Snyder, Neubauer, & Fischer, 2012 ). With the likely increase in youth who will present in prima ry care and be screened by their PCP for common pediatric behavioral health referral concerns such as externalizing behaviors associated ADHD, psychologists with training in integrated care approaches are uniquely qualified to fill this critical healthcare need. By demonstrating to HMO and other administrative decision makers the feasibility and clinical benefits of these treatment models in relation to SMC, psychologists can increase the likelihood that their services can be delivered within a service deli very platform in which they were previously not accessible. Inherent in the demonstration to these decision makers that psychologist - delivered treatment models can effectively and efficiently provide comprehensive patient care is the degree to which they can be aligned with the core principles of the patient - center ed, or more fittingly family - centered medical home. These principles, developed in a consensus statement by the American Academy of Family Physicians, the American Academy of Pediatrics, the American College of Physicians, and the American Osteopathic Asso ciation (2007), include (a) a relationship with a personal physician, (b) a physician - directed team of integrated and multidisciplinary providers, (c) whole person orientation, (d) care coordination/integration, (e) quality and safety, and (f) enhanced acc ess to care. These joint principles pertaining to the - including beneficence, nonmaleficence, and respect for autonomy and justice. To this end, a central tenet with in the rationale regarding behavioral health integration within primary care has been the emphasis on patient choice in the treatment decision making process by presenting multiple options along with the empirical evidence supportive each. The process of i nvolving 109 patient choice in the decision making process reflects the larger emphasis on biopsychosocial conceptualizations of health that have proliferated recently in health care design (Talen & Valeras, 2013). For externalizing symptoms associated wit h ADHD, these conceptualizations stand in stark juxtaposition with the current SMC of medication monotherapy with little attention paid to the clinical and ethical importance of eliciting caretaker preferences regarding treatment (Epstein et al., 2013). In fact, the current literature is replete with evidence demonstrating that both parent Johnston, Hommersen, & Seipp, 20 08; MTA, 1999a ). Owing to this finding, as well as the research literature indicating that the balance of risks to benefits favors behavioral treatments, the APA (2006) recommended that behavioral treatments involving the home and school be used as first l ine care. The results of the present study, namely the finding that approximately only a fifth (21.6%) of patients went on to receive medication within the IBH model of care, further reinforce the clinical and ethical importance of providing families with the option to receive evidence - based behavioral treatment either with or without medication as first line care. Allowing families to make an informed choice among a number of evidence - based treatment options with consideration for treatment efficacy, time, safety/side - effects, maintenance and adherence variables, along with concomitant parental, familial, environmental, and systems factors intuitively aligns with those ethical dimensions pertaining to - dividual family needs may improve not only clinical outcomes by ensuring treatment acceptability and by proxy treatment adherence, but also enhance patient access by offering alternative options to medication - resistive families who may otherwise not seek c are in medical settings. Providing these options within the context 110 of the on - site medical home may counter the demonstrated poor follow - through to off - site referral - based behavioral health providers. While behavioral health integration and FCMH efforts have centered largely on improving patient access and clinical outcomes, the current state of health care reform has also raised the issue of financial sustainability as an equally important indicator of program success (Goodheart, 2010). Despite decades o f medical cost - offset literature consistently demonstrating that the integration and utilization of behavioral medicine effectively reduces unnecessary medical costs (Chiles, Lambert, & Hatch, 1999), little research has systematically demonstrated the impa ct that behavioral health integration has on medical cost - offset for externalizing ADHD referrals in pediatric practice settings specifically. This initiative appears to be a critical need in future research in order to communicate to decision makers the i mpact and added value of integrated care programming for this increasingly common referral concern. In fact, for the field of psychology in general, the demonstration of the financial viability of behavioral health integration models is a requisite compone nt in ensuring that the provision of psychological delivery. This triple aim consists of improving patient access to care, improving the quality and outcomes of patient care, and reducing total healthcare costs (Katon & Unützer, 2011). Future research. The implications of the present study provide direction for several lines of future research pertaining broadly to integrated behavioral health care within chil d serving systems and specifically to best practice primary care for externalizing problems associated with ADHD clinic referrals. Broadly speaking, both scientists and practitioners in the field of psychology must recognize that the movement toward and ex pansion of behavioral health integration within medical settings has gained profound momentum. This emerging paradigm 111 shift offers the opportunity for psychologists to demonstrate that their unique competencies in the evidence - based assessment and treatmen t of mental health conditions and training in problem - solving consultation within multidisciplinary contexts are ideal for filling a healthcare need in the delivery of behavioral health services, particularly with the shortage of available pediatric mental health providers (Kim, 2003). The implementation and evaluation of psychologist - delivered treatments in primary care is particularly timely given APAs strategic plan of expanding psychology's role in the context of health care reform (Clay, 2011). Related to this goal, recent APA President Nadine Kaslow endorsed the promotion and evaluation of PCMHs as a primary initiative of her presidency. More specifically, by evaluating the extent to comes, acceptability, and adherence, the importance of psychologist - delivered treatments in primary care can be demonstrated, thus eliciting their demand within increasingly integrated systems of care. More specifically, the astounding prevalence of ext ernalizing ADHD concerns in child populations and their impact on the individual, family, and system(s) (Barkley, 2006) coupled with the relatively large amount of treatment research on the disorder compared with other common behavioral health concerns, ra ises the question of why current delivery systems are so discrepant from best practice, both from the perspective of the most esteemed medical (i.e., AAP) and psychological (i.e., APA) organizations. The recommendation that the school be part of any treatm an integrated fashion. However, the research literature demonstrating the gener al reluctance on the part of PCPs to involve the school in service delivery points to the practical and logistical barriers that may exist in the facilitation of this cross setting collaboration. Given the necessity 112 for collaborative consultation between m edical and school personnel to occur, inter - professional models must be cost - and time - efficient in order to be practically and routinely included as part of the standard of clinical care (DuPaul, 2011). To this end, there have recently been promising tech nological applications of Web - based programs using an online portal intended to facilitate cross setting communication and collaboration regarding assessment and treatment (e.g., myADHDportal.com; Epstein et al., 2013). With further development and refinem ent of these Web - based approaches to fostering enhanced collaboration with educational systems, it may be possible to reduce the existing barriers that health care providers experience in adhering to AAP and APA consensus guidelines. While the Michigan Algorithm may be positioned to promote a better quality and continuity of care for youth with externalizing ADHD referral concerns, this interdisciplinary model valuing school partnerships must be explored further using a larger scale and randomized - contro l design to evaluate the treatment in a more comprehensive and controlled manner. To this end, subsequent studies should explore potential mediators of treatment effects such as adherence to treatment, study blindness, and dosage/intensity amounts. Regardi ng the exploration of treatment dosage/intensity amounts in future studies, it will be important to examine the medication dosage amounts that were prescribed to participants in the IBH group compared with participants receiving medication within the SMC g roup as prior research (e.g., MTA, 1999a; Vitiello et al., 2001) has highlighted the additive effects of combined treatments which may allow for lowering medication dosing. Also, considering the chronic nature of externalizing ADHD symptoms as well as othe r comorbid disorders (e.g., ODD, CD), the addition of a one - and two - year post treatment follow - up assessment would contribute to the 113 relation to SMC. The degree to whi ch treatment effects are lasting may figure in evaluating the long term cost - effectiveness and/or cost - benefit needed to ensure program sustainability. Additionally, while this study evaluated the degree of reduction in externalizing ADHD symptoms in a b road sense, future studies can evaluate the degree of reduction in core versus peripheral externalizing symptoms specifically. Similarly, subsequent studies can also evaluate the degree of reduction in inattentive symptoms associated with ADHD clinic refer rals. There are several instruments which provide different subscales for inattentive and hyperactive - impulsive symptoms (e.g., SNAP - IV; Swanson, Lerner, March, Gresham, 1999). Future studies should also include a dependent measure focusing on changes in a cademic performance (e.g., Academic Performance Rating Scale; APRS; DuPaul, Rapport, & Perriello, 1991). Finally, while the emphasis on involving the home and school in clinical care is a major feature of the Michigan Algorithm, future research should expl ore ways in which technological or Web - based applications can be used to streamline the existing tri - directional communication and collaboration in order to foster enhanced feasibility and disseminability of the treatment model. 114 REFERENCES 115 REFERENCES responses to stimulant medication and two intensities of a behavioral intervention. Behavior Modification, 16, 193 203. doi: 10 .1177/01454455920162003 American Academy of Family Physicians, American Academy of Pediatrics, American College of Physicians, American Osteopathic Association. (2007). Joint principles of the patient - centered medical home . Retrieved from http://www.acponline.org/running_practice/delivery_and_payment_models/pcmh/demon strations/jointprinc_05_17.pdf American Academy of Pediatrics. (2001). Clinical practice guideline: Treatment of the school - aged child with attention deficit/hyperactivity disorder. Pediatrics, 108, 1033 1044. American Academy of Pediatrics. (2011). ADHD: Clinical practice guideline for the diagnosis, evaluation, and treatment of attention - deficit/hyperactivity disorder in children and adolescents. Pediatrics, 128, 1007 1022. doi: 10.1542/peds.2011 - 2654 American Academy of Pediatrics, Committee on Quality Improvement. (2010). Diagnosis and evaluation of the chil d with attention - deficit/hyperactivity disorder. Pediatrics, 105 , 1158 1170. doi: 10.1542/peds.2011 - 2654 American Psychiatric Association. (2013). Diagnostic and Statistical Manual of Mental Disorders (5 th ed.). Washington, DC: Author. American Psycho logical Association. (2006). Report of the Working Group on Psychotropic Medications for Children and Adolescents: Psychopharmacological, Psychosocial, and Combined Interventio ns for Childhood Disorders: Evidence Base, Contextual Factors, and Future Dir ections . Washington, DC: Author. Retrieved from http://www.apa.org/pi/families/resources/child - medications.pdf American Psychological Association. (2013). integrating mental health care with physical health care . Retrieved from http://www.apa.org/news/press/releases/2013/06/mental - physical.aspx American Psychological Association, Division of Clinical Child and Adolescent Psychology. (2012). Evidence - based psychosocial treatments for Attention - Deficit/Hyperactivity Disorder . Retrieved from http://effectivechildtherapy.com/content/attention - deficithyperactivity - disord er - adhd - 0 American Psychological Association, Presidential Task Force on Evidence - Based Practice. (2006). Evidence - based practice in psychology. American Psychologist, 61 , 271 285. doi: 10.1037/0003 - 066X.61.4.271 116 Anastopoulos, A. D., Shelton, T. L., DuPaul, G. J., & Guevremont, D. C. (1993). Parent training for attention - deficit hyperactivity disorder: Its impact on parent functioning. Journal of Abnormal Child Psycholo gy , 21 , 581 - 596. Retrieved from http://libres.uncg.edu/ir/uncg/f/A_Anastopoulos_Parent_1993.pdf Angold, A., Costello, E. J., & Erkanli, A. (1999). Comorbidity. Journal of Chi ld Psychology and Psychiatry, 40 , 57 - 88. doi: 10.1111/1469 - 7610.00424 Arbuckle, J. L. (1996). Full information estimation in the presence of incomplete data. In C. A. Marcoulides & R. E. Schumaker (Eds.), Advanced structural equation modeling: Issues and techniques (pp. 243 - 278). Mahwah, NJ: Lawrence Erlbaum Associates. ects of ethnicity on treatment attendance, stimulant response/dose, and 14 - month outcome in ADHD. Journal of Consulting and Clinical Psychology, 71 , 713 - 727. doi: 10.1037/0022 - 006X.71.4.713 Barkley, R. A. (2003). Attention - deficit/hyperactivity disorde r. In E. J. Mash & R. A. Barkley (Eds.), Child Psychopathology , 2 nd ed. (pp. 75 - 143). New York: Guilford. Barkley, R. A. (2006). Attention - deficit/hyperactivity disorder: A handbook for diagnosis and treatment (3rd ed.). New York: Guilford Press. Barkley, R. A., Murphy, K. R., & Fischer, M. (2008). ADHD in adults: What the science says . New York: Guilford Press. Barr, C. L., Wigg, K. G., Bloom, S., (2000). Further evidence from haplotype a nalysis for linkage of the dopamine D4 receptor gene and attention - deficit hyperactivity disorder. American Journal of Medical Genetics, 96 , 262 - 267. doi: 10.1002/1096 - 8628 Beauchaine, T. P., Hinshaw, S. P., & Pang, K. L. (2010). Comorbidity of attenti on - deficit/hyperactivity disorder and early - onset conduct disorder: Biological, environmental, and developmental mechanisms. Clinical Psychology: Science and Practice, 17 , 327 - 336. doi: 10.1111/j.1468 - 2850.2010.01224.x Belar, C. D. (2008). Clinical hea lth psychology: A health care specialty in professional psychology. Professional Psychology: Research and Practice, 39 , 229 233. doi: 10.1037/0735 - 7028.39.2.229 Blount, A (2007). The economics of behavioral health services in medical settings: A summary of the evidence. Professional Psychology: Research and Practice, 38 , 290 - 297. doi: 10.1 037/0735 - 7028.38.3.290 117 Blount, A. (1998). An introduction to integrated care. In A. Blount (Ed.). Integrated primary care: The future of medical and mental health collaboration . New York: W.W. Norton. Blount, A. (2003). Integrated primary care: Organizi ng the evidence. Families, Systems, and Health, 21 , 121 - 134. doi: 10.1037/1091 - 7527.21.2.121 Blum, N. J., Mauk, J. E., McComas, J. J., & Mace, F. C. (1996). Separate and combined effects of methylphenidate and a behavioral intervention on disruptive behavior in children with mental retardation. Journal of Applied Behavior Analysis, 29 , 305 - 319. doi: 10.1901/jaba.1996.29 - 305 Braddock, C. H., Snyder, L., Neubauer, R. L., & Fischer, G. S. (2012). The patient - centered medical home: An Ethical analysis of principles and practice. Journal of General Internal Medicine, 28 , 141 - 146. doi: 10 .1007/s11606 - 012 - 2170 - x Bradley, J. D., & Golden, C. J. (2001). Biological contributions to the presentation and understanding of attention - deficit/hyperactivity disorder: A review. Clinical Psychology Review, 21 , 907 - 29. doi: 10.1016/S0272 - 7358(00)00073 - 8 Brown Levey, S. M., Miller, B. F., & deGruy, F. V. (2012). Behavioral health integration: An essential element of population - based healthcare design. Pract ice and Public Health Policies , 1 - 8. doi: 10.1007/s13142 - 012 - 0152 - 5 Bryan, C. J., Morrow, C., & Appolonio, K. K. (2009). Impact of behavioral health consultant interventions on patient symptoms and functioning in an integrated family medicine clinic. J ournal of Clinical Psychology, 65 , 281 - 293. doi: 10.1002/jclp.20539 Campbell, S. B., Shaw, D. S., & Gilliom, M. (2000). Early externalizing behavior problems: Toddlers and preschoolers at risk for later maladjustment. Development and Psychopathology, 12 , 467 488 . doi: 10.1017/S0954579400003114 Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi - experimental designs for research on teaching. In N. L. Gage (Ed.), Handbook of research on teaching (pp. 171 246). Chicago, IL: Rand McNally. Cape, J., Whittington, C., Buszewicz, M., Wallace, P., & Underwood, L. (2010). Brief psychological therapies for anxiety and depression in primary care: Meta - analysis and meta - regression. BMC Medicine, 8 . doi: 10.1186/1741 - 7015 - 8 - 38 Carlson, C. L., Pelham, W. E., Milich, R., & Dixon, J. (1992). Single and combined effects of methylpheni date and behavior therapy on the classroom performance of children with ADHD. Journal of Abnormal Child Psychology, 20 , 213 - 232. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/1593027 118 Centers for Disease Control and Prevention. (2010). Increasing prevalence of parent - reported attention - Morbidity and Mortality Weekly Report, 59 , 1439 - 1443. Retrieved from http://www.cdc.gov/mmwr/preview/mmwrhtml/mm5944a3.htm?s_cid=mm5944a3_w Chaffee, B. (2009). Financial models for integrated behavioral health care. In L. C. James, W. T., hue (Eds.), The Primary Care Toolkit . New York, NY: Springer. Chafouleas, S. M., McDougal, J. L., Riley - Tillman, T. C., Panahon, C. J., & Hilt, A. M. (2005). What do daily behavior report cards (DBRCs) measure? An initial comparison of DBRCs with direc t observation for off - task behavior. Psychology in the Schools , 42, 669 - 676. doi: 10.1002/pits.20102 Chafouleas, S. M., Riley - Tillman, C. T., & Sassu, K. A. (2006). Acceptability and reported use of daily report cards among teachers. Journal of Positive Behavior Interventions, 8 , 174 - 182. doi: 10.1177/10983007060080030601 Cheung, M. W. L. (2013). Implementing restricted maximum likelihood estimation in structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 20 , 157 - 167. doi:10.1080/10705511.2013.742404 Chiles, J. A., Lambert, M. J., & Hatch, A. L. (1999). The impact of psychological interventions on medical cost offset: A meta - analytic review. Clinical Psychology: Science and Practice, 6 , 204 - 220. doi: 10.1093/clipsy/6.2.204 Christ, T. J., Riley - Tillman, T. C., & Chafouleas, S. M. (2009). Foundation for the development and use of direct behavior rating (DBR) to assess and evaluat e child behavior. Assessment for Effective Intervention, 34, 201 213. doi: 10.1177/1534508409340390 Chronis, A. M., Fabiano, G. A., Gnagy, E. M., Onyango, A. N., Pelham Jr., W. E., Lopez - treatment program for children with ADHD using a treatment withdrawal design. Behavior Therapy, 35 , 561 - 585.doi: http://dx.doi.org.proxy2.cl.msu.edu/10.1016 /S0005 - 7894(04)80032 - 7 Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates. Cohen, N. J., & Thompson, L. (1982). Perceptions and attitudes of hyperactive children and their mothers regarding treatment with methylphenidate. Canadian Journal of Psychiatry , 27 , 40 - 42. Cook, T. D., & Campbell, D. T. (1979). Quasi - experimentation designs and analysis issues for field settings. Boston: Houghton Mifflin. 119 Connor, D., F., Fletc her, K. E., & Swanson, J. M. (1999). A meta - analysis of clonidine for symptoms of attention - deficit hyperactivity disorder. Journal of the American Academy of Child and Adolescent Psychiatry , 38 , 1551 - 1559. doi: 0890 - 8567/99/3812 - 1551 Conners, C. K. (1969). A teacher rating scale for use in drug studies with children. American Journal of Psychiatry, 126 , 884 - 888. Retrieved from http://ajp.psychiatryonline.org/article.aspx? articleid=151727 Conners, C. K. (2002). Forty years of methylphenidate treatment in attention - deficit/ hyperactivity disorder. Journal of Attention Disorders, 6 , 17 - 30. Retrieved from htt p://www.medscape.com/medline/abstract/12685516 Cox, D. D. (2005). Evidence - based interventions u sing home - school collaboration. School Psychology Quarterly, 20, 473 497. doi: 10.1521/scpq.2005.20.4.473 Crenshaw, T. M., Kavale, K. A., Forness, S. R., & Reeve, R. E. (1999). Attention Deficit Hyperactivity Disorder and the efficacy of stimulant medication: A meta - analysis. In Scruggs, T. & Mastropieri, M. (Eds.), Advances i n learning and behavioral disabilities , (pp. 135 - 165). Greenwich, CT: JAI Press. Cross - Calvert, S. & Johnston, C. (1990). Acceptability of treatments for child behavior problems and implications for future research. Journal of Clinical Child Psychology, 19 , 61 - 74. doi: 10.1207/s15374424jccp1901_8 Corkum, P. V., Mckinnon, M. M., & Mullane, J. C. (2005). The effect of involving classroom teachers in a parent training program for families of children with ADHD. Child and Family Behavior Therapy, 27 , 29 - 49. doi: 10.1300/J019v27n04_02 Cummings, N. A. (1997). Pioneering integrated systems: lessons learned, forgotten, and relearned. In N. A. Cummings, J. L. Cummings, and J. N. Johnson (Eds.). Behavioral Health in Primary Care: A Guide for Clinical In tegration (pp. 23 - 36). Madison, CT: Psychosocial Press. Understanding the behavioral healthcare crisis: The promise of integrated care and diagnostic reform . New York: Routledge. Curran, P. J., Obeidat, K., & Losardo, D. (2010). Twelve frequently asked questions about growth curve modeling. Journal of Cognition and Development, 11 , 121 - 136. doi: 10.1080/15248371003699969 inet, M. S. (2003). Randomized trial of financial incentives and delivery methods for improving response to a mailed questionnaire. American Journal of Epidemiology, 157 , 643 - 651. 120 Dougherty, E. H., & Dougherty, A. (1977). The daily report card: A simpl ified and flexible package for classroom behavior management. In L. A. Hersov & M. Berger (Eds.), Aggression and antisocial behavior in childhood and adolescence (pp. 73 93). London: Pergamon Press. dosReis, S., Mychailyszyn, M. P., Evans - Lacko, S. E., Beltran, A., Riley, A. W., & Myers, M. A. (2009). The meaning of attention - initiation and continuity of treatment for their child. Journal of Child and Adolescent Psychopharmacology, 19 , 377 - 383. doi: 10.1089=cap.2008.0118 Dumas, J. E., & Wahler, R. G. (1983). Predictors of treatment outcome in parent training: Mother insularity and socioeconomic disadvantage. Behavioral Assessment, 5 , 301 - 313. DuPaul, G. J. (2011). Collaboration between medic al and educational professionals: Toward a proactive, integrated, and cost - efficient system of care. Journal of Educational and Psychological Consultation, 21 , 166 - 168. doi: 10.1080/10474412.2011.571523 DuPaul, G. J., & Eckert, T. L. (1997). School - based intervention for children with attention/deficit/hyperactivity disorder: A meta - analysis. School Psychology Review, 26 , 5 - 27. DuPaul, G. J., Rapport, M. D., & Perriello, L. M. (1991). Teacher ratings of academic skills: The development of the academi c performance rating scale. School Psychology Review, 20 , 284 - 300. Retrieved from http://www.nasponline.org/publications/spr/index.aspx?vol=20&issue=2 Eckert, T. L., & Hintze, J. M. (2000). Behavioral conceptions and applications of acceptability: Issues related to service delivery and research methodology. School Psychology Quarterly, 15 , 123 - 148. doi: http://dx.doi.org.proxy2.cl.msu.edu/10.1037/h0088853 Enders, C. K. (2010). Applied missing data analysis . New York, NY: Guilford. Epstein, J. N., Langberg, J. M., Lichtenstein, P. K., Kolb, R. C., & Simon, J. O. (2013). The myADHDportal.com improvement program: An innovative quality improvement intervention for improving the quality of ADHD care among community - based pediatricians. Clinical Practice in Pediatric Psychology , 1, 55 - 67. doi: 10.1037/cpp0000004 Epstein, J. N., Langberg, J. M., Lichtenstein, P. K., Mainwaring, B. A., Luzader, C. P., & Stark, L. J. (2008). A community - wide intervention to improve the ADHD assessment and treatment practi ces of community physicians. Pediatrics, 122, 19 27. doi: 10.1542/peds.2007 - 2704 Conners, C. K. (2007). Improving attention - deficit/hyperactivity disorder treatment outcome through use of a collaborative consultation treatment service by 121 community - based pediatricians: A cluster randomized trial. Archives of Pediatrics and Adolescent Medicine, 161, 835 840. Evans, S. W., & Youngstrom, E. (2006). Evidence - based assessment of attention deficit hyperactivity disorder: Measuring outcomes. Journal of the American Academy of Child and Adolescent Psychiatry, 45, 1132 1137. doi: 10.1097/01.chi.0000228355.23606.14 Fabiano, G. A., Pelham Jr., W. E., Co les, E. K., Gnagy, E. M., Chronis - Tuscano, A., & - analysis of behavioral treatments for ADHD. Clinical Psychology Review, 29 , 129 - 140. doi: 10.1016/j.cpr.2008.11.001 Fabiano, G. A., Vujnovic, R. K., Pelham, W. E., Waschbusc h, D. A., Massetti, G. M., Pariseau, programming for children with attention deficit hyperactivity disorder using a daily report card. School Psychology Review, 39 , 219 - 239. Retr ieved from http://eric.ed.gov/?id=EJ891850 Faraone, S. V., & Biederman, J. (2002). Efficacy of Adderall for attention - deficit/hyperactivity disorder: A meta - analysis. Journal of Attention Disorders , 6 , 69 - 75. doi: 10.1177/108705470200600203 Faraone, S. V., & Buitelaar, J. (2010). Comparing the efficacy of stimulants for ADHD in children and adolescents using meta - analysis. European Child and Adolescent Psychiatry, 19 , 353 - 364. doi: 10.1007/s00787 - 00 9 - 0054 - 3 Feldman, O. M., Lee, D. L., & Perez - Stable, E. J. (2006). Targeted child psychiatric services: A new model of pediatric primary clinician - child psychiatry collaborative care. Clinical Pediatrics, 45 , 423 - 434. doi: 10.1007/s10488 - 005 - 4233 - 3 Flo res, G., Olson, L., & Tomany - Korman, S. C. (2005). Racial and ethnic disparities in early childhood health and health care. Pediatrics , 115 , 183 - 193. doi: 10.1542/peds.2004 - 1474 Foster, M. E., Jensen, P. S., Schlander, M. L., Pelham, W. E., Hechtman, L. Arnold, E. - effective for more complex cases? Health Research and Educational Trust, 42 , 165 - 182. Fraz ier, T. W., Youngstrom, E. A., Glutting, J. J., & Watkins, M. W. (2007). ADHD and achievement: Meta - analysis of the child, adolescent, and adult literatures and a concomitant study with college students. Journal of Learning Disabilities, 40, 49 65. Froe hlich, T. E., Lanphear, B. P., Epstein, J. N., Barbaresi, W. J., Katusic, S. K., & Kahn, R. S. (2007) Prevalence, recognition, and treatment of attention - deficit/hyperactivity disorder in a national sample of U. S. children. Archives of Pediatric and Ado lescent Medicine, 161 , 857 - 864. 122 Habel, L., Schaefer, C., Levine, P., Bhat, A., & Elliott, G. (2005). Treatment with stimulants among youths in a large California health plan. Journal of Child and Adolescent Psychopharmacology, 15 , 62 - 67. doi:10.1089/ca p.2005.15.62. Hinshaw, S. P., & Lee, S. S. (2003). Conduct and oppositional defiant disorders. In E. J. Mash & R. A. Barkley (Eds.), Child psychopathology (2 nd ed., pp. 144 - 198). New York: Guilford Press. Hoza, B., effects of both behavior therapy and methylphenidate on the classroom performance of two ADHD children. Behavior Modification, 16, 164 192. doi : 10.1177/014544559201620 02 Hunsley, J., & Lee, C. M. (2007). Research - informed benchmarks for psychological treatments: Efficacy studies, effectiveness studies, and beyond. Professional Psychology: Research and Practice, 38 , 21 33. doi: 10.1037/0735 - 7028.38.1.21 Gage, J. D. ( 2002). - deficit/hyperactivity disorder: The effects of presentation and information delivery (Doctoral dissertation). Retrieved from http://sunzi.lib.hku.hk/ER/detail/hkul/2694321 Gage, J. D., & Wilson, L. J. (2000). Acceptability of attention - deficit/hyperactivity disorder interventions: A comparison of parents. Journal of Attention Disorders, 4, 174 - 182. doi: 10.1177/108705470000400304 Gardner, W., Kelleher, K. J., Pajer, K., & Camp, J. V. (2004). Follow - up care of children identified with ADHD by primary care clinicians: A prospective cohort study. Journal of Pediatrics, 145, 767 771. doi: 10.1016/j.jpeds.2 004.08.028 Gatchel, R. J., & Oordt, M. S. (2003). Clinical health psychology and primary care: Practical advice and clinical guidance for successful collaboration . Washington, DC: American Psychological Association. Gilmore, A., & Milne, R. (2001). Me thylphenidate in children with hyperactivity: Review and cost - utility analysis. Pharmacoepidemiology and Drug Safety, 10 , 85 - 94. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/11499858 Girio , E. L., & Owens, J. S. (2009). Teacher acceptability of evidence - based and promising treatment for children with attention - deficit/hyperactivity disorder. School Mental Health, 1 , 16 - 25. doi: 10.1007/s12310 - 008 - 9001 - 6 Goodheart, C. (2010). Economics and psychology practice: What we need to know and why. Professional Psychology: Research and Practice, 41 , 189 - 195. doi: 10.1037/a0019498 123 Greenhill, L. L., Swanson, (2001). Impairment and deportment responses to different methylphenidate doses in children with ADHD: The MTA titration trial. Journal of the American Academy Child and Adolescent Psychi atry, 40 , 180 187. doi: 10.1097/00004583 - 200102000 - 00012 Gresham, F. M. (1989). Assessment or treatment integrity in school consultation and prereferral interventions. School Psychology Review, 18 , 37 - 50. Retrieved from http://www.nasponline.org/publications/spr/abstract.aspx?ID=944 Hoagwood, K., Kelleher, K. J., Feil, M., & Comer, D. M. (2000). Treatment services for children with ADHD: A national perspective. Journal of the American Academy of Child and Adolescent Psychiatry, 39 , 198 - 206. doi: 10.1097/00004583 - 200002000 - 00020 Hoagwood, K., Jensen, P. S., Feil, M., Vitiello, B., & Bhatara, V. (2000). Medication management of stimulants in pediatric practice settings: A national perspective. Developmental and Behavioral Pedia trics, 21, 322 331. doi: 10.1097/00004703 - 200010000 - 00002 Co st - effectiveness of ADHD treatments: findings from the multimodal treatment study of children with ADHD. American Journal of Psychiatry, 162 , 1628 - 1636. doi: 10.1 176/appi.ajp.162.9.1628 Findings from the NIMH Multimodal Treatment Study of ADHD (MTA): Implications and applications for primary care providers. Journal o f Developmental and Behavioral Pediatrics, 22, 60 73. doi: 10.1097/00004703 - 200102000 - 00008 Johnston, C. Hommersen, P., & Seipp, C. (2008). Acceptability o f behavioral and pharmacological treatments for attention - deficit/hyperactivity disorder: Relations to child and parent characteristics. Behavior Therapy, 39 , 22 - 32. doi: 10.1016/j.beth.2007.04.002 Jones, H. A., Epstein, J. N., Hinshaw, S. P., Owens, E. C. (2010). Ethnicity as a moderator of treatment effects on parent - child interaction for children with ADHD. Journal of Attention Disorders, 13 , 592 - 600. doi: 1 0.1177/1087054709332158 Jurbergs, N., Palcic, J. , & Kelley, M. L. (2007). School home notes with and without response cost: Increasing attention and academic performance in low - income children with attention - deficit/hyperactivity disorder. School Psychology Quarterly, 22 , 358 379. doi: 10.1037/1045 - 3830.22.3.358 Kaplan, R. M., & Groessl, E. J. (2002). Applications of cost - effectiveness methodologies in behavioral medicine. Journal of Consulting and Clinical P sychology, 70 , 482 - 493. doi: 10.1037//0022 - 006X.70.3.482 124 Katon, W., Roy - Byrne, P., Russo, J., & Cowley, D. (2002). Cost - effectiveness and cost offset of a collaborative care intervention for primary care patients with panic disorder. Archives of Genera l Psychiatry, 59, 1098 1104. doi: 10.1001/archpsyc.59.12.1098 Katon, W., & Unützer, J. (2011). Consultation psychiatry in the medical home and accountable care organizations: Achieving the triple aim. General Hospital Psychiatry, 33 , 305 - 310. doi:10.1016/j.genhosppsych.2011.05.011 Collaborative management to achieve treatmen t guidelines: Impact on depression in primary care. Journal of the American Medical Association, 273 , 1026 - 1031. doi:10.1001/jama.1995.03520370068039 Kazdin, A. E. (1980). Acceptability of alternative treatments for deviant child behavior . Journal of Applied Behavior Analysis, 13 , 259 273. doi: 10.1901/jaba.1980.13 - 259 Kazdin, A.E. (1981). Acceptability of child treatment techniques: The influence of treatment e fficacy and adverse side effects. Behavior Therapy, 12, 493 - 506. doi: 10.1016/S0005 - 7894(81)80087 - 1 Kazdin, A. E. (2011). Single - case research designs: Methods for clinical and applied settings (2 nd ed.). Oxford University Press. Kazdin, A. E., Holland, L., & Crowley, M. (1997). Family experience of barriers to treatment and premature termination from child therapy. Journal of Consulting and Clinica l Psychology, 65, 453 - 463. doi: 10.1037/0022 - 006X.65.3.453 Kim, W. J. (2003). Child and adolescent psychiatry workforce: A critical shortage and national challenge . Academic Psychiatry, 27 , 277 - 282. doi:10.1176/appi.ap.27.4.277 Klein, R. G., & Abikoff, H. (1997). Behavior therapy and methylphenidate in the treatment of children with ADHD. Journal of Attention Disorders, 2, 89 114. doi: 10.1177/108705479700200203 Krain, A. L., Kendall, P. C., & Power, T. J. (2005). The role of treatment acceptability in the initiation of treatment for ADHD. Journal of Attention Disorders, 9 , 425 - 434. doi: 10.1177/1087054705279996 Kratochwill, T. R., Elliott, S. N., Loitz, P. A., Sladeczek, I., & Carlson, J. S. (2003). Conjoint consultation using self - administered manual and videotape parent teacher training: School Psychology Quarterly, 1 8, 269 302. doi: 10.1521/scpq.18.3.269.22574 Kollins, S. H., Sparrow, E., & Conners, C. K. (2010). Guide to assessment scales in attention - deficit/hyperactivity disorder . (2 nd ed.). London: Springer. 125 Kwan, B. M., & Nease, D. E. Talen, M. R. (2013). The state of the evidence for integrated behavioral health in primary care. In M. R. Talen and A. Burke Valeras (eds.), Integrated Behavioral Health in Primary Care: Evaluating the Evidence, Identifying the Essentials . doi: 10.1007/978 - 1 - 4614 - 6889 - 9_5 D. (2009). Are oppositional - defiant and hyperactive - inattentive symptoms developmental precursors to conduct problems in late childhood? Genetic and environmental links. Journal of Abnormal Child Psychology, 37 , 45 - 58. doi: 10.1007/s10802 - 008 - 9257 - 1 Lave, J. R., Frank, R. G., Schulberg, H. C., & Kamlet, M. S. (1998). Cost - effectiveness of treatments for major depression in primary care practice. Archives of General Psychiatry, 55 , 645 651. doi: 10.1001/archpsyc.55.7.645 Lee, C. M., Horvath, C., & Hunsley, J. (2013). Does it work in the real world? The effectiveness of treatments for psychol ogical problems in children and adolescents. Professional Psychology: Research and Practice, 44, 81 - 88 . doi: 10.1037/a0031133 Liu, C., Robin, A. L., Brenner, S., & Eastman, J. (1991). Social acceptability of methylphenidate and behavior modification for treating attention deficit hyperactivity disorder. Pediatrics, 88 , 560 - 565. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/1881737 Little, R. J. (1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83 , 1198 1202. doi:10.2307/2290157 Little, R. J., & Rubin, D. B. (1987). Statistical analysis with missing data . New York: Wiley Inc. Leslie, L. K., Rappo, P. D., Abelson, H. T., Sewall, S. R., & Jenkins, R. R. (2000). Final report of the FOPE II pediatric generalists of the future workgroup. Pediatrics, 106, 1199 1223. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/11073552 Leslie, L. K., Stallone, K. A., Weckerly, J., McDaniel, A. L., & Monn, A. (2006). Implementing ADHD guidelines in primary care: Does one size fit all? Journal of Health Care for the Poor and Underserved, 17, 302 327. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/16702717 Lerner, M., & Wigal, T. (2008). Long - term safety of stimulant medications used to treat children with A DHD. Journal of Psychosocial Nursing and Mental Health Service, 46 , 38 - 48. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/18777967 Lyon, A. R., Gershenson, R. A., Farahmand, F. K., Thaxter , P. J., Behling, S., & Budd, K. S. (2009). Effectiveness of teacher - child interaction training (TCIT) in a preschool setting. Behavior Modification, 33 , 855 - 884. doi: 10.1177/0145445509344215 126 Majewicz - Hefley, A., & Carlson, J. S. (2007). A meta - analysi s of combined treatments for children diagnosed with ADHD. Journal of Attention Disorders, 10 , 239 - 250. doi: 10.1177/1087054706289934 Malhi, P., Singhi, P., & Sidhu, M. (2008). Impact of parent and teacher concordance on diagnosing attention deficit h yperactivity disorder and its sub - types. Indian Journal of Pediatrics, 75 , 223 - 228. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/18376088 McCain, A. P., & Kelley, M. L. (1993). Managing the classroom behavior of an ADHD preschooler: The efficacy of a school - home note intervention. Child and Family Behavior Therapy, 15 , 33 44. doi: 10.1300/J01 9v15n03_03 McNeil, C. B., & Hembree - Kigin, T. L. (2011). Parent - child interaction therapy , 2nd edition. New York: Springer. McMenamy, J., Sheldrick, R., & Perrin, E. C. (2011). Early intervention in pediatrics offices for emerging disruptive behavior in toddlers. Journal of Pediatric Health Care, 25 , 77 - 86. doi: 10.1016/j.pedhc.2009.08.008 McGrath, R. E., & Moore, B. A. (Eds.). (2010). Pharmacotherapy for psyc hologists: prescribing and collaborative roles . Washington, DC: American Psychological Association. McCleary, L., & Ridley, T. (1999). Parenting adolescents with ADHD: Evaluation of a psychoeducation group. Patient Education and Counseling , 38 , 3 - 10. R etrieved from http://www.ncbi.nlm.nih.gov/pubmed/14528566 McIntosh, D. E., Rizza, M. G., & Bliss, L. (2000). Implementing empirically supported interventions: Teacher - child interaction therapy . Psychology in the Schools, 37 , 453 - 462. doi: 10.1002/1520 - 6807(200009)37:5 Meadows, T., Valleley, R., Haack, M. K., Thorson, R., & providing behavioral health in primary care. Clinical Pediatrics, 50 , 447 - 459. doi: 10.1177/0009922810390676 . (2010). Lifetime prevalence of mental disorders in US adolescents: Results from the National Comorbidity Survey Replication Adolescent Supplement (NCS - A). Journal of the American Academy of Child and Adolescent Psychiatry, 49 , 980 989. doi: 10.1016/j.jaac.2010.05.017 Mirman, D. (2014). Growth curve analysis and visualization using R. Boca Raton, FL: Chapman & Hall/CRC. Mitsis, E. M., McKay, K. E., Schulz, K. P., Newcorn, J. H., & Halperin, J. M. (2000). Parent - teacher concordance for DSM - IV attention - deficit/hyperactivity disorder in a clinic - 127 referred sample. Journal of the American Academy of Child and Adolescent Psychiatry , 39, 308 313. doi: 10.1097/00004583 - 200003000 - 00012 Morgan, P. L., Staff, J., Hillemeier, M. M., Farkas, G., Maczuga, S. (2013). Racial and ethnic disparities in ADHD diagnosis from kindergarten to eigh th grade. Pediatrics, 132 , 85 - 93. doi: 10.1542/peds.2012 - 2390 MTA Cooperative Group. (1999a). A 14 - month randomized clinical trial of treatment strategies for attention - deficit/hyperactivity disorder. The MTA Cooperative Group. Multimodal Treatment Stu dy of Children with ADHD. Archives of General Psychiatry, 56, 1073 - 1086. doi: 10.1001/archpsyc.56.12.1073 MTA Cooperative Group. (1999b). Moderators and mediators of treatment response for children with attention - deficit/hyperactivity disorder. Archives of General Psychiatry, 56, 1088 - 1096. doi: 10.1001/archpsyc.56.12.108 8 MTA Cooperative Group. (2004). National Institute of Mental Health multimodal treatment study of ADHD follow - up: 24 - month outcomes of treatment strategies for attention - deficit/ hyperactivity disorder. Pediatrics, 113 , 754 - 761. Retrieved from http://pediatrics.aappublications.org/content/113/4/754.full.html Mohammadi, M. R., & Akhondzadeh, S. (2007). Pharmacotherapy of attention - deficit/hyperactivity disorder: Non stimulant medication approaches. Expert Review of Neurotherapeutics, 7 , 191 - 205. doi: 10.1586/14737175.7.2.195 Molina, B. S., Flory, K., Hinshaw, S. P., Greiner, A. R., Arnold, L. E., Swanson, J. g substance use in the MTA at 36 months: Prevalence, course, and treatment effects. Journal of the American Academy of Child and Adolescent Psychiatry, 46 :1028 1040. Retrieved from http://www. ncbi.nlm.nih.gov/pubmed/17667481 Cooperative Group. (2009). The MTA at 8 years: Prospective follow - up of children treated for combined - type ADHD in a multisit e study. Journal of the American Academy of Child and Adolescent Psychiatry, 48 , 484 - 500. doi: 10.1097/CHI.0b013e31819c23d0 Muthén, B. O., & Curran, P. (1997). General longitudinal modeling of individual differences in experimental designs: A latent variable framework for analysis and power estimation. Psychological Methods, 2 , 371 - 402. Retrieved from http://www.statmodel.com/bmuthen/Article_071.pdf Na tional Association of School Psychologists (2002). Helping students with ADHD in the classroom: Information for teachers. Retrieved from http://www.nasponline.org/resources/ handouts/revisedPDFs/adhd.pdf 128 National Institutes of Mental Health. (2013). Attention Deficit Hyperactivity Disorder in Children and Adolescents Fact Sheet . Retrieved from http://www.nimh.nih.gov/health/publications/attention - deficit - hyperactivity - disorder - in - children - and - adolescents/index.shtml (2005). Treatment implemen tation following behavioral consultation in schools: A comparison of three follow - up strategies School Psychology Review, 34 , 87 - 106. Retrieved from http://eric.ed.gov/?id=EJ683507 Northup, J., Fusilier, I., Swanson, V., Huete, J., Bruce, T., Freeland, J., Gulley, V., & Edwards, S. (1999). Further analysis of the separate and interactive effects of methylphenidate and common classroom contingencies. Journal of Applied Behavior Analysis, 32, 35 50. doi : 10.1901/jaba.1999.32 - 35 Obama, B. (September 9, 2009). . The White House Office of the Press Secretary. The Patient Protec tion and Affordable Care Act [PPACA] [P.L. 111 - 148] (HR 3590). quadratic growth models for continuo us and dichotomous outcomes. In Y. Petscher, C. Schatschneider, & D. L. Compton (Eds.), Applied quantitative analysis in the education and the social sciences . New York, NY: Routledge. Offord, D. R., & Bennett, K. J. (1994). Conduct disorder: Long term outcomes and intervention effectiveness. Journal of the American Academy of Child and Adolescent Psychiatry, 33 , 1069 - 1078. hyperkinetic children. Clinical Pe diatrics, 15, 510 515. doi: 0.1177/000992287601500603 Restoring Psychotherapy as the First Line Int ervention in Behavioral Care , (pp. 1 - 30). New York: Ithaca Press. Owens, J. S., Murphy, C. E., Richerson, L., Girio, E. L., & Himawan, L. K. (2008). Science to practice in underserved communities: The effectiveness of school mental health programming. Journal of Clinical Child and Adolescent Psychology, 37 , 434 447. doi: 10.1080/15374410801955912 Owens, J. S., Holdaway, A. S., Zoromski, A. K., Evans, S. W., Himawan, L. K., Girio - Herrera, E., & Murphy, C. E. (2012). Incremental benefits of a daily rep ort card intervention over time for youth with disruptive behavior. Behavior Therapy, 43 , 848 - 861. Retrieved from www.sciencedirect.com/science/article/pii/S000578941200024X 129 Pastor, P. N., & Reuben, C. A. (2008). Diagnosed attention deficit hyperactivit y disorder and learning disability: United States, 2004 2006. Vital Health Statistics, series 10, no 237. Hyattsville, MD, Centers for Disease Control and Prevention, National Center for Health Statistics, July 2008. Retrieved from www.cdc.gov/nchs/data/series/sr_10/Sr10_237.pdf Patterson, G. R. (1976). Living with children: New methods for parents and teachers. (Rev. ed.). Champaign, IL: Research Press. Patterson, G. R., Dishion, T. J., & Chamberlain, P. (1993). Outcomes and methodological issues relating to treatment of antisocial children. In T. R. Giles (Ed.), Handbook of effective psy chotherapy (pp. 43 - 88). New York: Plenum Press. Patterson, G., & Chamberlain, P. (1994). A functional analysis of resistance during parent training therapy. Clinical Psychology: Science and Practice, 1 , 53 - 70. doi: 10.1111/j.1468 - 2850.1994.tb00006.x P elham, W. E., Wheeler, T., & Chronis, A. (1998). Empirically supported psychosocial treatments for attention deficit hyperactivity disorder. Journal of Clinical Child Psychology, 27 , 190 - 205. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/9648036 Pelham, W. E., Milich, R., & Walker, J. (1986). The effects on continuous and partial reinforcement and methylphenidate on learning in children with attention deficit disorder. Journal of Abnormal Psychology, 95 , 319 - 325. Retrieved from http://uknowledge.uky.edu/psychology_facpub/82/ treating ADHD Restoring Psychotherapy as the First Line Intervention in Behavioral Care . (pp. 297 - 325). New York: Ithaca Press. Pelham, W. E. (2007). Against the grain: A proposal for a psychosocial - first approach to treatin g ADHD the Buffalo treatment algorithm. In K. McBurnett, & L.J. Pfiffner (Eds . ) , Attention Deficit/Hyperactivity Disorder: Concepts, Controversies, New Directions (pp. 301 - 316). New York: Informa Healthcare. Pelham, W. E., Schnedler, R.W., Bender, M.E., Miller, J., Nilsson, D., Budrow, M., Ronnei, M., Paluchowski, C., & Marks, D. (1988). The combination of behavior therapy and methylphenidate in the treatment of attention deficit disorders: A therapy outcome study. In L. Bloomingdale (Ed.) Attention deficit disorders III: New research in attention, treatment and psychopharmacology (pp. 29 - 48). London: Pergamon. Pelham, W. E., Schnedler, R., Bologna, N., & Contreras, A. (1980). Behavioral and stimulant tre atment of hyperactive children: A therapy study with methylphenidate probes in a within - subject design. Journal of Applied Behavior Analysis, 13, 221 - 236. doi: 10.1 901/jaba.1980.13 - 221 130 Wigal, T. Parent and teacher evaluation of treatment in the MTA: Consumer satisfaction and perceived effectiveness. (submitted for publicatio n). Journal of Consulting and Clinical Psychology. Pelham, W. E., & Smith, B. H. (2000). Prediction and measurement of individual responses to Ritalin by children and adolescents with ADHD. In L. Greenhill & B. P. Osman (Eds.), Ritalin: Theory and prac tice (2 nd ed., pp. 193 217). New York: Mary Ann Liebert. McBurnett, K. (2000). Behavioral versus behavioral and pharmacological treatment in ADHD children attending a summer treatment program. Journal of Abnormal Child Psychology, 28 , 507 - 525. doi: 10.1023/A:1005127030251 Pelham, W. E., Foster, E. M., & Robb, J. A. (2007). The eco nomic impact of ADHD in children and adolescents. Ambulatory Pediatrics, 7, 121 131. doi: 10.1093/jpepsy/jsm022 Pelham, W. E., Pillow, D. R., Kipp, H. L., Greiner, A. E. (2002). Effects of methylphenidate and expectancy on children with ADHD: Behavior, academic performance, and attributions in a summer treatment program and regular classroom settings. Journal of Consulting an d Clinical Psychology, 70 , 320 - 335. doi: 10.1037/0022 - 006X.70.2.320 Pelham, W. E., & Fabiano, G. A. (2008). Evidence - based psychosocial treatment for ADHD: An update. Journal of Clinical Child and Adolescent Psychology, 37, 184 214. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/18444058 Pelham, W. E. & Waschbusch, D. (1999). Behavioral interventi on in attention - deficit/hyperactivity disorder. In H. Quay & A. Quay (Eds.) Handbook of disruptive behavior disorders. New York: Plenum Press. Pelham, W. E., Wheeler, T., & Chronis, A. (1998). Empirically supported psychosocial treatments for attention deficit hyperactivity disorder. Journal of Clinical Child Psychology, 27 , 190 205. doi: 10.1207/s15374424jccp2702_6 Pfiffner, L. J., Yee, M. A., Huang - Pollock, C., Easterlin, B., Zalecki, C., & McBurnett, K. (2007). A randomized, controlled trial of integrated home school behavioral treatment for ADHD, predominantly inattentive type. Journal of the American Academy of Child and Adolescent Psychiatry, 46 , 1041 - 1050 . doi: 10.1097/chi.0b013e318064675f Pham, A. V., Carlson, J. S., & Kosciulek. J. F. (2009). Ethnic differences in parental beliefs of attention - deficit/hyper activity disorder and treatment. Journal of Attention Disorders, 13 , 584 - 591. doi: 10.1177/1087054709332391 131 Phelps, L., Brown, R. T., & Power, T. J. (2002). Pediatric psychopharmacology: Combining medical and psychosocial interventions . Washington, DC: American Psychological Association. Pisterman, S., McGrath, P., Firestone, P., Goodman, J. T., Webster, I., & Mallory, R. (1989). Outcome of parent - mediated treatment of preschoolers with attention deficit disorder with hyperactivity. Journal of Consulting and Clinical Psychology , 57 , 628 - 635. doi: 10.1037/0022 - 006X.57.5.628 Pliszka, S. R., Crismon, M. L., Hughes, C. W., Conners, K. C., Emslie, G. J., Jens en., P. Revision of the Algorithm for Pharmacotherapy of Attention - Deficit/Hyperactivity Disorder. Journal of the American Academy of Child and Adolescent Psychiatry, 45, 642 - 6 57. doi: 10.1097/01.chi.0000215326.51175.eb Pliszka, S. R., Lopez, M., Crismon, L., Toprac, M. G., Hughes, C. W., Emslie, G. J., & Boemer, algorithm for the treatment o f ADHD. Journal of the American Academy of Child and Adolescent Psychiatry , 42, 279 287. Power, T. J., Hess, L. E., & Bennett, D. S. (1995). The acceptability of interventions for attention - deficit hyperactivity disorder among elementary and middle sch ool teachers. Developmental and Behavioral Pediatrics, 16 , 238 - 243. doi: 10.1097/00004703 - 199508000 - 00005 Power, T. J., Soffer, S. L., Mautone, J. A., Cost igan, T. E., Jones, T. E., Clarke, A. T., & Marshall, S. A. (2009). An analysis of teacher investment in the context of a family - school intervention for children with ADHD. School Mental Health, 1 , 107 - 117. Doi: 10.1007/s12310 - 009 - 9005 - x Pyne, J. M., Rost, K. M., Zhang, M., Williams, D. K., Smith, J., & Fortney, J. (2003). Cost - effectiveness of a primary care depression intervention. Journal of General Internal Medicine, 18, 432 441. doi: 10.1046/j.1525 - 1497.2003.20611.x Qui, M., Ye, Z., Li, Q., Liu, G., Xie, B., & Wang, J. (2011). Changes of brain structure and fu nction in ADHD children. Brain Topography, 24 , 243 - 252. doi: 10.1007/s10548 - 010 - 0168 - 4 Rappley, M. D., Gardiner, J. C., Jetton, J. R., & Houang, R. T. (1995). The use of methylphenidate in Michigan. Archives of Pediatrics & Adolescent Medicine, 149 , 675 - 9. doi:10.1001/archpedi.1995.02170190085015 Raudenbush, S. W. (2011). Optimal Design Software for Multi - level and Longitudinal Research (Version 3.01) [Software]. Available from www.wtgrantfoundation.org . 132 Reiff, M. I., & Stein, M. T. (2011). Attention - deficit/hyperactivity disorder. In R. R. Voight & S. M. Myers (Eds.) Developmental and Behavioral Pediatrics , (pp. 327 - 348). American Academy of Pediatrics. Regier, D. A., Narro w, W. E., Rae, D. S., Manderscheid, R. W., Locke, B., & Goodwin, F. (1993). The de facto US mental and addictive disorders service system: Epidemiologic Catchment Area prospective 1 - year prevalence rates of disorders and services. Archives of General Ps ychiatry, 50, 85 94. doi:10.1001/archpsyc.1993.01820140007001 Reynolds, C.R., & Kamphaus, R.W. (2005). Behavior assessment system for children (2nd ed.). Circle Pines, MN: American Guidance Service. Reynolds, C. R., & Kamphaus, R. W. (2009). Behavior as sessment system for children (2 nd ed.). Progress Monitor . Pearson Assessment. Riley - to using daily behavior report cards to monitor student behavior. Psychology i n the Schools, 44, 77 89. doi: 10.1002/pits.20207 Rushton, J. L., Fant, K. E., & Clark, S. J. (2004). Use of practice guidelines in the primary care of children with attention - deficit/hyperactivity disorder. Pediatrics, 114, 23 28. Retrieved from http://pediatrics.aappublications.org/content/114/1/e23.abstract Safer, D. J., & Allen, R. P. (1973). Factors influencing the suppressant effects of two stimulant drugs on the gr owth of hyperactive children. Pediatrics, 51 , 660. Retrieved from http://pediatrics.aappublications.org/content/51/4/660 Serby, M., Schmeidler, J., & Smith, J. (2002). Length of psychiatry clerkships: Recent changes and the relationship to recruitment. Academic Psychiatry, 26 , 102 - 104. doi: 10.1176/appi.ap.26.2.102 Schachter, H. M., Pham, B., King, J., Langford, S., & Moher, D. (2001). How efficacious and safe is short acting methylphenidate for the treatment of attention - deficit disorder in children and adolescents: A meta - analysis. Canadian Medical Association Journal, 165 , 1475 - 1488. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/11762571 Schell, R. M., Pelham, W. E., Bender, M. E., & Andree, J. A. (1986). The concurrent assessment of behavioral and psychostimulant interventions: A controlled case study. Behavioral Assessment, 8 , 373 - 384. Retri eved from http://ezproxy.msu.edu.proxy1.cl.msu.edu/login?url=http://search.proquest.com.proxy1.c l.msu.edu/docview/617374022?accountid=12598 Schoenbaum, M., Unutzer, J., Sherbourne, C., Duan, N., Rubenstein, L., Miranda, J., & Meredith, L. S. (2001). Co st - effectiveness of practice initiated quality improvement for depression. Journal of the American Medical Association, 286, 1325 - 1330. doi: 10.1001/jama.286.11.132 5 133 Staller, J., & Faraone, S. V. (2006). Attention deficit hyperactivity disorder in girls: Epidemiology and management. CNS Drugs, 20 , 107 - 123. doi: 10.2165/ 00023210 - 200620020 - 00003 Schmidt, N., & Woolaway - Bickel, K. (2000). The effects of treatment compliance on outcome in cognitive - behavioral therapy for panic disorder: Quality versus quantity. Journal of Consulting and Clinical Psychology, 68 , 13 - 18. doi: 10.1037/0022 - 006X.68.1.13 Seay, H. A., Fee, V. E., & Holloway, K. S. (2003). A multicomponent treatment package to increase anger control in teacher referred b oys . Child and Family Behavior Therapy, 25 , 1 - 18. doi: 10.1300/J019v25n01_01 Sherman, M., & Hertzig, M. E. (1991). Prescribing practices of Ritalin: The Suffolk County, New York study. In L. L. Greenhill & B. B. Osman (Eds.), Ritalin: Theory and patient management (pp. 187 - 193). New York: Liebert. (2001). Cost - effectiveness of a collaborative care program for primary care patients with persistent depression. American Journal of Psychiatry, 158, 1638 1644. doi: 10.1176/appi.ajp.158.10. 1638 Singer, J. D. & Willett, J. B. (2003). Applied longitudinal data analysis. Oxford, England: Oxford University. Spirrison, C. L., Noland, K. A., & Savoie, L. B. (1992). Factor structure of the treatment evaluation inventory: Implications for measur ement of treatment acceptability. Journal of Psychopathology and Behavioral Assessment, 14 , 65 - 79. doi: 10.1007/BF00960092 Springer, C., & Reddy, L. A. (2004). Measuring a dherence in behavior therapy: Opportunities for research and practice. The Behavior Therapist, 27 , 1 - 9. Springer, C., & Reddy, L. A. (2010). Measuring parental treatment adherence in a multimodal treatment program for children with ADHD: A preliminary investigation. Child and Family Behavior Therapy, 32 , 272 - 290. Steele, R. G., & Roberts, M. C. (2003). Therapy and interventions research with children and adolescents. In M. C. Roberts & S. S. Ilardi (Eds.), Handbook of research methods in clinical psychology (pp. 307 - 326). London: Blackwell Publishers. Strosahl, K. (1999). The psychologist in primary health care. In A. J. Kent & M. Hersen (Eds.), (pp. 87 112). Mahwah, NJ: Erlbaum. St rosahl, K. (2005). Training behavioral health and medical providers for integrated care: A core 134 (Eds.). Behavioral Integrative Care: Treatments that Work in the Primary Care Sett ing . New York: Brunner - Routledge. behavioral control depend on which disorder and which drug. Clinical Pediatrics , 26 , 258 - 263. Swanson, J. M., Lerner, M. A., Mar ch, J., & Gresham, F. M. (1999). Assessment and intervention for attention - deficit/hyperactivity disorder in the schools: Lessons from the MTA study. Pediatric Clinics of North America, 46 , 993 - 1009. Swanson, J. M., & Volkow, N. D. (2009). Psychopharmacology: Concepts and opinions about the use of stimulant medications. Journal of Child Psychology and Psychiatry, 50 , 180 - 193. doi: 10.1111/j.1469 - 7610.2008.02062.x. Swanson, J. M., Kraemer, H. C., Hinshaw, S. P., Arnold, E. L., Conn ers, C. K., Abikoff, H. rates based on severity of ADHD and ODD symptoms at the end of treatment. Journal of the American Academy of Child and Adolescent Psychiatry, 40 , 1 68 - 179. doi:10.1097/00004583 - 200102000 - 00011 N. D. (2007). Effects of stimulant medication on growth rates across 3 years in the MTA follow - u p. Journal of the American Academy of Child and Adolescent Psychiatry, 46 , 1015 1027. doi:10.1097/chi.0b013e3180686d7e Talen, M. R., & Valeras, A. B. (2013). Introduction and overview of integrated behavioral health in primary care (p. 3 - 4) In M. R. Tale n and A. Burke Valeras (Eds.), Integrated Behavioral Health in Primary Care: Evaluating the Evidence, Identifying the Essentials , doi: 10.1007/978 - 1 - 4614 - 6889 - 9_1 Toomey, S. L., Sox, C. M., Rusinak, D., & Finkelstein, J. A. (2012). Why do children with A DHD discontinue their medication. Clinical Pediatrics, 51 , 763 - 769. doi: 10.1177/0009922812446744 Udow - Phillips, M., Ogundimu, T., Ehrlich, E., Kofke - Egger, H. & Stock, K. (2010). CHRT: Health Care Variation in Michigan, 26 - 29. Ann Arbor, MI: Center for Healthcare Research & Transformation. Retrieved from http://www.chrt.org/ Ulrich, C. M., Danis, M., Kosiol, K., Garrett - Mayer, E., Hubbard, R., & Gra dy, C. (2005). Does it pay to pay? A randomized trial of prepaid financial incentives and lottery incentives in surveys of nonphysician professionals. Nursing Research, 54 , 178 - 183. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/15897793 U.S. Depart ment of Education, Office of Special Education and Rehabilitative Services, Office of Special Education Programs. (2004). Teaching children with attention - 135 deficit/hyperactivity disorder: Instructional strategies and practices. Washington, DC: U.S. Depar tment of Education. Retrieved from http://www2.ed.gov/rschstat/research/pubs/adhd/adhd - teaching - 2006.pdf DHD, treatments for ADHD, and Treatment acceptability: An initial investigation. School Psychology Review, 33 , 421 - 428. Retrieved from http://www.nasponline.org/publications /spr/abstract.aspx?ID=1715 Vile Junod, R. E., DuPaul, G. J., Jitendra, A. K., Volpe, R. J., & Cleary, K. S. (2006). Classroom observations of students with and without ADHD: Differences across types of engagement. Journal of School Psychology, 44 , 87 - 104. doi: 10.1016/j.jsp.2005.12.004 Vitiello, B., Severe, J. B., Greenhill, L. L., Arnold, E. L., Abikoff, H. B., Bukstein, O. Cantwell., D. P. (2001). Methylphenidate dosage for children with ADHD over time under controlled conditions: Lessons from the MTA. Journal of the American Academy of Child and Adolescent Psychiatry, 40, 188 - 196. doi: 10.1097/00004583 - 200102000 - 00013 Vujnovic, R. K. (2009). Examining the influence of treatment integrity: Accuracy of daily report card intervention implementation and student outcome. Disser tation. Waschbusch, D. A., Cunningham, C. E., Pelham Jr., W. E., Rimas, H, L., Greiner, A. R., Gnagy, preferences for treatment of young, medication naïve children with A DHD. Journal of Clinical Child and Adolescent Psychology, 40 , 546 - 561. doi: 10.1080/15374416.2011.581617 Warzak, W. J., & Floress, M. T. (2009). Time - out traini ng without put - backs, spanks, or restraint: A brief report of deferred time - out. Child and Family Behavior Therapy, 3 , 134 - 143. doi: 10.1080/07317100902910570 Webs ter - Stratton, C. (1985). Predictors of treatment outcome in parent training for conduct disordered children. Behavioral Therapy, 16 , 223 - 243. doi: 10.1016/S0005 - 7894(85)80048 - 4 Webster - Stratton, C., & Hammond, M. (1997). Treating children with early - onset conduct problems: A comparison of child and parent training interventions. Journal of Consulting and Clinical Psychology, 65 , 93 - 109. doi: 10.1037/0022 - 006X.65.1.93 Weisz, J. R., Huey, S. J., & Weersing, V. R. (1998). Psychotherapy outcome research with children and adolescents: The state of the art. In T. H. Ollendick & R. J., Prinz (Eds.), Advances in Clinical Child Psychology, 2 , 49 - 91. New York: Plenum. Schiller, E. (2000). Psychosocial treatment strategies in the MTA study: Rationale, 136 methods, and critical issues in design and implementation. Journal of Abnormal Child Psychology, 28, 483 505. doi: 10.1023/A:1005174913412 Wey andt, L., Swentosky, A., Gyda Gudmundsdottir, B. (2013). Neuroimaging and ADHD: fMRI, PET, DTI findings and methodological limitations. Developmental Neuropsychology, 38 , 211 - 225. doi: 10.1080/87565641.2013.783833 Williams, J., Klinepeter, K., Palmes, G ., Pulley, A., & Foy, J. M. (2004). Diagnosis and treatment of behavioral health disorders in pediatric practice. Pediatrics, 114 , 601 - 606. doi: 10.1542/peds.2004 - 0090 ents for attention - deficit hyperactivity disorder. Journal of Attention Disorders , 1 , 114 - 121. doi: 10.1177/108705479600100204 Winterstein, A. G., Gerhard, T., Sh uster, J., Zito, J., Johnson, M., Liu, H., & Saidi, A. (2008). Utilization of pharmacologic treatment in youth with attention deficit/hyperactivity disorder in Medicaid database. Annals of Pharmacotherapy, 42 , 24 - 31. doi: 10.1345/aph.1K143 Witt, J. C., & Elliott, S. N. (1985). Acceptability of classroom intervention strategies. In T. R. Kratochwill (Ed.), Advances in school psychology (Vol. 4, pp. 251 - 288). Hillsdale, NJ: Lawrence Erlbaum. Witt, J. C., & Robbins, J. R. (1985). Acceptability of reductive interventions for the control of inappropriate child behavior. Journal of Abnormal Child Psychology, 13 , 59 - 67. doi: 10.1007/BF00918372 Zarin, D., Suarez, A., Pincus, H., Kupersanin, E., & Zito, J. (1998). Clinical and treatment characteristics of children with attention - deficit/hyperactivity disorder in psychiatric practice. Journal of the American Academy of Child and Adolescent Psychiatry , 3 7 , 1262 - 1270. doi: 10.1097/00004583 - 199812000 - 00009