TEACHER MOTIVATION AND PERSPECTIVES ON IMPLEMENTING A TIER-1 READING INTERVENTION WITH TREATMENT INTEGRITY By Amanda Lee Hall A DISSERTATION Submitted to Michigan State University In partial fulfillment of the requirements for the degree of School Psychology- Doctor of Philosophy 2013       ABSTRACT TEACHER MOTIVATION AND PERSPECTIVES ON IMPLEMENTING A TIER-2 READING INTERVENTION WITH TREATMENT INTEGRITY By Amanda Lee Hall The adoption of evidence-based interventions (EBI) is increasingly being required in school settings due in large part to federal education policies, which mandate the use of scientifically-based practices. Authentic settings present challenges to adopting EBIs due to their complex nature, making implementation with treatment integrity is often difficult to achieve in community settings. This study examined elementary school teachers’ implementation of a specific EBI in reading through the lens of teacher motivation. The study used a survey and focus groups with teachers to explore the problems and prospects surrounding the implementation of an EBI. Student outcome data in reading were evaluated to examine how treatment integrity, teacher motivation, and student achievement are correlated. Teacher motivational beliefs were related to some dimensions of treatment integrity, but higher treatment integrity was not related to student achievement. In the focus groups, teachers discussed EBIs in relation to school practices (e.g., response to intervention), their pedagogical beliefs, and student characteristics and their experiences with implementing Reading Street, the program adopted by a intermediate school district as their primary reading curriculum. A major goal of this study was to examine the alignment between the mandate to use EBIs and their actual use in practice. Teachers described the challenge of balancing the use of Reading Street with fidelity with the needs of individual students. By developing a deeper understanding of teachers’ perspectives on   the implementation of EBIs, school psychologists may be better able to support teachers to promote the effective use of EBIs in schools.   ACKNOWLEDGEMENTS I would like to acknowledge my advisor Dr. Evelyn Oka and my entire committee for their support with this project. Their knowledge and expertise guided this work and without it I would have been lost. I would also like to acknowledge the teachers who gave up their valuable time to participate in the study. Without their insightful comments and thoughts on what it is like to use evidence-based interventions in classrooms, I would not have had a project. Finally, I would like to thank my friends and family for supporting me through the graduate school process. I would not be where I am today without them. iv     TABLE OF CONTENTS LIST OF TABLES………………………………………………………………………...……viii Chapter 1 Introduction………………………………………………………………………………………..1 Statement of the Problem………………………………………………………………….1 Purpose and Significance of the Study……………………………………………………6 Theoretical and Conceptual Framework………………………………………………….6 Conceptualizations of Treatment Integrity……………………………………….7 Research Questions……………………………………………………………………….8 Hypotheses………………………………………………………………………………..8 Overview…………………………………………………………………………………10 Chapter 2 Literature Review…………………………………………………………………….…………..11 Expectancy-Value Theory Overview…………………………………………………….11 Expectancy……………………………………………………………………………….13 Self-Efficacy……………………………………………………………………..13 Perception of task-demands…………………………………………………...…15 Attributions for previous successes and failures…………………………………16 Self-fulfilling prophecy…………………………………………………………..16 Value……………………………………………………………………………………..17 Attainment value…………………………………………………………………17 Intrinsic value…………………………………………………………………….18 Utility value……………………………………………………………………...19 Cost………………………………………………………………………………19 Teacher Motivation………………………………………………………………………19 Evidence-Based Intervention Movement in Education………………………………….21 Defining Implementation, Fidelity, and Treatment Integrity……………………………22 Implementation Fidelity…………………………………………………………23 Defining Treatment Integrity……………………………………………………24 The Relationship between Implementation, Treatment Integrity, and Outcomes………26 Implementation and outcomes…………………………………………………..27 Treatment integrity and outcomes………………………………………………29 Studies demonstrating mixed or lack of relationship between treatment integrity and outcomes…………………………………………………..30 Studies demonstrating a positive relationship between treatment integrity and outcomes……………………………………………………………32 Teacher Factors in the Implementation of EBIs with Treatment Integrity………………33 Teacher characteristics and perceptions…………………………………………34 Motivational factors to implement educational innovations…………………….36   Chapter 3 v     Methods……...……………………………………………………………………………..…….42 Participants……………………………………………………………………….………42 Research Design…………………………………………………………………….……44 Data Sources………………………………………………………………….……….....44 Evidence-Based Intervention: Reading Street………………………………...………...45 Variables and Measures……………………………………………………………...…..47 Teacher views of evidence-based intervention…………………………………..48 Motivation………………………………………………………………...……...48 Expectancy……………………………………………………………….48 Value……………………………………………………………………..48 Cost………………………………………………………………………49 Treatment integrity………………………………………………………………49 Adherence………………………………………………………………..49 Participant responsiveness……………………………………………….51 Exposure…………………………………………………………………51 Teacher understanding of treatment integrity……………………………51 Curriculum-based measures……………………………………………………...51 Procedures……………………………………………………………………..…………52 Survey data………………………………………………………………….……52 Focus groups…………………………………………………….……………….53 Chapter 4 Results……...…………………………………………………………………………..………...56 Data Analyses……………………………………………………………………………56 Descriptive statistics……………………………………………………………..56 Pearson’s R Correlation………………………………………………………….56 Qualitative analysis………………………………………………………………56 Rationale for qualitative methods………………………………………..56 Site selection……………………………………………………………..58 Researcher’s role…………………………………………………………58 Data collection methods………………………………………………….59 Data management………………………………………………………...59 Data analysis strategy……………………………………………………59 Trustworthiness features…………………………………………………60 Missing Data……………………………………………………………………..62 Research Question 1: What are teachers’ knowledge of and experiences with implementing evidence-based interventions?....................................................................63 Assessment………………………………………………………………………63 Response-to Intervention………………………………………………………...64 Evidence-based Interventions in Classroom Practice……………………………65 Implementation Barriers…………………………………………………………67 Treatment Integrity………………………………………………………………73 Research Question 2: What are teachers’ knowledge of and experiences with implementing Reading Street?..........................................................................................74 Teacher Factors Influencing Beliefs and Perceptions of Reading Street………..74 Systems Factors Influencing Beliefs and Perceptions of Reading Street………..77 vi     Benefits and Challenges of Reading Street……………………………………....77 Research Question 3: What is the teachers’ level of motivation to use Reading Street?..78 Influences on Expectations……………………………………………………....79 Benefits and Challenges of Reading Street………………………………………80 Research Question 4: What is the teachers’ understanding of treatment integrity? Do they view treatment integrity as unidimensional or multidimensional?....................................81 Research Question 5: How are motivation, treatment integrity, and student learning related?...............................................................................................................................86 How is teacher motivation related to treatment integrity?.....................................86 How is treatment integrity related to students’ reading achievement?..................87 How is teacher motivation related to students’ reading achievement?..................88 Chapter 5 Discussion……..………………………………………………………………………...……….89 Teacher and System Factors Influencing Perceptions and Beliefs………………………89 Evidence-based intervention……………………………………………………..89 Reading Street……………………………………………………………………95 Expectancy and Value………………………………………………………..….97 Treatment Integrity……………………………………………………………..101 Teacher Motivation to Implement Reading Street through Expectancy-Value Theory..102 Perceptions of Treatment Integrity (or the “F” Word)………………………………….105 Relationship between Teacher Motivation and Treatment Integrity…………………...108 Relationship between Teacher Motivation and Student Outcomes…………………….109 Relationship between Treatment Integrity and Student Outcomes…………………….110 Conclusions…………………………………………………………………………….112 Limitations……………………………………………………………………………..114 APPENDICES………………..………………………………………………………………...118 Appendix A: Teacher Motivation Survey………………………………………………119 Appendix B: Questioning Route ……………………………..………………...………122 Appendix C: Reading Street Teacher Activity Logs……………………………..…….124 Appendix D: Aimsweb and DIBELS Cut-off Scores……………………..……………129 Appendix E: Recruitment Letters……...…….…………………………………………130 Appendix F: Consent Forms………………….……………………………………...…134 BIBLIOGRAPHY………………………………………………………………………………138 vii       LIST OF TABLES Table 1 Regression Coefficients for Abrami et al. (2004) and Wozney (2006)………………….………38 Table 2 Demographic Information of Focus Group Participants…………………………………..……..43 Table 3 Data Sources and Data Analyses by Research Question……………….………………………..61 Table 4 Frequency of Mentions of Implementation Barriers……………………………………………..68 Table 5 Descriptive Statistics for Motivational Beliefs…………………………………………………..78 Table 6 Number of Comments Involving Aspects of Treatment Integrity……………………………….82 Table 7 Descriptive Statistics for Aspects of Treatment Integrity………………………………………..86 Table 8 Correlations between Motivation, Treatment Integrity, and Student Achievement.…………….87 Table 9 Correlation Coefficients Between Treatment Integrity Variables and Student Outcomes……….…………………………………………………………………………….….87 viii     Chapter 1 Introduction Statement of the Problem The evidence-based practice movement is relevant to both the fields of education and psychology. This movement began in the 1990’s in the field of medicine. Sackett, Rosenberg, Gray, Haynes, and Richardson (1996) defined evidence-based medicine as having three components- clinical expertise, research evidence, and patient characteristics. As this movement evolved in medicine, the field of psychology also began to consider how evidence-based practice applied to its profession. In 2005, the president of the American Psychological Association formed a task-force on evidence-based practice in psychology (EBPP). This taskforce has defined evidence-based practice in psychology as, “the integration of the best available research with clinical expertise in the context of patient characteristics, culture, and preferences (APA Presidential Task Force on Evidence-Based Practice, 2006, p. 273). Although there has been controversy with the EBPP movement, it has been influential in the way that psychology is practiced and in the policies guiding mental health services. Following in the footsteps of medicine and psychology, the field of education has also been affected by the evidence-based practice movement; however, the roots of the movement in this field are different than in the other professions. In medicine and psychology, the evidencebased practice movement gained momentum through professional organizations forming task forces and groups (APA Presidential Task Force on Evidence-Based Practice, 2006; Chambless & Hollon, 2001; Kratochwill & Shernoff, 2003), but in education, the movement has been largely imposed on the field from federal policies. Both the No Child Left Behind Act in 2001 and the Individual with Dabilities Education Improvement Act in 2004 specifically reference the 1   need for educators to use research-based practices. Although the terminology is slightly different, the call for the use of scientifically based practices and the evidence-based practice movement on the conceptual level is very similar. An important aspect of the evidence-based practice movement is using specific evidence-based interventions that have been demonstrated through research to be efficacious. NCLB requires the use of instructional programs, strategies, and methods that are supported with scientific research. In psychology, the term empiricallysupported treatment or evidence-based intervention is used to describe a specific intervention that has research support (Kratochwill & Stoiber, 2002). Thus, although different terms are used, the underlying concept of specific programs or interventions having research support for their use is the same. Despite the mandate requiring the use of research-based practices, their use in schools has remained infrequent in special education (Burns & Ysseldyke, 2009), in drug use prevention curricula (Ringwalt, et al. 2009), and in supporting student behavior (Sugai & Horner, 2006) In addition to the infrequent use of evidence-based interventions, the level of treatment integrity of EBIs when they are implemented also poses challenges to the evidence-based practice movement. Treatment integrity is a vital part of the evidence-based practice movement as this construct refers to implementing interventions or practices “as intended” (Gresham, 1989). The phrase “implementation as intended” can have several meanings, such as implementing an intervention as intended by a consultant, implementing an intervention as intended from a manual, or implementing an intervention as intended by the researchers who validated it. Without implementing an evidence-based practices as intended, the validity of the practice is undermined because variability has entered the process of implementation. Therefore, it can no longer be determined whether the progress or lack of progress that a group of students may 2   experience is due to the EBI, its improper implementation, or to other factors. Failing to implement an EBI with integrity threatens to undermine the very basis of the evidence-based practice movement. A survey of nationally certified school psychologists revealed that although 97.6% of those surveyed believed treatment integrity to be important to measure, only 11.3% of the sample reported measuring treatment integrity on a regular basis (Cochrane & Laux, 2008). For the evidence-based practice movement to achieve the goal of improving student outcomes, treatment integrity is an area that needs to receive more attention. Treatment integrity is a construct that has been narrowly defined and often ignored in the past (Gresham, Gansle, and Noell, 1993; McIntyre, Gresham, DiGennaro & Reed, 2007). More recently scholars have been considering treatment integrity as a multidimensional construct; however, there is little work that validates these dimensions (Sanetti & Kratochwill, 2009). Considering treatment integrity as a more comprehensive construct is important within the EBP movement as it enables educators to have a more complete understanding of how an intervention is being implemented. Given that treatment integrity involves both the content and the process of delivery of an intervention (Perepletchikova & Kazdin, 2006), it seems logical that higher treatment integrity would result in higher outcomes for students. Interestingly, however, the relationship between treatment integrity and outcomes is mixed. Evidence-based interventions have the potential to improve student outcomes in academic, behavioral, and social realms, and there has been a plethora of research that focuses on validating EBIs and establishing empirical support for specific interventions. However, there is less research about the implementation of EBIs in practice settings. The available research examining the transportability of EBIs from highly controlled research settings to practice settings has suggested that it is more difficult to 3   implement EBIs in real world settings, which can lead to lower levels of treatment integrity (Henggeler, Melton, Brondino, Scherer, & Hanley, 1997). There is also the potential for interventions that have been demonstrated to be efficacious in controlled research settings to be implemented incorrectly in real world settings. This is problematic as some research has shown that higher levels of intervention implementation and treatment integrity result in better outcomes for students (Benner, Nelson, Stage & Ralston, 2010; Hirschstein, Van Schoiack Edstrom, Frey, Snell & MacKenzie, 2007; O’Donnell, 2008; Ysseldyke & Bolt, 2007). Although there is research that supports the relation between higher treatment integrity and better student outcomes, there is other research that indicates more mixed findings (Dane & Schneider, 1998; Lee, Penfield & Maerten-Rivera, 2010; Perepletchikova and Kazdin, 2005; Sterling-Turner, Watson & Moore, 2003; Wickstrom, Jones, LaFleur and Witt, 1998). Many factors have been implicated to explain these conflicting findings; however, there is not conclusive evidence to explain how treatment integrity functions in authentic settings. Teacher behavior and characteristics have been suggested as important factors that predict the implementation of interventions with treatment integrity (Kratochwill & Shernoff, 2003). Despite the mixed relationship between treatment integrity and outcomes, treatment integrity remains a critical construct to measure in authentic settings. Having a thorough understanding of the implementation is crucial to making appropriate decisions about the validity of the intervention. Understanding the validity of an intervention is especially important within Response to Intervention (RtI) frameworks. Response to Intervention is, “a process of implementing high-quality, scientifically-validated instructional practices based on learner needs, monitoring student progress, and adjusting instruction based on the student’s response.” (Bender & Shores, 2007, p. 7). There are several different models; however, the following provides an 4   overview of RtI generally. RtI employs a three or four tier system. In this framework all students are provided with core instruction, which is considered the universal level or tier 1. It is assumed that about 80-85% of students will be successful at this tier. Students who are not successful are moved to the secondary tier. It is assumed that about 10-15% of students will need strategic intervention to be successful. When students are not successful at tier 2 receive even more intensive intervention at the tertiary level or tier 3. It is assumed about 5% of students will need this level of assistance to be successful. In an RtI framework, decisions about which tier students need services are made through universal screening and progress monitoring with curriculum-based measurements (Bender & Shores, 2007). The core curriculum that all students receive as well as the strategic and intensive interventions that students receive should be evidence-based. Students’ progress with these interventions is then monitored and decisions about the students’ educational process are made based on their response to the intervention. Thus, it is important to not only monitor the student outcomes but also to monitor the implementation of the intervention. Without implementing interventions with treatment integrity, conclusions about whether or not students benefited from an intervention cannot be reliably and validly drawn. As more schools move toward using RtI frameworks, the issue of implementing evidence-based interventions with treatment integrity is becoming more important. One way to address this issue is through examining teacher motivation. Teachers are increasingly required to implement EBIs in their classrooms, but there is little known about teachers’ perspectives on EBIs and on their motivation to use them. This is problematic, as it seems logical that teacher factors such as motivation would affect their implementation of an intervention with treatment 5   integrity. Ultimately, EBIs are becoming a part of school cultures without a great deal of knowledge as to how they are being implemented and how teachers view and conceptualize EBIs. Purpose and Significance of the Study The study examines teacher perspectives on the implementation of evidence-based interventions to understand why teachers may or may not be motivated to implement an EBI with treatment integrity. Currently, research suggests that interventions are not implemented with a great deal of treatment integrity (O’Donnell, 2008) and that treatment integrity is rarely measured (Cochrane & Laux, 2008). As previously mentioned, measuring treatment integrity is important to implementing EBIs in schools and to the RtI movement to ensure that the outcomes can be attributed to the intervention being delivered as intended. However, it is difficult for educators to find the time and resources to measure this crucial construct, even when it is defined simply. It seems unlikely that treatment integrity will be measured multi-dimensionally when it is rarely measured as a single dimensional construct in schools. Gaining teacher perspectives on their experience with implementing evidence-based interventions will also shed light on issues of treatment integrity. Understanding implementation with treatment integrity from the teachers’ perspectives will allow administrators and school psychologists to better understand the alignment between the requirement to use evidence-based interventions and their actual use. Ensuring this consistency is important to the evidence-based practice and the RtI movement, both of which are growing in schools. Theoretical and Conceptual Framework One way to consider the implementation of evidence-based intervention in classrooms is through the lens of teacher motivation. As the primary agent in implementing the intervention, the teacher has the power to determine the effectiveness of the intervention by choosing whether 6   or not to implement it and how to deliver it. A widely used approach to examine teacher motivation is the expectancy-value theory of motivation (Atkinson, 1957; Eccles, 1983). This theory has primarily been used to describe how students’ expectations and values combine in a multiplicative way to create motivation for academic achievement; however, the components of this theory can also be used to view the motivation and effort teachers put forth when implementing EBIs. The origins of expectancy-value theory are in the 1950’s with work by Atkinson; however, work by Eccles and her colleagues have re-conceptualized this theory and applied it to the motivation to achieve in school (Eccles, 1983; Wigfield & Eccles, 2000). This study used an expectancy-value theory to examine teachers’ motivation to implement evidencebased interventions with treatment integrity. Under this framework, the proposed study viewed the attainment of treatment integrity as an achievement behavior that results from teachers’ expectancies and values for the evidence-based intervention. Conceptualizations of Treatment Integrity. Treatment integrity will be considered from a multidimensional perspective. Historically, treatment integrity was considered simply as “treatment as planned” (Yeaton & Sechrest, 1981; Gresham, 1989). This definition of treatment integrity generally applied to behavioral interventions that had easily observable components. As the term, treatment integrity, began to be applied to other interventions including academic interventions, social-emotional curriculums, and prevention programs, this definition of treatment integrity was not comprehensive enough to obtain an accurate assessment of the success of the implementation of the intervention. Dane and Schneider (1998) proposed five aspects of treatment integrity including adherence, quality of delivery, exposure, program differentiation, and participant responsiveness. These five dimensions were identified through a review of the ways in which empirical studies of prevention programs viewed treatment integrity. 7   Other authors have also suggested a multidimensional view of treatment integrity (Mowbray, Holter, Teague & Bybee, 2003; Perepletchikova & Kazdin, 2005; Power et. al, 2005). Of the different works suggesting a multidimensional view of treatment integrity, the five aspects proposed by Dane and Schneider (1998) have been most widely used (Sanetti & Kratochwill, 2009), and will be examined within the proposed study. Research Questions The proposed study will examine the following questions to address the issues of the teacher motivation to implement an evidence-based intervention with treatment integrity. The evidence-based intervention that is the focus of this study is the core reading curriculum, Scott Foresman’s Reading Street (Pearson, 2011). 1. What are teachers’ knowledge of and experiences with implementing evidence-based interventions? 2. What are teachers’ knowledge of and experiences with implementing Reading Street? 3. What is the teachers’ level of motivation to use Reading Street? a. Do teachers expect Reading Street to work? b. Do teachers view Reading Street as important to teaching students to read? 4. What is the teachers’ understanding of treatment integrity? Do they view treatment integrity as unidimensional or multidimensional? 5. How are motivation, treatment integrity, and student learning related? a. How is teacher motivation related to treatment integrity? b. How is teacher motivation related to students’ reading achievement? c. How is treatment integrity related to students’ reading achievement? Hypotheses 8   The nature of many of the research questions is exploratory and the goal is to describe and document teachers’ knowledge, experiences, and beliefs concerning EBI. Therefore, it is difficult to hypothesize how teachers will describe their experiences with implementing Reading Street and how they will define treatment integrity. In regards to the questions about teacher motivation, the hypotheses are based on expectancy-value theory. This theory proposes that when individuals value a task and expect to be successful at it, they will be more motivated to invest more effort into the task (Eccles & Wigfield, 2002). Therefore, it is predicted that teachers who endorse having higher value for the intervention and endorse having higher expectations for the success of the intervention will put more effort into the implementation of the program, which will lead to achieving higher levels of treatment integrity. For the aspects of adherence and exposure, higher expectancy and higher value will result in the teacher using the program more often (exposure) and more closely following the manual (adherence). These teachers with high expectancy and high value will also be able to elicit more participant responsiveness due to their investment in the program. In regards to teacher motivation and student outcomes it is predicted that these variables will be positively correlated. Student motivation for learning occurs in a social context that is influenced by others in that context (Wentzel & Wigfield, 1998). Therefore, students in classrooms of more motivated teachers are more likely to be more motivated and to achieve higher reading scores. In regards to treatment integrity and student outcomes in reading, it is expected that adherence, exposure, and participant responsiveness will all positively correlate with student outcomes in reading. This hypothesis is based on the research that supports the relationship between high treatment integrity and positive student outcomes (Benner, Nelson, Stage & 9   Ralston, 2010; Hirschstein, Van Schoiack Edstrom, Frey, Snell & MacKenzie, 2007; O’Donnell, 2008; Ysseldyke & Bolt, 2007). Overview Chapter 2 reviews the literature in the areas of: 1) the expectancy-value theory of motivation; 2) theories of teacher motivation and related research; 3) teachers’ roles in the EBI movement in education 4) the distinction between implementation fidelity and treatment integrity; 5) the empirical research on the relationship between treatment integrity and student outcomes; 6) the empirical research on the teacher level factors that relate to the implementation of interventions and educational practices with treatment integrity. Chapter 3 details the methods used to answer the research questions and chapter 4 presents the analyses of the quantitative data and the results of the focus group interviews. Chapter 5 discusses the findings, their implications and the limitations of the study. 10   Chapter 2 Literature Review Expectancy-Value Theory Overview Contemporary expectancy-value theory is based on work by Eccles and Wigfield (Eccles & Wigfield, 2002), which expanded upon Atkinson’s earlier motivational ideas. Atkinson (1957) viewed motivation in terms of three variables: expectancy, incentive, and motive. Expectancy was conceptualized as a cognitive anticipation aroused from situational-based cues. Incentive was considered to be the relative attractiveness of a specific goal, and motive was conceptualized as a nondirective but energizing drive. Motive was considered dispositional while the other two variables were situation dependent. Atkinson further conceptualized motive as dichotomous with there being a motive to achieve and a motive to avoid failure. In this framework, the principle of motivation was expressed as the equation, Motivation= f (Motive x Expectancy x Incentive) and incentive is the inverse of probability of success. According to research that used this equation to understand motivation, motivation achieved the highest levels when the probability of success was around 0.5, regardless of whether motive was achievement based or failure avoidance based (Atkinson, 1957). People with an achievement motive were found to prefer tasks with an intermediate level of risk, and people with a failure avoidance motive tended to chose tasks that were very hard or very easy. The Atkinson (1957) model of expectancy motivation was mathematical and used primarily in research settings. The more contemporary version of expectancy-value theory (Eccles and colleagues) stresses the importance of cognitive factors and attributional processes for determining level of motivation rather than focusing on the dispositional motive construct as research did not support the dichotomous motives (Spence & Helmreich, 1983). Wigfield and 11   Eccles (2000) put forth the most recent conceptualization of expectancy-value theory. In this model, individuals’ expectancies and values directly influence their achievement related choices, which include performance, effort, and persistence. Expectancies and values are influenced by task-specific beliefs including ability beliefs, the perceived difficulty of the task, the individual’s goals, self-schemas, and affective memories. In addition to these influences, values are also influenced by the attainment value, intrinsic value, utility value, and cost of the task. Furthermore, expectancies and values are looked at within the context of the cultural milieu and these researchers consider gender roles and cultural stereotypes in their work on motivation. Eccles (1983) felt that it was necessary to consider these other areas because the earlier models were not comprehensive enough to explain all of the factors that influence motivation. Although the present model considers more factors than the earlier models, contemporary expectancy-value theory retains the view of motivation as the product of factors. In this modern equation, expectancy multiplied by value equals motivation for a specific task. An important tenant of this theory is that it is assumed no effort will be expended if either expectancy or value is zero (Brophy, 2010). According to this theory, people will not engage in tasks that they do not value even if they expect to be successful, and people will not engage in tasks that they do not believe they will be successful even if the task is highly valued. The consequences of the various combinations of expectancies and values were demonstrated in Hansen (1989). In this study 193 students from eight different elementary school classrooms were observed over three years. From these observations, four tactics were put forth to describe how students engaged in tasks depending on their levels of expectancy and value. The four tactics were engagement, dissembling, evading, and rejection. When the task was valued and there was a high expectation for success this was considered engagement. When the 12   task was valued but there was a low expectation for success this was considered dissembling. When using this tactic students may want to do the academic task but are unsure how. In order to protect identity and self-esteem, a student dissembles by making excuses or pretending to understand. When the task value was low and the expectation for success high, this was considered evading. Students may “go through the motions” of the task, but their attention is elsewhere. When both value and expectation for success were low this was known as rejection. Students did not engage or defend against the task, rather they were passive and disengaged from the task altogether. Expectancy The expectancy portion of expectancy-value theory is influenced by several cognitive factors including self-efficacy, perception of task-demands, and attributions. Individuals’ current and past perceptions of themselves and specific tasks are used to determine whether or not they expect to be successful again. This assessment of whether or not they expect to be successful influences the level of motivation they will have for a task. Self-Efficacy. Bandura (1997) defined self-efficacy as people’s confidence in their abilities to carry out a task or solve a problem. In Bandura’s self-efficacy theory, high selfefficacy results in greater effort and persistence, which leads to better outcomes and higher efficacy. This cycle is also maintained for lower efficacy, which leads to lower effort and persistence and poorer outcomes resulting in lower efficacy. Bandura’s theory also considers expectation for success. Bandura’s construct of selfefficacy is divided into outcome expectations and efficacy expectation. The former referring to the beliefs that certain behaviors will lead to specific outcomes, and the latter referring to beliefs about whether an individual can effectively produce the behaviors needed to result in a certain 13   outcome. Efficacy expectations have been implicated as important factors in goal setting, activity choice, persistence, and effort (Bandura, 1997). Although Eccles and Wigfield’s expectancyvalue model is often seen as emphasizing outcomes, these authors argue that their conceptualization of self-concept of abilities is similar to the Bandura’s efficacy expectations, as their research has focused on measuring individuals’ own expectations for success rather than their outcome expectations (Wigfield & Eccles, 2000). Another important aspect of self-efficacy is the specificity of the construct. Self-efficacy is typically measured specific to particular areas and situations (Schunk, Pintrich & Meece, 2008). Bandura (1997) suggests that self-efficacy is situationally specific and fluctuates based on different factors. Self-efficacy has been studied in regards to many areas including, academic subjects, athletic performance, career pursuits, managing physical and mental health conditions, and organizational functioning. Self-efficacy has also been studied in terms of efficacy teachers have for teaching. Teacher self-efficacy is a teacher’s belief about his/her own abilities to effectively teach. Teacher self-efficacy acts as a cognitive filter that shapes how teachers view their experiences and in turn shapes their thoughts and actions (Woolfolk, Hoy & Davis, 2009). Woolfolk, Hoy and Davis (2009) proposed a cycle of teachers’ efficacy judgments. In this model, the sources of teachers’ efficacy beliefs include vicarious experience, verbal persuasion and feedback, affective state, and mastery experience. These efficacy beliefs undergo analysis and interpretation, which result in an assessment of teaching competence. This assessment informs teachers’ perceived sense of efficacy, which leads to goal setting, effort and persistence, and resilience. These consequences result in outcome performances at the individual teacher level and the collective school level. These outcome performances then inform teachers’ self-efficacy beliefs and the cycle starts 14   again. In this model, teacher self-efficacy has consequences for teacher behavior and student behavior. Both teacher and student behavior have implications for student outcomes, and it is therefore, important to consider the influence of teacher self-efficacy in classrooms. Examining teacher efficacy is an example of how the construct of self-efficacy is meant to be used for specific tasks rather than more generally. To measure teacher self-efficacy Gibson and Deno (1984) developed the Teacher Efficacy Scale (TES; Woolfolk, Hoy & Davis, 2009). This measure has undergone several revisions and has been used in many studies; however, this scale is not considered to be a precise and valid measure of teacher self-efficacy (Woolfolk, Hoy & Davis, 2009). Due to the limitations of the global measures of teacher self-efficacy, teacher efficacy has also been studied in regards to specific areas such as content areas including science, math, and language and literacy and for other areas of teaching including working with diverse students, inclusion, and using technology in the classroom. Teacher self-efficacy for using evidence-based interventions is another specific area of teaching that can be studied. Perception of task-demands. This aspect of expectancy-value theory involves how an individual views a specific task, such as the level of difficulty (Schunk, Pintrich, Meece, 2008). Perception of task-demands was a part of the earlier views of expectancy-value theory. Kukla (1972) examined perception of task-demands as a variable that influences motivation and choice behaviors. According to Kukla’s theory, easier tasks should involve less effort and the relationship between perception of difficulty and intended effort should be negative and linear. However, different people will view the same task as being more or less difficult. Therefore, the relationship between perceived difficulty and effort takes the form of an inverted U. Tasks that are perceived as very easy and very hard are more likely to result in minimal effort being expended. When a task is very easy, individuals assume more effort is not needed, and when a 15   task is very difficult individuals assume expending effort will not change the outcome. Tasks that involve an intermediate level of difficulty are associated with increased effort and performance. Attributions for previous successes and failures. Understanding success and failure on previous tasks has been explained with attribution theory (Weiner, 1985). Attribution theory is based on the assumption that people have the goal of attaining knowledge in order to make their environments more predictable, and therefore, seek to understand their successes and failures in terms of causes (Weiner, 1985). Attributions are the perceived causes of outcomes, and these attributions influence individuals’ expectancies for future success. Weiner (1985) put forth a framework with which to view attributions. Motivation is based on the evaluation of three causal dimensions: locus, stability, and controllability. The dimension of stability is the most relevant to expectancy-value theory. Weiner put forth the expectancy principle, which stated that changes in the expectation of success for a task following an outcome is influenced by the perceived stability of the events surrounding that outcome. If success was attributed to a stable cause then the expectation for that outcome in the future increased. If success was attributed to an unstable cause then the expectation for the outcome in the future was unchanged or thought to be different. Self-fulfilling prophecy. Wigfield and Eccles’ expectancy-value model is concerned with the context in which individuals are embedded. Within these contexts individuals may be subjected to stereotypes, gender norms, and the influence of their socializers. When considering the profession of teaching one area of research that captures these areas is the self-fulfilling prophecy. The research on self-fulfilling prophecies has primarily focused on how teacher perceptions of students create expectations for the students. Teachers then begin to differentially 16   treat students based on the expectations that they created for them, and students in turn act in ways that confirm the teacher’s expectation (Jussim, Robustelli, & Cain, 2009). Although this research has been popular in the fields of education and psychology, the results from studies on self-fulfilling prophecies are mixed. Rosenthal and Jacobson (1968) initiated the original line of this research. The impact of this work was monumental in terms of the interest in self-fulfilling prophecies that it created; however, more recent consideration of the data suggests the effect of teacher’s expectations on student performances was not as significant as was originally supposed (Jussim, Robustelli, Cain, 2009). Research in this area indicates that self-fulfilling prophecies do occur in both experimental and naturalistic studies with effect sizes between 0 and 0.4 and with an average an overall mean effect size between 0.07-0.17 (Jussim, Robustelli, Cain, 2009). Self-fulfilling prophecies are an example of how teacher expectations can influence their behavior in classrooms, although the effect sizes may be smaller than originally thought. Value According to expectancy-value theory, how much value an individual places on a task will influence how motivated the individual will be on that task. There are four components of the value portion of Expectancy-Value theory. These components include attainment value, intrinsic value, utility value, and cost (Brophy, 2010; Eccles, 1983). Attainment value. Attainment value refers to the value the task has for affirming selfconcepts, and core needs, values or identities (Brophy, 2010; Eccles, 1983). Battle (1966) conducted a study that used attainment value as an independent variable in relation to academic performance. This study conceptualized attainment value as how important doing well on a task was to a student. The study further broke attainment value into relative attainment value and 17   absolute attainment value. Relative attainment value is the importance a student places on doing well in a particular subject compared to the importance of doing well in other subjects, and absolute attainment value is more general construct for placing importance on being competent in academics regardless of subject. Battle (1966) found absolute attainment value more predictive of academic performance in both English and math than relative attainment value. Overall, this study supported the idea that attainment value is a motivational factor that is able to predict academic performance. The current perspective on attainment value is similar to Battle’s (1966) definition of the importance of doing well on a task. This construct relates closely to aspects of the self-schema. Attainment value allows people to confirm or disconfirm areas of importance to their selfschemas. If an area is important to an individual’s self-schema then there will be a high attainment value for that area (Schunk, Pintrich & Meece, 2008). Intrinsic value. Intrinsic value is the enjoyment an individual experiences when doing a task, or the subject interest in the task. Intrinsic value is very closely related to intrinsic motivation, which has been discussed at length by Deci (1975). Intrinsic motivation is when individuals engage in a task for their own sake and not for a reward or other means to an end, or when there is no apparent reward for engaging in the task (Deci, 1975). Earlier definitions of intrinsic motivation focused on humans’ desire to maintain a certain level of stimulation. This motivation is present in Deci’s theory. However, Deci expanded the theory to also consider how people are intrinsically motivated not only to control level of stimulation but to feel competent as well. When people are under-stimulated they seek out opportunities to behave in a competent way. When a person is over-stimulated they are intrinsically motivated to overcome and conquer that challenge which also helps them to feel competent (Deci, 1975). According to this theory 18   activities or task that are the right amount of stimulation and challenge are the most intrinsically motivating. Utility value. Utility value, or instrumental value, refers to the usefulness of a task for helping an individual achieve future goals (Schunk, Pintrich & Meece, 2008). This type of value relates to the ends of the task rather than the means, and it relates to individuals’ short term and long term goals. Utility value relates closely to extrinsic motivation; however, Eccles and Wigfield (2000) note that extrinsic theories come from distinct, different theoretical backgrounds. Cost. Cost relates to utility value in that it considers the external forces that are involved with individuals’ goals. Cost specifically relates to the negative aspects of engaging in a task (Schunk, Pintrich & Meece, 2008). These negative aspects include amount of effort required of the task, anticipated emotional states, lost opportunity due to engaging in one task over another (Wigfield & Eccles, 2000). Considering these aspects influences the value of the task to the individual because typically individuals want to avoid costs if at all possible. Teacher Motivation A vast majority of the research on motivation in schools focuses on students and their motivation for achievement; however, teacher motivation is also an important area to examine as teachers’ actions influence student outcomes (Retelsdorf, Butler, Streblow, & Schiefele, 2010). Neves de Jesus and Lens (2005) put forth a model of teacher motivation using a different framework. These authors surveyed Portuguese teachers to validate a model of teacher motivation that integrates the expectancy-value theory and learned helplessness theory. The proposed model suggested that teachers start out motivated but once they experienced failure, a sequence of events began that compromised their motivation. After teachers experience failure they might attribute the failures to internal and stable causes, which leads to external control 19   expectations. Alternatively, teachers may experience success but attribute these to external and unstable causes, which leads to low efficacy expectancy. Either of these events leads to low expectancy for success and to lowered intrinsic motivation, which leads to lowered goal value. In this study, teacher motivation was conceptualized as professional engagement, which included activities such as, participation in extracurricular activities, student praise, efforts with course preparation, availability to students outside of class, and attempts to diversify teaching strategies. The results of the study suggested that the only variable, which exerted direct influence on professional engagement (teacher motivation), was goal value. The majority of the other relationships proposed in the integrated model were also significant. The exception was attributions (both for successes and failures); these were not significantly related to efficacy expectancy. Overall, this study offers a model for understanding teacher motivation, which can be used as a framework for other studies on teacher motivation and integrate this area of research. The framework presented in Neves de Jesus and Lens (2005) has been adopted to explore teacher motivation in other studies. Cave and Mulloy (2010) used the framework to understand teacher motivation to implement a Technology-Rich Authentic Learning Environment in early childhood classrooms. This study was exploratory and qualitative, and interviewed four teachers who used the program. Two of the teachers were identified as high implementers and two were identified as low implementers. One of the interesting findings from this study was that high implementers made internal attributions for success, whereas, low implementers made external attributions for success, which, may have led to lower expectancies for success. These findings are relevant for understanding teacher motivation because it suggests that teachers’ perceived locus of control relates to whether or not teachers will implement a program. The findings of this study, however, have low generalizability due to the very limited sample size. The strength of 20   this study is that it took a different approach to studying motivation. Instead of relying on surveys to assess motivation, the study sought a deeper understanding of teacher motivation for implementation through interviewing and qualitative methods. Evidence-Based Intervention Movement in Education A major tenet of the evidence-based movement in all professions is that professionals should use practices, procedures, techniques, and therapies.that have demonstrated efficacy to promote desired outcomes through empirical research. As a part of promoting this tenet of NCLB, the United States Department of Education (USDE) developed the What Works Clearinghouse to evaluate and publish online reviews of educational interventions. The goal of the clearinghouse is to disseminate knowledge about the “scientifically based practices” that are called for in NCLB. Teacher motivation is a crucial issue to consider when it comes to the evidence-based intervention movement in schools because, although a method of dissemination of knowledge was developed, ultimately it is the teacher’s responsibility to implement the intervention in the classroom. Research has suggested that teachers’ use of and knowledge of evidence-based interventions in classrooms has remained limited. Most of the research conducted on the prevalence and use of evidence-based interventions in schools comes from the special education literature. Burns and Ysseldyke (2009) reported that two of the most frequently used techniques by 174 surveyed special education teachers were the least effective (social skills training and modality training). A more recent study has suggested social skills training is more effective than Burns and Ysseldyke (2009) suggest (Durlak et al., 2011). However, Burns and Ysseldyke still suggested that the prevalence of EBIs in special education classrooms is low. 21   Boardman, Arguelles, Vaughn, Hughes, and Klingner (2005) conducted focus groups with special education teachers to better understand their views of research-based practices. The results of these focus groups indicated that special education teachers did not feel pressured to use research-based practices. Rather than selecting practices based on research, this study suggested special educations teachers select practices based on feasibility, appropriateness for students, ability to be individualized for multilevel classrooms, and access to necessary materials and professional support. Furthermore, Stormont, Reinke and Herman (2011) reported that teachers lack even the knowledge of evidence-based interventions for behavior and emotional concerns. In this study it was found that most teachers had not heard of nine out of ten of the evidence-based interventions, and 57% of teachers were unsure if their schools engaged in functional behavior assessments and intervention planning. Overall, there is widespread evidence that there is not only a lack of use of evidence-based interventions among teachers, but that there is a lack of knowledge as well. In addition to teachers’ lack of use and knowledge of EBIs, there is a lack of information on the issues of implementation. Implementation of evidence-based interventions has been studied in the fields of public health, mental health, and K-12 education with the majority of this research focusing on public and mental health (O’Donnell, 2008). The diversity of fields studying implementation has yielded a great deal of knowledge about this issue; however, this diversity has also led to a lack of consensus on how to define the construct of implementation and the constructs related to implementation. Defining Implementation, Fidelity, and Treatment Integrity One attempt to centralize the work on implementation was through Fixsen and colleagues’ (2009) monograph. In this piece, Fixsen and colleagues conceptualized implementation as 22   having a source, a destination, a communication link, a feedback mechanism, and as occurring within a sphere of influence. Fixsen and colleagues also detailed the stages of the implementation process, which include exploration and adoption, program installation, initial implementation, full operation, innovation, and sustainability. Although the authors delineated six steps of implementation, they noted that research has primarily occurred at the first two steps. In addition to the conceptual framework and stages of implementation, Fixsen and colleagues discuss core components, which are the aspects of interventions that are essential and indispensible parts of the program. According to the monograph, it is important for practitioners to implement core components with high fidelity. High fidelity is considered to have been achieved when a practitioner interacts with a client in a way that is based on research findings. The process of implementing core components includes selection of a program, preservice training, consultation and coaching, staff evaluation, program evaluation, and facilitative administrative supports. It is also integrated and compensatory as some programs that have weaknesses in one area of the process make up for it by having strengths elsewhere in the process. Implementation Fidelity. In Fixsen and colleagues’ monograph it is difficult to separate implementation from implementation fidelity. According to this framework, implementation implies high fidelity; however, other researchers have conceptualized implementation fidelity differently than Fixsen and colleagues (2005). Durlak and Dupre (2008) considered implementation as having eight aspects including adherence, exposure, quality, participant responsiveness, and program differentiation, monitoring control/comparison conditions, program reach, and adaptation. All of these aspects are specific parts of delivering an intervention, which 23   can be measured. Despite the potential for these aspects to be measured, differential attention has been paid to these aspects in research (Durlak & Dupre, 2008). This difference in conceptualization may be partly due to wanting a way to measure high versus low implementation, as some researchers were interested in how level of implementation influences outcomes. Under Fixsen and colleagues’ (2005) model it is difficult to measure implementation that was low, or different than what was intended by the developers. As adaptation occurs frequently when implementing EBIs in authentic settings (Higa & Chorpita, 2008) it is important to be able to operationally define implementation as an independent variable to assess what influence it has on dependent variables of interest such as social emotional, mental health, and academic outcomes. Defining Treatment Integrity. Treatment integrity is often a term that is used interchangeably with implementation fidelity; however, it is important to distinguish between these two constructs, as the literature’s habit of using these terms interchangeably has contributed to the lack of consensus on definitions for the terms within implementation science. Originally, treatment integrity has been defined as treatment as intended (Yeaton & Sechrest, 1981; Gresham, 1989). Even when treatment integrity was conceptualized simplistically, it was rarely measured in research (Gresham, Gansle & Noell, 1993; McIntyre, Gresham, DiGennaro & Reed, 2007). More recent research has considered treatment integrity to have multiple dimensions (Sanetti & Kratochwill, 2009). Implementation fidelity has been also defined in terms of level of adherence to a protocol or program manual (Mowbray, Holter, Teague & Bybee, 2003). In contrast to treatment integrity, this construct has not been broadened to include multiple dimensions. The two terms are close in meaning, however, the term treatment integrity 24   includes several dimensions while implementation fidelity continues to refer to a dichotomous construct. This re-conceptualization of the term treatment integrity has come out of the evidencebased practice movement. Evidence-based interventions range in complexity from school wide Positive Behavior Support (Sugai & Horner, 2006) to simple individual student interventions such as Cover, Copy, and Compare (Skinner, McLaughlin, & Logan, 1997). While the original definition of treatment integrity might have been suitable for a straightforward, academic intervention such as Cover, Copy, Compare, valuable information about implementation is lost when the implementation of more complex interventions are considered in a dichotomous framework (e.g., was this implemented as intended or not?). Dane and Schneider (1998) reviewed studies of prevention and early interventions for public and mental health. In this review, the authors identified five dimensions of treatment integrity that were present in the literature including adherence, quality of delivery, exposure, participant responsiveness, and program differentiation. The dimensions outlined in this model are relevant to the field of school psychology as well. Adherence aligns closely with previous, simplistic definitions of treatment integrity. This construct considers whether or not the intervention components were implemented as intended. Exposure refers to how much of the intervention was implemented. Quality of delivery relates to how well the intervention was implemented. Participant responsiveness refers to how engaged and alert the participants were to the intervention during its implementation. Finally, program differentiation refers to how different the program or intervention is from existing programs in the setting. This multidimensional conceptualization of treatment integrity allows for a more in depth understanding of the phrase, “as intended.” For example, the components may be able to 25   implemented with high levels of adherence but with a lower level of quality than was intended by the developers. Mowbray, Holter, Teague and Bybee (2003) considered there to be treatment integrity to structure as well as treatment integrity to process. Structure includes the dimensions of adherence, exposure, and program differentiation, and process includes the dimensions of quality of delivery and participant responsiveness. Sanetti and Kratochwill (2009) offer a framework that draws together several models of treatment integrity, which apply to the field of school psychology. In this framework, other models of treatment integrity and implementation are synthesized. The aspects of treatment integrity that Dane and Schneider (1998) put forth are the center of this framework as these constructs have been adopted in at least two conceptual models of treatment integrity (Sanetti & Kratochwill, 2009). Although some researchers have called for the assessment of all of these dimensions in comprehensive assessments of treatment integrity, Sanetti and Kratochwill (2009) caution that the data supporting these dimensions as critical to student outcomes is not yet available. However, these authors do support the notion that early, simplistic definitions of treatment integrity are inadequate and propose considering treatment integrity as a broader construct. According to Sanetti and Kratochwill (2009), “Treatment integrity is the extent to which essential intervention components are delivered in a comprehensive and consistent manner by an interventionist trained to deliver the intervention” (p. 448). The Relationship between Implementation, Treatment Integrity, and Outcomes Previous research on the transportability of evidence-based interventions has examined the relationship between program implementation and outcomes. However, the literature is mixed on whether degree of implementation results in better outcomes. One reason for the mixed results is the overlapping and inconsistent use of terms for constructs. When implementation and 26   treatment integrity are viewed as distinct constructs a clearer picture of the relationship emerges. In the present study, implementation is considered broadly as the delivery of a specific intervention or program. This study will use Sanetti and Kratochwill’s (2009) definition of treatment integrity, which includes the five dimensions of treatment integrity laid out by Dane and Schneider (1998). Implementation and outcomes. Exposure is one of the dimensions of treatment integrity that is closely related to implementation. It answers the question of how much of the intervention was implemented. It is difficult to separate these two constructs; however, when exposure is measured in isolation it should be considered implementation because the focus is simply on whether a certain program, practice, intervention, or activities were delivered. When exposure is measured with concern for how other aspects of the intervention or program were implemented it becomes part of the larger construct of treatment integrity. The following studies focused on linking how much of an intervention was implemented without measuring other aspects. Therefore, these studies investigated implementation under the present framework. Durlak and Dupre (2008) conducted an extensive meta-analysis of over 500 studies on prevention programs to investigate the relationship between level of implementation and program outcomes. All of the studies used in this meta-analysis were conducted in real world settings and focused on prevention programs for physical health, academic performance, drug use, violence, bullying, and positive youth development. Results suggested that higher levels of implementation were associated with higher program outcomes. Although there was strong evidence that high levels of implementation were important for achieving successful outcomes, the authors still argued that adaptation of programs is acceptable in authentic settings. In line with Fixsen and colleagues’ (2009) concept of core components, Durlak and Dupre (2008) 27   suggested that practitioners should implement theoretically important aspects of the program with high fidelity and modify less important aspects to create a good ecological fit with the setting. O’Donnell (2008) conducted a review of empirical studies on academic interventions to determine the relationship between implementation and outcomes. This review only considered studies on an intervention that could be implemented by a single teacher in a classroom and only included academic interventions. With such stringent inclusion criteria, only five studies were included in the analysis. However, all five studies suggested that higher levels of implementation resulted in higher outcomes. Noell, Gresham, and Gansle (2002) conducted a study on the implementation of a computerized math intervention. The aim of the study was to examine the influence of treatment integrity on outcomes; however, the dimensions of quality of delivery and adherence were controlled for by using the computer to implement the intervention. The participants in the study received different amounts of exposure to the crucial components of the intervention; therefore, the study looked solely at exposure. The findings indicated that the students who received the highest level of exposure to the intervention component of instructional prompts made the most achievement gains, and students who received fewer prompts had poorer outcomes. These findings suggested that the higher the exposure, the better the outcomes. Ysseldyke and Bolt (2007) primarily intended to study the effects of a class-wide progress monitoring math program on student achievement in math; however, this study provided an example of the importance of implementation. The authors did not set out to measure treatment integrity or implementation fidelity; however, the level of implementation proved to be important to this study’s findings. Initial results suggested that the program did not 28   promote achievement on standardized math assessments. However, due to differential use of the program in the various classrooms the researchers analyzed data based on the extent to which teachers participated in the program. Participation was conceptualized as the number of objectives students mastered across the year. The three groups were non-implementers, low implementers, and high implementers. When looking at the data in this way, consistent large gains were observed for the high implementer group as opposed to the other two groups. This finding led the authors to conclude that this program needed to be used with a high level of implementation to promote the desired intervention outcomes. This study underscores the importance of measuring the level of implementation that was achieved, as some teachers did not implement any of the program elements. Although the authors did not originally measure either implementation or treatment integrity, it was necessary to consider level of implementation to uncover the intervention effects. It is not surprising that a program would fail to improve student outcomes when it is not implemented, and therefore, it was necessary for the authors to analyze the data in groups based on level of implementation. Without considering level of implementation, inaccurate conclusions may have been drawn that masked the intervention’s actual effectiveness. Although the authors only retroactively measured one aspect of treatment integrity, exposure, they concluded that treatment integrity is critical for interventions in schools. Treatment integrity and outcomes. When studies that consider only the exposure dimension of treatment integrity are considered, a clearer picture of implementation and treatment integrity emerges. Higher levels of implementation, often only measured in terms of exposure, results in higher student outcomes. This pattern fits logically as one cannot expect an intervention that was not implemented to influence student outcomes, as was seen in Ysseldyke 29   and Bolt (2007). The complexity of treatment integrity and outcomes appears to result primarily when other dimensions such as adherence, quality of delivery, program differentiation, and participant responsiveness are taken into account. When considering these dimensions, findings about the relationship between treatment integrity and outcomes become more complex and mixed. Studies demonstrating mixed or lack of relationship between treatment integrity and outcomes. Dane and Schneider (1998) conducted a meta-analysis of studies in the field of prevention science. They reviewed 162 studies of primary and early secondary prevention programs from 1980-1994. Of these studies, only 39 measured treatment integrity, and of these only 13 studies examined how differing levels of treatment integrity affected the program. With a low sample size, this study did not offer firm conclusions on the relationship of dimensions of treatment integrity and outcomes. With regards to adherence there were five studies measuring this dimension. Two of the studies showed a positive relationship between adherence and outcomes, two showed mixed results, and one study demonstrated a nonsignificant relationship. Only one study measured quality of delivery, and no significant effects were found. However, the finding that only 39 of 162 studies measured treatment integrity demonstrated that treatment integrity was understudied in this field as it has been in other fields (Gresham, Gansle, & Noell, 1993). This study suggested that perhaps, some of the mixed findings of the effects of treatment integrity on outcomes are due to the limited number of studies that have examined this relationship. Perepletchikova and Kazdin (2005) provided a thorough review of issues of treatment integrity in regards to the therapeutic change literature. In this literature, as in the prevention science literature, the relationship between treatment integrity and outcomes is conflicting and 30   complex. Through a review of the literature the authors found that characteristics of the intervention, characteristics of the client, and characteristics of the therapist influence treatment integrity. These characteristics also influence treatment outcomes, which adds to the complexity of determining how treatment integrity relates to outcomes. These characteristics may moderate, mediate, or confound the relationship between treatment integrity and outcomes. In addition to the overlapping characteristics, the authors implicated inadequate and differing methods as a potential reason for the complexity in how treatment integrity and outcomes are related. Overall, it is difficult to separate the change agents in studies of treatment integrity. In the area of school-based behavioral consultation, some studies have examined how the level of treatment integrity with which the consultee implements a plan affects student outcomes. Wickstrom, Jones, LaFleur and Witt (1998) studied treatment integrity as a dependent variable in order to determine whether certain types of consultation would promote it. The findings suggested that teachers only implemented the planned intervention 4% of the time. However, there were reductions in student behaviors despite the low levels of treatment integrity. SterlingTurner, Watson and Moore (2003) also examined treatment integrity as a dependent variable to study ways to promote it. This study suggested there is a mixed relationship between treatment integrity and outcomes because of the four consultation cases that were studied only three of them demonstrated that higher treatment integrity resulted in better outcomes. The student that was a part of the fourth consultation case made improvements despite low levels of treatment integrity. Lee, Penfield, and Maerten-Rivera (2009) studied the effects of the dimensions of quality of delivery through looking at the quality of science instruction for English language learners. In this study, both teacher self-report and classroom observations were used to assess quality of 31   delivery. Contrary to the authors’ hypotheses, student achievement was not related to either measure of quality of delivery. The authors cited the limited number of classroom observations, two per teacher, as a reason for the insignificant findings. This study, like others, highlights the difficulty of measuring treatment integrity as a major reason for the mixed findings in the relationship between integrity and outcomes. Studies demonstrating a positive relationship between treatment integrity and outcomes. Although there is research that demonstrates a negligible or mixed relationship between treatment integrity and learning outcomes there are many studies that provide a more positive picture of this relationship. Haggler et al. (1997) conducted an effectiveness study of Multisystemic Therapy (MST). In this study, the researchers implemented MST in a manner that was similar to everyday clinical practice. There was less influence from university collaborators and less monitoring of treatment integrity. The results of this study showed that MST had more modest effects than it had when implemented more stringently in efficacy settings. Additionally, therapist adherence to MST protocol was a significant predictor at 1.7-year follow-up for important MST outcomes. Even though it is disheartening that MST is less effective in authentic settings, these findings suggest that implementing MST with high levels of adherence was important to achieve the desired outcomes. Benner and colleagues (2010) studied the influence of treatment integrity on student outcomes for a reading intervention. In this study two dimensions of treatment integrity were examined, adherence and quality of delivery. Adherence to the reading intervention was conceptualized as implementing the intervention as intended by developers. Quality of delivery was conceptualized as the process of how well the implementers implemented the intervention. This construct was measured through direct observation of teachers implementing the 32   intervention. A five-point Likert scale was used to rate teachers on five specific behaviors related to the interventions. The results of the study indicated that 22% of the variance in student gains in basic reading and 18% in comprehension were due to overall treatment integrity. Teacher adherence to lessons had a statistically significant effect on reading outcomes, and quality of delivery factors that predicted reading outcomes were re-teaching the lessons when needed and use of established error procedures. Overall, this study adds to the literature suggesting higher levels of treatment integrity support higher student outcomes. Hirschstein, Van Schoiack Edstrom, Frey, Snell and MacKenzie (2007) investigated the relationship of adherence and quality of delivery with outcomes as a part of an evaluation of the Steps to Respect program, which aims to prevent bullying. Teacher adherence to lessons was shown to promote higher ratings of peer social skills, which supports the notion of higher adherence results in better outcomes. Paradoxically, the quality of delivery of the lessons was associated with greater self-report victimization and more perceived difficulty responding to bullying. These findings suggest that higher quality of delivery failed to produce the positive expected outcomes of an anti-bullying program suggesting that this intervention has low effectiveness despite demonstrated efficacy. Teacher Factors in the Implementation of EBIs with Treatment Integrity As EBIs have the potential to promote positive student outcomes and teachers are required by No Child Left Behind to use educational practices that are researched based, it is important to examine what factors promote and inhibit their implementation in schools with treatment integrity. Despite the lack of a solid relationship between outcomes and treatment integrity, it is necessary to ensure treatment integrity in practice settings to control for implementation issues. Without implementing an EBI with treatment integrity, educators are 33   unable to attribute change or lack of change to the intervention and left uncertain about the reasons for the outcome. This uncertainty about the change or lack of change an intervention produced does not allow for valid conclusions to be drawn about the effect of the intervention. Thus, measuring treatment integrity is still important regardless of its relation to outcomes. Measuring treatment integrity is a matter of validity. This issue of validity is especially important for schools that use an RtI framework for making educational decisions, such as special education eligibility. Factors that promote and factors that hinder the implementation of evidence-based intervention with treatment integrity in schools can occur on any level (Kratochwill & Shernoff, 2003), however, as teachers are often expected to be the primary implementers, examining the teacher related factors seem especially relevant to effective implementation of EBIs in school settings. Teacher characteristics and perceptions. Han and Weiss (2005) conducted a review on the sustainability of teacher implemented mental health interventions. The authors noted that a majority of the research on evidence-based intervention in schools is at the system level, and teacher factors are often overlooked. In this review, several teacher factors important to the successful implementation of school-based mental health interventions were discussed. First, teacher perception of principal support was identified as an important factor of teacher implementation of interventions. Second, teachers with higher self-efficacy for teaching were more likely to implement new practices and to invest greater effort into implementation. Finally, teacher perceptions of the intervention prior to implementation influenced their motivation to implement it. The factors identified in this review have received empirical support from several studies. 34   Biggs, Vernberg, Twemlow, Fonagy, and Dill (2008) examined treatment integrity in relation to teacher attitudes when implementing the CAPSLE anti-bullying program. Although the authors used the term, adherence, to describe treatment integrity in this study, the items on the self-report questionnaire of adherence dealt only with issues of exposure or how often a teacher implemented major components of the intervention. The findings of the study suggested that perceived helpfulness of CAPSLE and the consistency between teacher beliefs about bullying and the intervention’s guiding principals were related to whether or not teachers integrated the program into their classrooms. Contrary to the hypothesis, the study also suggested teachers’ use of the intervention was not related to their attitudes about classroom management techniques. Ransford, Greenberg, Domitrovich, Small and Jacobson (2009) studied teacher psychological experiences including burn out, self-efficacy, and perceptions of curriculum support and how these were associated with implementation of a social-emotional learning program. Higher perceived administrative support was associated with higher implementation quality, and positive perceptions of training and support were associated with higher implementation dosage and quality. Baker, Kupersmidt, Voegler-Lee, Arnold, and Willoughby (2010) examined factors that predicted teacher implementation of evidence-based intervention designed to promote school readiness in preschool children. Implementation was quantified as the percentage of possible activities implemented across the course of the intervention. The results of the study indicated that teacher concerns about implementing the intervention predicted lower participation. Positive participation was predicted by teachers’ perceptions that their center directors’ were supportive, and teacher job satisfaction and commitment. Unlike other research in this line, this study did not 35   find teacher experience and self-efficacy to significantly predict implementation of the intervention. Stein and colleagues (2008) conducted a study that examined how the level of support given to teachers when implementing Kindergarten Peer Assisted Learning Strategies (K-PALS) affected student outcomes. Contrary to the hypothesis, all three of the treatment levels in the study (training manual only, workshop, workshop with booster sessions) were associated with similar gains in student achievement. However, the level of support was associated with differing levels of treatment integrity with the most intensive level of support being associated with the highest levels of treatment integrity. This study suggested that the level of support provided to teachers is an important factor in the level of treatment integrity that is achieved during implementation. Motivational factors to implement educational innovations. The previous studies demonstrated that organizational factors and intervention characteristics influence teacher implementation of interventions with treatment integrity. The following line of research examined how motivational factors affect teachers’ educational practices. Abrami, Poulsen, and Chambers (2004) investigated the use of cooperative learning strategies in classrooms. Cooperative learning is a research-supported set of principles that teachers may choose to implement in their classrooms. The main focus of this study was the differences between users and non-users of cooperative learning strategies. The authors developed a questionnaire that measured teachers’ expectancies, values, and costs specifically in regards to cooperative learning. The questions were designed to measure if the teachers expected cooperative learning to work, if they valued the constructs underlining cooperative learning, and if the costs were prohibitive to using cooperative learning. Using this measure teachers were 36   surveyed, and it was found that the most important factor that differentiated the two groups was expectancy for success. Value was also a significant predictor but to a far lesser degree. Cost was the least predictive factor involved, which is surprising as teachers often cite prohibitive costs as reasons not to implement programs. The authors concluded that perhaps targeting issues of expectancy for success during professional developments on cooperative learning would lead to increases in the implementation of these strategies. Wozney (2006) also focused on teachers’ motivation to implement specific practices in classrooms. Instead of examining cooperative learning, the focus of this study was on the integration of technology in classrooms. This study used a modified version of the measure in Abrami et al. (2004). The term cooperative learning was replaced with technology use, and then the measure was used to determine the factors that related to teachers incorporating technology. Therefore, the measure in this study focused on the same constructs as Abrami et al. (2004). The only difference was the focus on a different intervention. The questions focused on if the teachers expected technology use to improve student outcomes, if they valued technology use, and if the costs of technology were prohibitive to adoption. The findings of this study were similar to Abrami et al. (2004), as expectancy was the most predictive factor. Moreover, the regression coefficients also followed a similar pattern across the two studies with expectancy being the largest predictor, value taking a far second, and cost coming in third. Similar to the conclusions drawn in Abrami et al. (2004), this author called for attention to be paid to teachers’ expectancies for success during professional developments. Wozney (2006) also echoed the surprise in cost being a small predictor as access resources are often thought to be a barrier in school settings. The regression coefficients found for each variable are displayed in Table 1. 37   Table 1 Regression Coefficients for Abrami et al. (2004) and Wozney (2006) Expectancy Value Cost Abrami et al. (2004) Wozney (2006) 0.44 0.04 0.01 Total Variance In Implementation Explained by EVT 0.40 0.39 0.15 0.14 0.33 In a recent study, Foley (2011) used an expectancy-value framework to examine teacher motivation to implement evidence-based reading comprehension instructional strategies in the primary grades. The Abrami and colleagues (2004) and Wozney (2006) studies took place in Quebec, Canada and focused on the implementation of broad instructional practices whereas this study took place in the United States and was focused on evidence-based practice. Similar to the other studies, Foley developed a measure to assess expectancies, values, and costs that were specific to comprehension strategy instruction (CSI). Consistent with previous research, expectancy and value significantly predicted levels of implementation, supporting recommendations to target these areas in professional development to increase implementation. Overall, these three studies examined the implementation of three different types of educational practices, but used the same framework for understanding why teachers do or do not choose to implement them. Similar questionnaires were used to measure teacher expectancies, values, and costs; however, the questionnaires were tailored specifically toward the educational practice of interest. Despite studying three different instructional practices (cooperative learning, technology use, and reading comprehension instruction), expectancy was consistently found to be the most predictive motivational factor. The convergence of these three separate studies demonstrating that expectancy best predicts the implementation of three very different programs 38   is strong evidence that expectancy is the most relevant aspect of expectancy-value theory. These findings provide strong evidence that professional development should target teachers’ expectancies. These previous studies indicate that teacher motivation is an important aspect of implementing educational innovations in classrooms, however, these studies do not address the issue of treatment integrity. Ringwalt and colleagues (2003) studied the implementation of substance-abuse programs in schools through an expectancy-value framework and considered treatment integrity. In this study, teachers that implemented substance abuse prevention programs were surveyed about their implementation of these programs. A representative sample of 1,905 participants responded to questions about motivational factors, treatment integrity, and their school context. Results indicated that teachers with higher perceived effectiveness or expectation for the program resulted in higher adherence to the curriculum. In addition, teachers with greater self-efficacy for teaching substance abuse implemented the curriculum with higher adherence. On the value side of this theory, teachers who liked teaching substance abuse also implemented the program with higher adherence. Although this study advanced the literature on teacher motivation to implement educational innovations by considering adherence, the measure for adherence was a reporting on a 1-5 scale of how much they adhered to their specific curriculum’s manual rather than a direct measure of behavior. Overall, the literature has provided evidence that teachers’ expectancies and values are important aspects of the implementation of educational innovations in classrooms. Other characteristics such as school/administrative support have also been implicated. This literature needs to be extended to better understand teachers’ views on implementing evidence-based practices. The previous research forms a foundation, which clearly demonstrates that teachers’ 39   expectancies, and values are important motivational components for implementing programs in schools, but the majority of the studies did not focus on mandated evidence-based interventions. The top down nature of being mandated to use a particular educational practice or program could have important implications for teachers’ expectancies and values for the intervention. This study will contribute to the literature through providing an understanding of teachers’ perspectives and motivation to implement evidence-based interventions. To explore these issues, the study will focus on teachers’ experiences with implementing the evidence-based reading curriculum Reading Street. Reading Street was selected for this study as several school districts have mandated that it be used as the core reading program. The nature of it being mandated makes for a interesting investigation for two reasons. First, it means a large number of teachers are using it so there will be a wide subject pool. Second, teachers’ expectancies and values may be uniquely altered due to it being mandated. Unfortunately, Reading Street has not yet been evaluated by the U.S. Department of Education’s Institute of Education Sciences (IES). The IES rigorously evaluates educational programs and disseminates the results through the What Works Clearinghouse website. However, there is research supporting Reading Street’s efficacy. The development of the current program began with Scott Foresman’s Reading Links to Reading First, an intervention for struggling readers. In this study, children who received the Links to Reading First program outscored nonusers of the program with an effect size of 0.388 (Newman & Jaciw, 2005). Another early quasi-experimental study indicated that 88% of the users of the Scott Foresman program saw increases in state scaled reading achievement scores when comparing pre-implementation scores to post-implementation scores (Gatti, 2003). In addition to these studies, the developers of the program wanted independent research conducted 40   that met the rigorous criteria of the What Works Clearinghouse for evidence-based interventions. These evaluations indicated that Reading Street was effective in raising students’ reading achievement; however it did not improve scores over similar basal reader programs (Gatti, 2003). Overall, Reading Street is an effective program in promoting reading achievement in its users based on evidence-based research practices, but it does not outperform other reading programs. This study will use teachers’ experiences with implementing Reading Street as a vehicle for understanding their motivation to implement an evidence-based intervention with treatment integrity. 41   Chapter 3 Methods Participants For this study, 154 teachers from eight elementary schools across four school districts within a larger regional school district were invited to participate. Inclusion criteria for the study were that the teacher must teach reading using Reading Street in the general education setting in grades kindergarten through fifth grade. Of the participating teachers, 46 started the survey with 42 teachers completing the survey. All participants in the survey were also invited to participate in focus groups. There were 17 teachers who indicated interest in participating in the focus groups on the survey. Of these teachers 13 attended one of four focus groups. The 42 participants taught kindergarten, first grade, second grade, third grade, fourth grade, and fifth grade. Participants’ years of experience were five or less years (n=8), six to ten years (n=13), eleven to fifteen years (n=8), and over fifteen years (n=13). Participants’ education levels were bachelor’s degree (n=2), some graduate courses (n=7), master’s degree (n= 21), master’s degree plus some graduate credits (n=12). Of the 42 participants, 25 implemented Reading Street for one year and 17 implemented it for two years. District 1 (n=3) and District 2 (n=11) were small, rural districts with one elementary school. District 3 (n=13) was a small, rural district with two elementary schools. One of these schools was a K-2 building, and the other school was a 3-5 building. District 4 (n=15) is a larger, suburban district with six K-4 elementary schools. In all four districts, teachers were expected to implement the same core reading curriculum as mandated by their school districts. Inclusion criteria for the study were that the teacher must teach reading using Reading Street in the general education setting in grades kindergarten through fifth grade. 42   The focus groups included 13 participants from three districts. The demographic make-up of the groups is shown in Table 2 (names are pseudonyms). Table 2 Demographic Information of Focus Group Participants Name Group Grade Taught District Cora 1 Kindergarten 3 Mary 1 Kindergarten 1 Edith 2 1 grade Sybil 2 Isobel st 4 4 grade th 3 2 Kindergarten 1 Violet 3 2 Elsie 3 2 Sarah nd grade 4 grade 4 3 3 grade 1 Anna 3 Kindergarten 3 Daisy 4 Kindergarten 4 Ethel 4 2 Martha 4 4 Matthew 4 nd rd nd grade 1 grade 1 4 grade 4 th th To recruit participants, the researcher contacted the principals at each school and asked to speak briefly at a staff meeting to introduce the study and to distribute flyers with information pertaining to the study. They were informed that this study involved taking a survey on their 43   views on Reading Street and on their implementation of Reading Street. Teachers were also informed of the opportunity to take part in more in-depth focus groups on the same topics. A follow-up e-mail about the study was sent to the teachers via email addresses accessed from school websites. Teachers were assured of confidentiality, as none of the data collected were presented in individual form. The Teacher Activity Log information that was collected through the ISD used a code to ensure teacher anonymity. This same code system was used for the teacher information on the survey instead of identifying data such as name or email address. This code was used to align teacher survey data to classroom level student outcome data. In addition, administrators were not informed of which teachers participated and which did not. Confidentiality information and institutional review board information were reviewed on the online survey prior to teachers consenting to complete it, and it was also reviewed prior to the focus groups. Research Design This study used a mixed methods design to explore the issues surrounding teacher motivation to implement an evidence-based intervention and the relationships among the variables of motivation, treatment integrity, and student outcomes. Qualitative methods including open-ended survey questions and focus groups were used to gain an in-depth understanding of teachers’ perspectives on the implementation of the EBI, Reading Street. With these methods, treatment integrity and motivation were also examined. A correlational design was used to examine how motivation and treatment integrity were related to student reading achievement. Each variable was correlated with student outcomes separately. The intercorrelations among these independent variables were also examined to determine the strength and direction of the associations between them. 44   Data Sources Treatment integrity, measured in terms of implementation adherence, was assessed using extant data collected by the regional school district. Student reading achievement data were obtained from existing data collected by the schools. Data about teachers’ experiences with EBI and their motivation to use an EBI was collected as part of the study. The data were collected through a teacher survey and focus groups. The survey measured teacher perspectives on evidence-based interventions, teacher motivation to implement the reading curriculum, and teacher knowledge about treatment integrity. The focus groups provided data about teachers’ experiences and perspectives on implementing Reading Street and were formed based on the participants’ availability and the schools at which the teachers worked. Groups were formed so that there was a diversity of schools represented. Evidence-Based Intervention: Reading Street Given the regional school district’s use of a response to intervention framework, the local school districts were required to adopt an evidence-based core reading curriculum. The schools in this study used Scott Foresman’s Reading Street (Pearson, 2011) as the core reading curriculum. The program is based on several instructional methods that have been empirically supported. First, the program focuses on the priority skills that were identified in the National Reading Panel’s Teaching Children to Read report. These five areas include phonemic awareness, phonics, fluency, vocabulary, and comprehension. Moreover, Reading Street delivers these five skills in developmentally appropriate sequences. The second evidence-based feature in this program is differentiated instruction. Differentiated instruction is a method that has been shown through meta-analysis to improve student outcomes (Vaughn et al., 2003). Finally, Reading Street uses progress monitoring to ensure student progress throughout the year. Progress 45   monitoring has also been shown through research to improve student outcomes (Fuchs, Mock, Morgan, & Young, 2003). Reading Street is composed of several evidence-based strategies, and the developers continue to study the program as a whole to verify it as an evidence-based intervention. One of the goals of the creators of Reading Street was for the program to be “teacher friendly” and easy to use. The format of the program is generally the same across all grade levels. A daily implementation of Reading Street involves beginning with Concept Talk, a time when the “question of the week” is discussed. Next an activity called Anchored Talk is used. This activity surrounds the question of the week and is designed to develop students’ oral vocabulary. Oral vocabulary is also addressed through Amazing Words activities and readings. Commonly, Amazing Words involves the teacher introducing vocabulary words and then reading a story containing these words. Phonics is also a daily part of the Reading Street Routine. The phonics skills are explicitly taught with modeling, guided practice, and corrective feedback. The skills are then practiced independently of the context and then in context. The student workbook, Reader’s and Writer’s Notebook, includes independent practice for phonics skills. Reading Street also includes Decodable Practice Readers that are designed to reinforce phonics skills. The Decodable Practice Readers are also used to practice fluency through having students read the stories orally three to four times. To address the area of reading comprehension Reading Street includes a section called Skill  Strategy. This section encompasses many aspects of reading comprehension. Some examples include compare and contrast, author’s purpose, main ideas and details, drawing conclusions, fact and opinion, and plot and theme. There is also a Vocabulary Skill  46   Strategy section. In this section topics such as homonyms, antonyms, compound words, prefixes and suffixes, and context clues are covered. Reading Street also covers language arts on a daily basis. Grammar conventions, spelling, and writing are covered everyday. A topic for each of these areas is addressed for the entire week. For example, in one week the grammar convention could be singular possessive nouns and for writing the topic could be writing a friendly letter. For spelling, the words for the week focus around words that share certain characteristics. For example, a weekly list might include the consonant diagraphs /sh/, /th/, and /ch/. The teacher’s manual for Reading Street includes a weekly overview for how to teach the curriculum so that English language learners are able to benefit from it. The manual also includes specific activities for differentiated instruction. Reading Street differentiates with four groups including strategic intervention, on level, advanced, and English language learners. For ELL students and students needing additional support these activities preview and reinforce ideas discussed in whole group times. For on-level and advanced students the activities are designed to expand their skills. The activities typically focus around leveled readers that are appropriate for the skill level of each group. Variables and Measures In this study, the main variables of interest were teacher views of evidence-based intervention, teachers’ perspectives on treatment integrity, and teachers’ motivation. Motivation and treatment integrity were both assessed by examining more specific factors that form the larger construct. Motivation was examined in terms of expectancy, value, and cost. The construct of treatment integrity included the dimensions of adherence, quality of delivery, exposure, participant responsiveness, and program differentiation; however, due to constraints only 47   adherence, exposure, and participant responsiveness were measured. The other dimensions of quality of delivery and program differentiation were considered through qualitative analysis of focus group data. Teacher views of evidence-based intervention. The question, “How are evidence-based interventions a part of your teaching,” was asked on both the survey and in the focus groups to gain insight on how teachers view evidence-based interventions. Please see Appendix A and B for survey and focus group questions. Motivation. This study sought to investigate teachers’ motivation to engage in implementing Reading Street with treatment integrity. Motivation was conceptualized through an expectancy-value framework (Eccles & Wigfield, 2002). Previous research that applied an expectancy-value framework to motivation suggested expectancy, value, and cost all relate to teacher motivation to implement educational practices in their classrooms (Abrami et al., 2004; Foley, 2011; Wozney, 2006). Therefore, these three aspects of motivation were measured in this study with a teacher survey. The constructs of value and cost are closely related. However, they were broken into separate categories because of their oppositional nature to each other. Value captures the positive views toward engaging in a task, whereas, cost captures the negative aspects of engaging in the task. Expectancy. The variable of expectancy was measured with items adapted from the Cooperative Learning Implementation Questionnaire (Abrami et al., 2004). On the original scale the items assessing expectancy had a Cronbach alpha of 0.86. The adaptations that occurred were changing the phrase “cooperative learning” to “Reading Street”. On the present scale a Cronbach alpha of 0.83 was obtained. 48   Value. Value was assessed on the same measure as expectancy and cost; however, it included measures adapted from Wozney (2006). Wozney (2006) investigated teacher implementation of technology in the classrooms and adapted the CLIQ (Abrami et al., 2004) to reflect the implementation of technology instead of cooperative learning. The Cronbach alpha achieved for the value items in Wozney (2006) was higher (α=0.86) than the alpha achieved for the value items in Abrami et al. (2004), (01=0.74). Therefore, the value items from Wozney (2006) were used in the present scale. The items were adapted by replacing “technology use” with “Reading Street.” These items were then used to replace the value items in the CLIQ, which allowed the order of expectancy, value, and cost items to remain consistent with the original measure. On the present scale, a Cronbach alpha of 0.93 was achieved. Cost. The variable of cost was assessed with the cost items from the CLIQ (Abrami et al., 2004). The Cronbach alpha achieved for these items was 0.87. Similar to the other two aspects of motivation, these items were adapted by replacing cooperative learning with Reading Street. On the present scale a Cronbach alpha of 0.71 was achieved. See Appendix A for a copy of the complete Teacher Motivation Survey. Treatment integrity. The variable of treatment integrity was considered a multidimensional construct in this study. Adherence, exposure, and participant responsiveness were measured with a Teacher Activity Log and a survey. The other dimensions of quality of delivery and program differentiation were assessed using the focus group data. The multidimensional conceptualization of treatment integrity was used as a basis for designing tfocus group questions and for analyzing teachers’ responses to open-ended survey and focus group data. 49   Adherence. Adherence was measured through a teacher self-report measure called the Teacher Activity Log that was completed in the schools. This measure was called the Teacher Activity Log rather than a treatment integrity log to minimize response bias and perceived evaluation. As part of existing school practices, teachers completed this measure for one week out of each month. The weeks that data was to be collected were determined based on school schedules and which weeks it would be most likely that the teachers would be able to monitor their implementation. For example, the short weeks or weeks right before holiday breaks were avoided. At the beginning of the data collection week, the teachers implementing Reading Street received an email that it was an Activity Log data collection week. The email contained a PDF version of the data collection sheet, and teachers were encouraged to print this and to use it to keep track of their implementation for the week. On Friday, a second email was sent out with an email link to the online version of this measure, and teachers were asked to complete this online measure. This data collection method was used to reduce the amount of time teachers spent monitoring treatment integrity while still getting a sample of treatment integrity data. The adherence measure was created collaboratively by the primary investigator and the ISD literacy consultants. The developers of the measure examined the list of “non-negotiable” components included in the Reading Street administrator manual and developed it into a userfriendly format to easily report the adherence to these components. In the schools this measure was known as the Reading Street Weekly Activities Log. The measure was specific to each grade level as the Reading Street program changes for each grade. However, the versions were similar and used the same format. The differences between the versions involved specific components that may be present at one grade level and not at another. For example, in Kindergarten, phonemic awareness was an important component, but this early literacy skill was not part of the 50   program in the older grades. The logs for each grade are available in Appendix C. A total level of adherence was calculated through dividing the total number of components intended to be implemented by the actual number of components implemented. Participant responsiveness. There was one item on the teacher reported Reading Street Weekly Activities measure that asked teachers to report students’ engagement in the independent activities that are a part of the program. Teachers rated how engaged their students were in the program on a five point Likert scale. This rating occurred every day for one week of the month. Exposure. Exposure refers to how much of the intervention was implemented. Given that Reading Street is the core curriculum for the schools, there should be uniform use of the program across teachers and schools. However, this assumption may or may not hold and, therefore, was measured through asking teachers to report how many times per week they implemented the program and to report how many minutes per day they implemented the program. Teachers were asked these questions on the self-report survey. Teacher understanding of treatment integrity. To gain insight into how teachers conceptualized treatment integrity, an open-ended question was included on the teacher selfreport survey. This question was analyzed qualitatively with content analysis. Teachers’ responses were analyzed based on units of thoughts. These units were then used to identify themes. Curriculum-based measures. Curriculum-based measures were used as the outcome variable in this study. The schools that participated in this study used curriculum-based measures to collect universal screening data on all students. This data was used to assess student outcomes. Some schools used Dynamic Indicators of Early Literacy Skills Next Edition (DIBELS Next) and others used Aimsweb. Curriculum-based measures such as these are valid and reliable 51   measure of reading achievement, as they have been observed to be high correlated with other constructs of reading competence (Shinn, Good, Knutson, Tilly & Collins, 1992). Curriculumbased measures have also been found to be reliable with test-retest reliability ranging from 0.820.92 and alternate form reliability ranging from 0.89-.096 (Shinn & Shinn, 2002). In this study, student CBM data was used to create a dichotomous variable of reading on grade level. Cut off scores for reading below grade level were determined using Aimsweb or DIBELS data depending on which measure the school employs for universal screening. First through sixth grade students scoring at least one standard deviation below the reading fluency subtest mean of the nationally normed benchmarks provided by the developers of these assessments,were considered reading below grade level. Kindergarten students were not tested for reading fluency. The phoneme segmentation fluency subtests were used to determine cut-off scores, as this subtest has been shown to be the most predictive of later reading achievement for early readers (Good & Kaminski, 1996). Given that the schools did not use the same reading measures, a consistent dependent variable was developed to compare students across schools. This consistency was accomplished through conceptualizing the dependent variable as the dichotomous variable of “reading on grade level”. This method also accounted for the fact that the measures for each grade are not equivalent. Please see Appendix C for the cutoff scores for reading on grade level. Using these subtests and cut-off scores, a percent of students at grade level was calculated for each participant in the study and used for the correlational analyses Procedures Survey data. The survey was hosted on an online survey website. After the study was introduced in person, an e-mail was sent to the teachers in participating schools with a link to the online survey. This e-mail served to remind teachers of the study and to invite them again to 52   participate in the study. Teachers received a $15.00 gift certificate to an online bookstore for completing the survey. Focus groups. There were a total of four focus groups and each group had two to four teachers. Teachers received a $35.00 gift certificate for participating in a focus group. The focus groups were composed of teachers from at least two different districts. It was thought that teachers would be more comfortable talking about the potentially sensitive issue of implementation if they were not concerned about what their colleagues thought of their responses. Krueger and Casey (2009) suggest three to four focus groups as the rule of thumb for reaching saturation, the point at which a range of ideas have been expressed and new information is not forth coming. Focus groups were led through using a questioning route (Kruger & Casey, 2009) The questioning route included an opening question, introductory questions, transition questions, key questions, and an ending question. The questioning route in this study was designed to elicit conversation around teachers’ opinions on evidence-based intervention broadly, their motivation to implement Reading Street specifically, and the implementation of Reading Street with treatment integrity. Please see Appendix D for the questioning route for this study. The moderator in this study was the primary researcher. According to Krueger and Casey (2009) the moderator should have several characteristics including, understands the topic and purpose of the study, communicates clearly, is open and not defensive, and can get the most useful information. The moderator was comfortable with the questioning route and moderated the focus groups in a comfortable, conversational way. As suggested in Krueger and Casey (2009) no more than two focus groups were conducted in one day, so that the moderator did not become mentally and emotionally fatigued. 53   The focus groups were held at a neutral location for teachers. The location was also selected based on convenience of travel for the teachers who worked in schools spread across an entire county. It was important for the location to be set away from areas the teachers may have associated with school administration or the intermediate school district to ensure they were comfortable in giving candid responses about their motivation for and experiences with Reading Street. At the beginning of the focus groups, participants were given informed consent paperwork as they arrived. Any questions participants had were addressed at that time. Light refreshments were served to increase the comfort of the situation. All technical equipment was set up and tested prior to participants’ arrival to prevent disrupting the session. The questioning route was designed to take about 90 minutes to complete. Once all participants arrived and the focus group moved into the questioning route portion, the sessions were audio recorded with a digital voice recorder and later transcribed. The session began with a welcome, an overview of the topic and the study, and ground rules for the focus group. The ground rules for the focus group were designed to encourage participation through telling them there are no right or wrong answers and that all opinions are valued. After the introduction, the moderator began with the opening question on the questioning route. To promote discussion, the moderator used the two essential techniques outlined in Krueger and Casey (2009). The first technique was the five-second pause. This technique was used to elicit additional points of view. The second technique used was the probe. This technique was an additional request for information. Some probes suggested in Krueger and Casey (2009) included: “Would you explain further, can you give us an example, tell us more, and please describe what you mean.” (p. 99). The moderator also encouraged differing points of viewing by 54   asking questions such as, “does anyone see it differently?” (Krueger and Casey, 2009, p. 99). The session concluded with thanking the participants for their time and sharing their thoughts. The participants then each received a gift card for their participation. 55   Chapter 4 Results Data Analyses Descriptive statistics. Descriptive statistics were used to examine the average level and range of treatment integrity achieved during implementation and the average level and range of teacher motivation to implement Reading Street with treatment integrity. For the level of treatment integrity achieved, the dimensions of adherence, exposure, and participant responsiveness were examined. Measures of central tendency were calculated for the teacher motivation variables of expectancy, value, and cost. Pearson’s R Correlation. Correlations were calculated with Pearson’s R to determine the relation between teacher motivation, level of treatment integrity achieved, and student outcomes. The correlation coefficients were used to examine the strength and the direction of the relationship between the variables. Three different correlations were calculated. First, a the level of teacher motivation from the survey data was correlated with teacher level of adherence. Second, a correlation between level of adherence and student outcomes was calculated. Finally, a correlation between teacher motivation and student outcomes was calculated. Qualitative analysis. Qualitative procedures were used to analyze data from the focus groups. Marshall and Ross (1999) stated that there are typically eight topics that need to be explicitly discussed with qualitative research methods including rationale for qualitative methods, population selection, researcher’s role, data collection methods, data management, data analysis strategy, trustworthiness features, and a timeline. Rationale for qualitative methods. Qualitative methods were determined to be appropriate because the goals of this study were both descriptive and exploratory. The study documented and 56   described teacher motivation and views on evidence-based intervention and treatment integrity. It also explored how the mandate for using evidence-based interventions in schools is being practiced and experienced by teachers. Marshall & Rossman (1999) suggested that qualitative methods are justified when “research seeks to explore where and why policy and local knowledge and practice are at odds.” (p. 57). This study used qualitative methods to explore this potential discrepancy in depth. The analysis looked for common threads in the teachers’ responses to questions about their motivation and their process of implementation of Reading Street. The main purpose of this analysis was to better understand teachers’ perspectives on and motivation for implementing Reading Street and their understanding of treatment integrity. Marshall and Rossman (1999) indicated that qualitative research should emerge rather than be predefined. As the primary researcher engaged with teachers and administrators about the study, it evolved. Originally, the study was intended to be quantitative in nature with a focus on measuring treatment integrity in several different ways. Interactions with administrators at the ISD and teachers at the schools within the ISD made it clear that this type of study would not be feasible. In these schools, the idea of evidence-based intervention was still new and seemingly uncomfortable for many teachers. The very nature of treatment integrity can feel threatening to teachers, and administrators highly cautioned against observations, due to the potential for them to seem evaluative. Although this was not data in a quantitative sense, these interactions were a form of data. The interactions indicated that teachers were not ready to participate in a study focused on treatment integrity and that a study that sought to gain their views on evidence-based intervention was likely to be more valuable both to the systems in which they worked and to theory. 57   Site selection. The site in which this research took place was selected because it is a school system that is using evidence-based interventions. To gain understanding about teachers’ views on EBI, it was thought the richest data would come from teachers with some experience with using them. Over the past two years the school districts in the ISD have purchased and mandated the use of the evidence-based intervention, Reading Street. The use of the same evidence-based intervention gives the study a consistent focus. Researcher’s role. To be able to gain entrance into a school setting to conduct research about the implementation of EBIs in practical settings, I worked to build trust with administrators and teachers. I first contacted a school psychologist and an RtI implementation coordinator at the local ISD to share my research interests. They were eager to explore opportunities to collaborate on developing measures of treatment integrity given that many of the ISD districts were starting to use Reading Street. After meeting with administrators at the ISD, we agreed that I would act as a consultant to help develop treatment integrity measures and in exchange, the district would support my research. This process helped to develop trust and a working relationship at the ISD level; however, it was equally important to gain the trust of teachers whose perspectives were the focus of the research.. . The use of Reading Street was surrounded by controversy in these districts and teachers had differing opinions about its value in the classroom. This larger context was an important consideration when designing and conducting the study. I recognized how important building strong relationships with the teachers would be to allow them to feel comfortable sharing their views honestly on this controversial topic. Instead of sending out a mass e-mail soliciting participants for the study, I took part in staff meetings at the participating schools and explained my research to the teachers in person. I believed that allowing teachers to “put a face” to the 58   researcher would be the first step in eliciting trust in the study. Through building trust, I hoped that teachers would feel comfortable in giving honest, thoughtful views on evidence-based intervention, treatment integrity, and their motivation for using Reading Street. Data collection methods. Marshall and Rossman (1999) described four data collection methods involved with qualitative research including participation in the setting, direct observation, in-depth interviewing, and analyzing documents and material culture. Of these methods, participation in the setting, in-depth focus groups, and analyzing documents were used. Participation in the setting occurred through informal consulting with the RtI team on the development of a treatment integrity measure and participating in several meetings on Reading Street and treatment integrity that were attended by principals and teachers. Analyzing documents occurred through examining the Teacher Activity Logs. Finally, in-depth interviews took the form of focus groups. Data management. The data were managed in several ways. First, focus group data were audio recorded and then transcribed into a word processing program. These transcriptions were then used for the analysis. The second data management system was the extant CBM data and Teacher Activity Log data that were stored in an electronic spreadsheet. These data were stored by a teacher code and did not contain any identifying teacher or student data. Data analysis strategy. Data were analyzed with both “template” and “editing” analysis strategies (Crabtree & Miller, 1992). Template strategies involved applying preconfigured codes to the data. For analyzing teachers’ understanding of treatment integrity preconfigured codes based on Dane and Schneider’s (1998) five dimensions of treatment integrity were used. For the data on teachers’ views on evidence-based intervention and implementing Reading Street, the 59   analysis was less structured and involved searching text for codes and categories of meaning. The primary research questions were used as a base when searching the text. Marshall and Ross’s (1999) analytic procedures guided the analysis. Data were organized with the qualitative research computer program, ATLAS.ti. Next data were analyzed to generate codes, themes, and overall ideas throughout the groups. Throughout the process of coding, I “tested for emergent understandings” (Marshall & Ross, 1999, p. 157) through checking how useful the categories and codes were in understanding the posed research questions. Simultaneously I also “searched for alternative explanations” (Marshall & Ross, 1999, p. 157). In this phase, the categories and codes were challenged through critically thinking about other reasons for the data. Trustworthiness features. Brantlinger, Jimenez, Klingner, Pugach, and Richardson (2005) discuss trustworthiness or credibility features in special education research. These measures include triangulation, disconfirming evidence, researcher reflexivity, member checks, collaborative work, external auditors, peer debriefing, audit trail, prolonged field engagement, thick detailed description, particularizability. To establish the reliability and validity of the qualitative data collected in this study several of these measures were used. First, these data were triangulated through the use of varied data sources including survey data, focus group data, and extant data (CBM data and Teacher Activity Logs). These data sources were examined for convergence and consistency. The second measure used was researcher reflexivity. This measure helped to understand and disclose any assumptions and biases I may have held. I come from a school psychology background. As a whole the field of school psychology emphasizes the importance of evidencebased intervention, and using evidence-based interventions is part of the National Association of 60   School Psychologist’s practice model (NASP, 2010). As such, I have been trained that using evidence-based interventions with treatment integrity is an important aspect of educational service delivery. This assumption may not be shared by teachers and others in schools that come from different areas of education. I may be biased toward looking favorably upon using evidence-based interventions and treatment integrity and was mindful of this when conducting the study. The third measure that was used was collaborative work. Collaboration took the form of writing analytic memos and meeting with dissertation committee members as the data were coded and analyzed. Finally, the fourth measure that was used to establish trustworthiness was particularizability. This involved using thick description when presenting the data so that “readers can determine the degree of transferability to their own situations” (Brantlinger et al. 2005, p. 201). Using this feature of trustworthiness was important to this study because it allowed those wishing to apply this research to determine if the results of the study are applicable to their school and teachers. The data sources and types of analyses used to analyze each research question is summarized in Table 3. Table 3 Data Sources and Data Analyses by Research Question Research Question Data Sources 1. What are teachers’ experiences with and Focus groups perspectives on evidence-based interventions and (N=13) implementing them in their classrooms? 2. What have teachers’ experiences been specifically with implementing Reading Street? 61 Focus groups (N=13) Data Analyses Qualitative analyses Qualitative analyses   Table 3 Cont’d 3. What is the teachers’ level of motivation to use Reading Street? a. Do teachers expect Reading Street to work? Why? b. Do teachers view Reading Street as important to successfully teaching children to read? 3. Focus groups (N=13) a. Survey (N= 42); focus groups (N=13) b. Survey (N=42); focus groups (N=13) 3. Qualitative analyses a. Descriptive statistics, qualitative analyses b. Descriptive statistics, qualitative analyses 4. How do teachers understand treatment integrity? Do they view treatment integrity as unidimensional or multidimensional? Focus groups (N=13) 5. How are motivation, treatment integrity, and student 5a. Survey, outcomes correlated? Teacher Activity Logs (N=42) a. How is teacher motivation related to treatment integrity? 5b. Survey (N=42), CBM b. How is teacher motivation related to students’ data (N=42) reading achievement? 5c. Teacher c. How is treatment integrity related to students’ activity log reading achievement? (N=42), CBM data (N=42) Qualitative analyses Pearson’s Correlations Missing Data There were several instances of missing data in this study. Ten of the participants who completed the survey did not complete any of the treatment integrity logs, and therefore, average treatment integrity data for these participants was not calculated. There were five participants 62   who took the survey but did report any curriculum-based measure data, resulting in no student outcome data for these participants. Two participants did not complete every item on the motivation survey resulting in missing values for the composite motivation score. Little’s missing completely at random (MCAR) tests indicated that there were no meaningful patterns to any of the missing data. Therefore, maximum likelihood was used to estimate the missing values for all of the missing data, as it is considered one of the most appropriate methods for handling data that is missing at random (Baraldi & Enders, 2009). Research Question 1: What are teachers’ knowledge of and experiences with implementing evidence-based interventions? This research question was posed as it was hypothesized that teachers’ knowledge and experiences with implementing EBIs would be related to their expectancy and value beliefs for their future use. Data to answer this question were collected from the open-ended questions on the survey and from the focus groups. Important topics discussed in the focus groups included assessment, response to intervention, classroom practices that are not EBIs, implementation barriers, and aspects of treatment integrity. These five themes were interrelated and it was difficult to separate assessment from response to intervention as universal screeners such as DIBELS and Aimsweb are used to inform intervention. Assessment. Although the questioning route did not specifically address the issue of assessment, it was the third most commonly discussed construct with 39 references to it across all of the focus groups. Teachers made reference to specific assessments such DIBELS, Aimsweb, Rigby, and Developmental Reading Assessment. Participants also made more specific 1 references to the process of assessment. Mary , kindergarten teacher, explained:                                                                                                                 1 To protect teacher confidentiality, pseudonyms were used. 63   "We try to do our assessments on Fridays. So, if you’re on a four-day week, you’re trying to teach and assess. So, it can be tricky, but sometimes those are the weeks we don’t do a whole lot of the formal progress monitoring stuff, we kinda just do that with the kids who need it the most." Teachers also discussed their views and thoughts about the form of assessment. Isobel, a kindergarten teacher, shared, "I can’t say that I would use Aimsweb as an indicator of [progress]. I’m not a fan and I just have too many kids that are insecure and you know you have strangers taking the kids and they’re quiet." Quotations such as these indicated that teachers viewed assessment as a part of the process of using evidence-based interventions. Response-to Intervention. When discussing how evidence-based interventions were used within their schools, several teachers answered in terms of a response to intervention framework. In particular, teachers referenced pull-out or tier- two interventions. Teachers also discussed how curriculum-based measures such as DIBELS and Aimsweb are used to monitor students' progress with these interventions. Cora, a kindergarten teacher, shared how her school implements RtI: “We are an RtI school so we’re doing, a lot of our interventions are Road to the Code. We use DIBELS as our indicator for literacy. We don’t have math interventions at this point that’s something that we’re working towards. In terms of tier two things, we have implemented K-PALS in the past in the classroom, but that has not happened, it’s not a universal thing and not something that’s even I think occurring at all this year…We have a lot of behavior support pieces in places in terms of check in check out.” Sybil, a teacher at another district, shared how RtI functions in her school: 64   “We definitely have a good deal of tier two interventions going on with reading, where kids are pulled out daily for forty minutes. Kids that are identified as having or needing strategic or intense intervention, so they get the daily reading, one on one depending upon what their level or what their weakness is. It might be a phonics group. It might be a six-minute solution group where they’re working on fluency. It might be Read Naturally on the computer where they’re working on fluency and comprehension. So, that happens everyday within our school building. And we’re just kind of, focus is kind of steering now into math, using AIMS Web data to collect and kind of prescribe what sort of interventions a child might need.” Finally, Matthew, a fourth grade teacher from the same district as Sybil gave his view on RtI in another group: “Well in our school, for thirty-five minutes a day a group of students leaves the general ed. classroom and gets, has intervention time with separate teachers during which kids who do not have the intervention stay back in the general ed classroom. And generally what the teachers have been doing are things like science, and social studies, maybe writing, but reading and math are not touched at that time the belief being that those two subject areas are the subject areas that are most important to hit with everybody…So, the whole group is tested a couple times a year, but every six weeks the borderline and kids who are already in the interventions get re-tested just to make sure they are where they are supposed to be and see if they are responding to an intervention.” Evidence-based Interventions in Classroom Practice. When discussing evidence-based interventions, teachers frequently discussed specific, commercially available programs. Some of 65   the programs mentioned were reading curriculums such as Balanced Literacy, Reading Workshop, and Daily Five. Teachers also mentioned specific interventions designed for small groups such as Kindergarten Peer Assisted Learning Strategies, My Sidewalks, Road to the Code, Rewards, Six-Minute Solution, Read Naturally, Reading Recovery, Peer Assisted Learning Strategies, RAZ, and Rocket Math. Teachers also mentioned using websites such as Florida Center for Reading Research and InterventionCentral.com to find evidence-based interventions for their classrooms. Mary, a kindergarten teacher says. "I’ve gone to like interventioncentral.com and they’ve got different intervention techniques. Sometimes its just different ways to use flashcards that have been research based. " Daisy, a kindergarten teacher from a different district discussed how at her school teachers incorporated EBIs that the regional area school district had suggested as well as others and how this has been difficult to achieve, especially in terms of generalizing evidence-based intervention to all aspects of the day and not just reading: “We’ve incorporated some other evidence-based interventions in the classroom as well that the ISD has kind of pushed. K-PALS is one, Road to the Code, and kind of trying to figure out how all these pieces kind of when I feel like we’re giving so much isolation instruction how we can help that transfer come back to the classroom and that’s the piece right now that I’m kind of feeling like we have a lot of tools, a lot of things we can offer kids, but it’s that piece of taking that isolated evidence- based instruction and bringing that greater application back to the classroom and applying it to their learning for the other five and half hours of the day they’re at school.” In addition to speaking about the evidence-based interventions that they were using, 66   teachers also talked about the use of non-evidenced-based strategies in classroom practices. The non-evidence-based teaching practices teachers referenced were generally aimed at increasing the amount of reading students were hearing or doing. Mary, a kindergarten teacher stated, "I felt like I’ve really had to supplement Reading Street with ‘read alouds’, regular old ‘read alouds’." Violet, a second grade teacher stated: "I don’t know that what I do is a evidence-based program per se, but I believe there’s a lot of research back there and evidence to support the more reading they’re doing, the more engaged they are with the reading they’re doing that’s on their independent level, then the stronger they’re going to get as a reader, particularly when you’re doing them with guidance." Other teachers referenced finding other ways to present material or to re-teach materials. For example, "we have time set aside as well and then we might not necessarily be using an evidence-based approach, we might be using just re-teaching of some of the things that go along, using some of lessons again, using other books to read with them." In another example, Edith, first grade teacher discussed an alternative way to teach phonics: “I do a dance for the phonics. I guess that’s an adaptation, like I had learned it before and it’s not research-based just cause she hasn’t done it, but there is a teacher in Ohio who sells this phonics dance, so I do that and I’ll look and see what phonics is coming up. So, I’ll teach them the dance a week or two ahead of time and it’s like oh look we know this because we know the dance to it or whatever." Implementation Barriers. The focus group questions on evidence-based interventions also gave teachers an opportunity to discuss why EBIs are difficult to implement in the classroom. All 67   of the focus groups discussed implementation barriers at some point during the focus groups. There were several barriers identified across the groups. Commonly mentioned barriers are displayed in Table 4 below Table 4 Frequency of Mentions of Implementation Barriers Frequency of Mentions for each Barrier Characteristics of Students 32 Time Factors 31 Quality of Materials 22 Scheduling 13 Financial Considerations 11 Organizational Issues 6 In regards to client characteristics, one teacher discussed how she thought that the demands of Reading Street were too high for her students. “I would say especially at the kindergarten level, how much of that is whole group. They sit down on the carpet and we teach them and it’s really a mater of beyond the targets too, I have a limited amount of time I’m going to maintain focus on the carpet. And those lessons go on and on, no matter how much I try to cram them in. It’s not physically possible to hold a kindergartner’s attention to cover everything that is in a day." The characteristic of age was also a barrier for kindergarten teacher Isobel. “" I can’t say I do it with fidelity because with five and six year olds, teaching them to work in partnerships has not worked for me." 68   Ethel also discussed how the demands of Reading Street were too high for her students but due to their instructional levels instead of their ages. She is concerned that Reading Street inhibits her from meeting individual student needs. "I always felt very frustrated because I am a very individual oriented teacher, also, but with twenty-seven individuals, two thirds of whom are below level, it’s just really struggling to meet everybody’s needs, especially when the system isn’t there. Because we not only didn’t have the interventions organized for a while, we didn’t have a system for studying struggling kids, write a path for them. So, everyone’s kinda going, what are we supposed to do? These kids aren’t making progress and we got nothing. Violet, a fourth grade teacher, also voiced concern about providing instruction at students’ instructional with Reading Street. ‘So you’re sitting there sometimes with children at the table and sometimes going that book is too easy for them or that book is too hard." Martha, a fourth grade teacher, also discussed the influence that the class characteristics as a whole can have on the implementation of the program. "[Reading Street] is very complex and I had a difficult class and it’s so, you know, there’s so much direct teaching to prepare the kids for.” Finally, Mary brought up the cultural context of her students as an important characteristic to consider. “And there’s really not a lot of time for [read alouds], which is sad, but I feel like that is so important because I have a population of kids who do not have books at home and they do not get read to very often." Mary’s concern over ensuring her students are read to overlaps with the second most 69   commonly referenced barrier to implementation, time factors. Teachers felt that Reading Street and other EBIs took a great deal of time to learn and implement and that this inhibited them from using them effectively. For example, Sarah a third grade teacher stated, “And that’s one of the things I feel like right cause we’re trying all these new things all at once. You can’t do anything well if you’re doing a thousand things.” Ethel echoed her concerns stating, “and every time I would try to get something put together that would work, the time crunch with it and I would never get it going." Teachers also discussed how the time consuming nature of EBIs influence the selection process and professional development. Daisy stated, "our evidence-based interventions aren’t necessarily maybe the best, but the ones that are most affordable or the ones that are most time efficient." Matthew discussed how going to trainings to learn how to use EBIs are time consuming. "Well sometimes, they want, I’m not really sure who they are but people want you to be trained at a time when you would otherwise be teaching. So, here’s some release time, well now I have to write sub plans" In addition to being concerned with effectively and efficiently implementing EBIs, teachers also discussed was an overall sense of urgency from administrators to use EBIs and Reading Street right away and how this pressure made it difficult to implement them well. For example, Daisy, a kindergarten teacher stated, "there’s also the piece about sense of time, the urgency of wanting to get interventions in place versus the time to help teachers get acquainted with those interventions and help to organize it." Another barrier was financial. Several teachers discussed how funding influenced which EBIs were used in her school Daisy stated, "our evidence-based interventions aren’t necessarily maybe the best, but the ones that are most affordable or the ones that are most time efficient." 70   Similarly, teachers discussed how Reading Street was selected for their schools. Matthew, a fourth grade teacher stated: "My best understanding of it is that the ISD was offering a really good deal on [Reading Street] and it was supposed to be, it was originally intended for lower elementary and then they said well if you do this, you could also get these ones for third, fourth and fifth grade for a song and people jumped." Cora made a similar statement in regards to her district, “ I think the fact that the ISD was kicking in part of the cost, was gonna contribute part of the cost, was a big deciding factor why to go with Reading Street over others." Mary discussed how money was a factor in deciding the scale of implementation in her district. “We could only implement, we couldn’t even do the whole school at once because it was a lot of money, so we did like third through fifth, I think and then K-two was the subsequent year." Another issue frequently discussed as hindering implementation of EBIs, especially for Reading Street, was the quality of materials. Sybil, a fourth grade teacher said, "we have small passages, a lot of times on worksheets that we’ll do. So, say it’s a compare and contrast, it might be a story about you know, kids racing water bottles pretending they are cars, that’s one we had, and the scientific reasoning that we’re reading with the kids was actually inaccurate." Other teachers notice the quality of the leveled readers is not up to par as well. Martha, also a fourth grade teacher said, "then you get to the leveled readers, the leveled readers are so stupid because they have these vocabulary words and they’re hiring writers to write a little dippy book using these words about this topic and they’re just dumb." In addition to teachers finding the instructional materials of poor quality, teachers also noted that the assessments did not meet her expectations as well. Sarah stated, 71   “Our tests don’t always seem to correlate with what we’ve been working on that week. Like if you think, you’ve ben working on a lot of compare and contrast, there’d be a lot of compare and contrast questions. Sometimes and sometimes it’s things from way back. And I get that they want to keep it in front of them, but it’s just really odd that some of those skills and strategies from that week aren’t the focal point." Isobel also discussed the assessments as not having the expected quality and how it influences the effectiveness of the assessments. “A lot of times I find with Reading Street, the questions…the pictures that they have choose from are not specific enough for them to be able to determine the correct answer." Teachers also noted that scheduling was a barrier to implementation of both Reading Street and the larger RtI process. Martha shared her struggles with scheduling. "[Pull-out intervention] is supposed to not touch your literacy time, but with the scheduling problems, the time that my kids are pulled out it actually is during my literacy time. So, I have to be careful to juggle and make sure I’m doing small groups at that time that don’t involve those kids in that whole group instruction. The scheduling has been a nightmare." Daisy also discussed the challenges of scheduling, especially in regards to “specials” (i.e. art, music, physical education, library time). "Part of it has to do with special schedules. I know that’s been a major challenge for us as well…I know that piece is huge hurdle." Ethel discussed the challenges of fitting everything into the school day and how it can result in some students missing out on having a well rounded educational experience. “I work so hard to put everything else in, including science…well these kids go to interventions, these kids 72   get science and social studies. I have such a problem with that.” Finally, teachers cited organizational issues as being barriers to implementation. For example Ethel, a second grade teacher, shared: "It took our building a while to get organized enough to know what are these interventions even going to do. In the meantime, the teachers were given mixed messages like you need to be doing such and such interventions on your own in your classroom. They didn’t give us anything they were supposed to." Daisy, a kindergarten teacher, identified communication at the system level as an issue: "And then there’s no communication with the people that… but we met like twice a year or twice this year…we’re all interventionists and everyone’s doing all these different things and I thought the whole point of us moving to this was that we’re going to all be on the same page. " Treatment Integrity. This discussion of implementation barriers often led into issues of treatment integrity. Teachers explained how these challenges to implementation created difficulties in implementing interventions to the full extent intended. Daisy, a kindergarten teacher, discussed how she had seen a lack of training lead to less treatment integrity. "I found too that when we’re asking the classroom teachers to add those in, if they’re not getting trained, then I’m not really certain that they’re really being implemented in the way that they’re supposed to be." Daisy went on to tie her concerns with financial issues and systemic issues with treatment integrity. "I hope that we’ll become more flexible, but part of it was because of the expense piece of it, and I think our curriculum director did not want to be wrong in their purchase of this extremely exorbitantly expensive program." Overall, this question explored teachers’ knowledge of and implementation of evidence- 73   based intervention. Themes that were identified included assessment, response to intervention, classroom practices that are not EBIs, implementation barriers, and aspects of treatment integrity. Teachers’ discussions stemmed from their personal experiences and largely they focused on barriers and drawbacks to evidence-based intervention in the classroom. However, teachers did discuss how they overcame some of the barriers and challenges they experienced. Research Question 2: What are teachers’ knowledge of and experiences with implementing Reading Street? This research question was designed to understand teachers’ perceptions specifically, of Reading Street, as it was hypothesized that these perceptions would influence their motivation to use this specific evidence-based program. Some of the themes that arose in this question were very similar to the themes above including implementation issues, classroom practices and EBI, and implementation barriers. This pattern is not surprising since the teachers frequently responded to question 1 in the context of Reading Street. During the focus groups teachers moved quickly to discussing Reading Street without much discussion of other general evidencebased interventions. Themes that arose that were unique to Reading Street included teacher factors influencing their beliefs and perceptions of Reading Street, system factors influencing teachers’ beliefs and perceptions of Reading Street, and the benefits and challenges of Reading Street. These constructs shed light on where some of the beliefs about Reading Street originated from as well as described teachers’ experiences with the implementation of Reading Street. Teacher Factors Influencing Beliefs and Perceptions of Reading Street. A major theme that emerged was the teachers’ pedagogical philosophy. Teachers’ beliefs about teaching broadly, were discussed in connection with their views on Reading Street. Cora, a kindergarten teacher, specifically defined herself in terms of her literacy pedagogy: 74   "One of things, I am a whole language person...I kinda taught myself to read. You know, I just picked up books and I read. That whole language piece just was really comfortable for me, just fit how I am. That piece I was really edgy about, just how directed the instruction was and how was I going to feel about that?" Sybil, a fourth grade teacher, compared Reading Street to the training she received and saw a discrepancy between the two. "It’s kind of the antithesis of how I was taught to teach and how I believe teaching should look in the classroom, or at least a fourth grade classroom." The fact that Reading Street promotes a very specific pedagogy was not problematic in all cases. Mary, a kindergarten teacher, discussed how Reading Street was a benefit to her team. "We had some big division between some of the things that we used to think philosophy-wise about. And [Reading Street] helped us, we’re all doing the same thing now." Violet, a second grade teacher, discussed how she incorporated Reading Street with her own teaching style: "I think in a lot of ways, it’s your teaching, it’s the good teaching…I haven’t thrown this program to the side, but I look at it and go okay here’s the text that I’m using, here’s the skill, the concept that I’m trying to get across. How am I going to go about doing that with these seven and eight year olds in front of me are going to be able to grasp? What do I need to do as a teacher, how do I have to my classroom set up? So, I didn’t expect it to make a difference one way or the other." In addition to teachers' pedagogical beliefs, their views on the nature and purpose of reading influenced their perceptions of Reading Street. One of the problems that teachers frequently noted about Reading Street was that it does not promote enjoyment of reading or allow students to read literature in its entirety. Sybil, a fourth grade teacher stated, "I don’t like 75   that we don’t get to whole literature. We read snippets, snippets and we never have time for anything that’s kinda that whole-istic feel." Similarly, Cora a kindergarten teacher, described how she felt that she was not able to get to read to her students for enjoyment during the first year of implementation, but in her second year she made a point to include other books: “That is a piece that I made a big point of this year with moving to full day [of kindergarten] is really trying to get some extra books in there and at least once a day we’re reading something that’s not a Reading Street book [laughs]... It is, like you said, love that passion that just pure enjoyment, reading a story for just the enjoyment of a story. And part of what makes that happen is the person reading it has to really buy into and like the story and has to sell. So, some of the books I can do that with Reading Street, some of them I cannot." Teachers also voiced concerns that Reading Street focused on one aspect of reading, phonics, at the expense of other important areas of reading. Edith, a first grade teacher explained, "what I have found a little bit though is even though my kids can read anything you put in front to of them; their comprehension level is still not there." Sybil, a fourth grade teacher, identified the influence grade plays in this issue of developing well-rounded readers: "Yes, we’re still going to identify those kids that needed help with phonics and fluency, yada, yada, yada, but we’re going to start teaching them how to be, I don’t even know how I even want to put it, how to be not just worrying about the fluency and how they sounds as readers, but what they understand and how they understand as readers, I guess. So, we got a program that’s Pearson still, but it’s six -eighth [grade] and it’s more of a literature-based program. And I think four/ five [grades] kind of falls in that middle and maybe that’s the discomfort that 76   we’re feeling, is that, it’s still, we’re not teaching phonics so to speak, but we are working on fluency, which I think is fine, but it doesn’t have enough meat to it." Daisy, a kindergarten teacher, summarized this issue through the concept of building lifelong readers. "Right, or building life long readers. That’s one of my big worries is that if we’re not exposing them to looking at and going through the experience of reading a book." Systems Factors Influencing Beliefs and Perceptions of Reading Street In addition to personal beliefs about pedagogy and the nature and goals of reading, system characteristics influenced teachers' perceptions of Reading Street. The overarching systemic issue identified was that teachers felt a lack of professional autonomy due to the hierarchical decision making structure of their schools. Many of the teachers described how Reading Street was selected by administrators without input from teachers. Daisy, a kindergarten teacher explained, "we had a reading workshop approach to literacy instruction and we did not even know we were in the market for a new program. It dropped in our lap when we came back and our heads were spinning, so that naturally created a lot of anxiety." Benefits and Challenges of Reading Street. Finally, teachers' spoke about how their perceptions of the challenges and benefits of Reading Street was related to their views on the program. The challenges of the program overlap with the implementation barriers especially in regards to scheduling and time issues. Teachers identified challenges that included the characteristics of their students, grouping students for small group activities, the quality of the Reading Street materials, and accessing materials. Some teachers found Reading Street to be easy to prepare for, whereas others thought the preparation required for Reading Street was “excessive”. The teachers identified the following features to be strengths of Reading Street: the clear 77   scope and sequence, vocabulary, oral language, promotion of teacher collaboration, consistency across classes, student progress, and positive student responsiveness. Teachers found the clear scope and sequence to be very helpful in understanding when and how to deliver important components of learning to read. The vocabulary and oral language portions were found to be a benefit as these parts of the language arts are often skipped over. Teachers found having consistency across classrooms beneficial as it made teacher collaboration easier and more productive. Several of the teachers noted that they perceived student progress as greater since using Reading Street. Finally, some teachers noted that students responded positively and engaged in Reading Street. Research Question 3: What is the teachers’ level of motivation to use Reading Street? This question was examined with both qualitative and quantitative data. The Teacher Motivation Survey yielded expectancy, value, and cost scores, as well as a composite motivation score. These items on the survey used a five-point Likert scale that ranged from completely disagree (1) to and completely agree (5). As can be seen in Table 5, teachers rated their expectancy the highest, followed by cost. They rated their perceived value of the intervention the lowest. A one-way analysis of variance (ANOVA) demonstrated that the means of the three constructs were significantly different (F=(2,82), 7.792 p<.001). Paired t-tests indicated that expectancy significantly differed from value (t=7.09, p<.001) and from cost (t=2.71, p<.01). Value and cost were not significantly different from each other indicating that teachers’ expectations for the success of Reading Street was higher than their perceptions of how much they valued the program and how costly they perceived the program. Table 5 Descriptive Statistics for Motivational Beliefs N=42 Expectancy Value Cost 78 Motivation   Table 5 Cont’d M (SD) 3.57 (.42) M (SD) 2.97 (.78) M (SD) 3.15 (.64) M (SD) 69.78 (24.31) Range 2.25 3.31 2.86 98.80 In the focus groups questions were posed that addressed expectancy and value including: • At the beginning of the year did you expect Reading Street to increase your students’ reading performance? • Has this expectation changed over the course of the year? • What are the benefits of using Reading Street? • What reading program would you use if you had a choice? The themes that arose from these questions included influences on expectations and the benefits and challenges of Reading Street. Influences on Expectations. Teachers described several critical sources that influenced their expectations for Reading Street. First, they were swayed by the personal testimonies of other teachers. Sybil, a fourth grade teacher stated, "I already had heard some not so great feedback from some peers at the elementary school." Teachers were also influenced by their peers whom they may have never met. Edith, a first grade teacher, discussed her knowledge of Reading Street from the Internet: "I think I did have a little bit more of a open mind I guess going into it just cause I was like oh, I read blogs and in the blog world, you hear about the different curriculum and I had heard about Reading Street, so I was kind of excited. And I kinda expected a lot from it." Finally, Violet, a second grade teacher, discussed her experience with her training program and the process of becoming board certified as influencing her expectations for Reading 79   Street: "I didn’t expect it to make a great deal of difference either way because, you know, in the course of taking, you know getting my masters, my understanding from all the research that’s been done around reading for a long, long time is pretty much close to 80% of the kids are going to be successful regardless of whether you use a basal reading program, use guided reading, use whole language that they’ll be successful with it. They will learn how to read, then you’re going to have that 20% that have a range and you’re going to have to say okay, not quite getting it, I got to narrow in and target more and more what it is that I need to do to help this child become a reader.” Benefits and Challenges of Reading Street. Teachers’ shared their views of the program, both positive and negative, which related to their value for the program. These discussions tended to reflect how the program did or did not help them to reach their goal of teaching children to read. Edith, a first grade teacher, discussed how her views on Reading Street became more positive over time. "It was hard in the beginning, like number one I was new to the district and new to Reading Street. Reading Street was new to my district as a whole this year…In the beginning I would try to like do as much as I could and I hit the targets that I needed to, at least and then you know, I got a little bit better as time went on." Mary, a kindergarten teacher, stated she found the program to be valuable because of what it has done for her students. "I would say that I like Reading Street. I have seen it do amazing things for my kids. I will say that’s its not that different than what I used to do with Balanced Literacy, it’s just more organized." Other teachers took a more neutral stance on 80   Reading Street. Cora, a kindergarten teacher, described the overwhelming nature of the program as influencing her neutral view. “I like Reading Street and I say that like half-heartedly…it’s very overwhelming and then you dig into it and it’s a little less overwhelming and then you dig into a little more and its a little less." Overall, these quantitative data suggested that teachers had higher expectations for Reading Street than other programs but valued it less. Teachers expected the program to work despite reporting lower value for it. Qualitative data demonstrated that teachers’ expectations and values for the program spanned the continuum. Some teachers had higher expectations and values while others had low expectations and values. Other teachers reported feeling neutral toward the program and not having a predetermined expectation for it. Research Question 4: What is the teachers’ understanding of treatment integrity? Do they view treatment integrity as unidimensional or multidimensional? Given the tendency for treatment integrity to seem evaluative to the implementers of interventions, this topic was approached broadly during the focus groups to elicit honest thoughts and reflections on the implementation of Reading Street with treatment integrity. It was also thought that approaching treatment integrity more broadly and less personally would more naturally allow teachers’ perspectives on the topic to emerge without biasing their responses with the preconceived framework of this study. The questions that were designed to evoke conversation about treatment integrity included: • When planning your Reading Street lessons how do you use the manual in your plans. • Sometimes, adaptations are necessary. Have you found times when you need to adapt Reading Street for your students? What adaptations have you made and why? • Are there other instructional materials that you use and find helpful to teaching reading? 81   • Do you feel you have been successful in implementing Reading Street? How do you know? Treatment integrity was conceptualized in terms of several variables including adherence, participant responsiveness, and exposure. As is seen in Table 6, teachers discussed several aspects of treatment integrity. Of the 51 comments made about treatment integrity, 29 or 57% represented a unidimensional perspective, known as adherence in Dane and Schneider’s model, and 34 or 43% made reference to other aspects of a multidimensional view. Table 6 Number of Comments Involving Aspects of Treatment Integrity Number of Comments Involving Aspects of Treatment Integrity Adherence 29 Quality of Delivery 1 Exposure 4 Participant Responsiveness 17 Program Differentiation 0 Total Comments 51 The Reading Street manual describes how the program is intended to be used but teachers had a range of views on the purpose of the manual and how strictly it had to be followed. At one end of the spectrum Edith described how the manual provided the basis for the PowerPoint presentations she used in her classroom: "I print [the PowerPoints] because they say everything. I even begin with our learning goals for the week, so it says everything. It's pretty much to the “T” from the teachers’ manual because we were told to teach with fidelity.” Although some teachers endorsed the view that treatment integrity needed to include verbatim instruction from the manual, other teachers used the manual with more flexibility. Cora stated, "I kind of look through and pick out these are going to be my most important pieces, these are going to be my target things and these if I get to it, I can get to it. 82   If not, we’ll fit it in or where we’re at for there. That’s generally how I plan for it." Statements such as these indicated that some teachers viewed the manual more as a guide that supported their lesson planning without dictating it. Using the manual as a guide to support implementation rather than rigidly adhering to it, speaks to the issue of adaptation. Teachers discussed the adaptations that they made to Reading Street and discussed a variety of reasons for making these planned changes to the program. The most common reason for adaptations involved better meeting students' needs. For example, Isobel stated, "I’ve incorporated dancing, we dance to the songs, I use letter tiles where they come up and manipulate the tiles and make words for me instead of me writing them up on the board and those kinds of things." Isobel is a kindergarten teacher and found incorporating movement was a way to better engage her students. Anna adjusts the way in which she uses the leveled readers to better meet her students needs. "In the classroom trying to manipulate the fidelity system, while still struggling with the fact that my strategic intervention kids and my advanced kids are supposed to read the exact same text more than one day a week. That’s a hard thing for me and so, I have found success with not being able to pull out of the book room pulling previous on level books. So, for say we’re on unit four, pulling an unit two on level book for my strategic intervention kids, so that they’re still working on the same skills, still seeing the same high frequency words that we’ve introduced more than once, but not as, it’s just more on level for them. So, fitting that into program while still adapting." Similarly, Matthew adjusted how independent reading is done in his classroom so that students are able to enjoy reading novels. 83   "And that’s where we don’t really, I don’t really use the leveled readers. I have them available, students will go over and select them sometimes, but they know that some of our independent reading time is time when they can pick up a novel and have that experience of great pleasure of being able to work your way through it. It’s not only a feeling of maybe awe and mystique about something, but it’s also a feeling of accomplishment when you finish a novel and it’s like wow and I’m doing this." In addition to meeting student needs. Edith referenced adapting the program through using PowerPoints. "Well, doing the power point one is a huge adaptation. And you know I’m banking on that next year will be, well it wont be a breeze cause I don’t have the whole school year but that if I get a lot done this summer, the school year will be better cause they’ll already be done." Finally, Elsie’s school adapted the writing portion of Reading Street based on the teachers' values of certain writing skills. "We’ve adapted the witting. I think that we’ve found kinda the happy medium. We went through and picked the skills that we thought were the most valuable, like this week they’re supposed to write a limerick, I don’t see that being very valuable. So, we went thought and we picked the things that we thought were the most valuable. And instead of trying to jam them into one week, we spread them over two and we found that’s a reasonable amount of time. So, we’ve adapted the entire writing part. We’ve only done it for a short time this year, we’ll do it the whole year 84   next year and see how it goes, but it’s a big adaptation." In some cases, teachers use other instructional methods and programs in addition to Reading Street. Violet discussed using the Daily Five program to supplement Reading Street's centers, which she found did not meet her students' needs. "We were, many of us were Daily Fiving and we were told we couldn’t use that as an instructional model or a center model, you must do the literacy centers that came with the program, that’s how you implement it. And after several weeks of, you know, blunt honest to goodness you’ve got to be kidding me? This is busy work at its epitome. No, I’m doing the three components of the Daily Five that I know." Similarly, Sarah adjusted the small reading group portion through using a program she had previous training in, Reader's Workshop. Through using this program, she feels she is providing them the opportunity to read at their independent reading level. "My small reading groups have been hit or miss, just depending on time and different things that are going on. My biggest adaptation is my kids aren’t doing literacy centers or we’re not using word stations. They do readers workshop. So, that’s how I tie into, I sneak in my they’re reading their just right independent chapter book at that time. So, I’ll use that time to do fluency screenings, you know intermittently, and then meet with kids sometimes or have them whisper read to me or different things like that. We’ll do different, I have some strategy journal cards, so sometimes we’ll do those on a day where they’re using their book and applying their strategies. So, that’s an adaptation." 85   Research Question 5: How are motivation, treatment integrity, and student learning related? How is teacher motivation related to treatment integrity? There were four measured aspects of treatment integrity. First, the average number of completed items on the Teacher Activity Logs was used to operationalize adherence. Due to differences in the items deemed non-negotiable there were differences in the total number of items for each grade level. Therefore, percentages were used to assess level of adherence. Second, participant responsiveness was measured with an item on the Teacher Activity Log that asked teachers to rate students’ participation and engagement in Reading Street activities on a scale of 1-5?. Finally, exposure was measured in terms of number of days per week teachers’ reported implementing Reading Street and how many minutes per day teachers’ reported implementing the program. The means and standard deviations for these variables are displayed in Table 7. Table 7 Descriptive Statistics for Aspects of Treatment Integrity N=42 Adherence Participant M % (SD) Responsiveness M (SD) Table 7 Cont’d .61 (.14) Range Number of Days Per Week Implemented M (SD) 2.27 (.53) 4.90 (.30) Number of Minutes per Day Implemented 1 M (SD) 5.73 (.50) .67 3.00 1.00 2.00 1 Data were collected categorically with the categories of 1= less than 15 minutes, 2= 15-30 minutes, 3=30-45 minutes, 4= 45-60 minutes, 5= 60-90 minutes, and 6 90+minutes. Each of the treatment integrity variables was correlated with the expectancy, value, and the composite motivation variables. As can be seen in Table 8, expectancy significantly correlated with exposure dimension of treatment integrity (r=0.37, p<0.05) as measured by the number of days per week Reading Street was implemented. Value and exposure as measured by 86   number of days per week had a significant correlation of 0.53. Value and exposure as measured by number of minutes per day Reading Street was implemented implemented had a significant correlation of 0.41. Cost and exposure as measured by the numbers of days per week implemented had a significant negative correlation of -0.48. These results suggest that part of the hypothesis regarding teacher motivation and treatment integrity was supported: the dimension of exposure was significantly related to the motivational variables of expectancy and value. Table 8 Correlations between Motivation, Treatment Integrity, and Student Achievement Expectancy Value Cost Adherence 0.37 0.33* -0.19 Participant Responsiveness 0.06 -0.07 -0.10 Number of Days Per Week Implemented 0.37* 0.53** -0.48** Number of Minutes Per Day Implemented 0.21 0.41** -0.24 Percent At-Grade Level 0.04 0.17 0.02 * Significant at the 0.05 level; ** Significant at the 0.01 level Motivation 0.25 -0.06 0.47** 0.12 0.15 In regards to the other dimensions of treatment integrity, adherence was significantly related to value. There was a 0.33 correlation between these two variables. This supports the part of the hypothesis that predicted that teachers’ who valued the intervention more would implement it with higher adherence. How is treatment integrity related to students’ reading achievement? Pearson’s correlations between the treatment integrity and students’ reading achievement are reported in Table 9. There were no significant correlations between any of these dimensions and student achievement. This result failed to support the hypothesis that higher treatment integrity would be correlated with higher student achievement. Table 9 Correlation Coefficients Between Treatment Integrity Variables and Student Outcomes Adherence Number of Days Per Number of Minutes Per Participant Week Implemented Day Implemented Responsiveness At-Grade Level -0.01 -0.13 -0.03 -0.10 87   How is teacher motivation related to students’ reading achievement? Students’ reading achievement was quantified as the average number of students who were achieving at-grade level at the end of the school year. There were no significant correlations between any of the motivation variables and student outcomes. These results suggested the hypothesis that teacher motivation would be positively related to student reading achievement is rejected. Correlation coefficients between teacher motivation and students’ reading achievement are displayed in Table 8. 88   Chapter 5 Discussion This study sought to address the issue of teachers’ motivation to implement an evidencebased reading curriculum with treatment integrity. The research questions were examined with both quantitative and qualitative data. This mixed methods approach was beneficial to the study as the qualitative data elucidated the quantitative data by providing thick descriptions of the variables of interest. Three major aspects of evidence-based interventions were examined. First, teachers’ perceptions and beliefs about their experiences with evidence-based interventions were assessed using a survey and focus groups. Second, teacher motivation for using evidence-based interventions and Reading Street, in particular, was measured with a survey and focus group interviews. Third, implementation with treatment integrity was assessed with teacher activity logs and focus group interviews. Teacher and System Factors Influencing Perceptions and Beliefs Across the focus groups and throughout the discussions within each, teachers discussed factors that influenced their perceptions and beliefs on the four main topics addressed in this study: evidence-based intervention, Reading Street, expectancy and value theory of motivation, and treatment integrity. The participants also discussed how system factors influenced their perceptions and beliefs on these same areas. Evidence-based intervention. In this study, there was a general consensus among the teachers that evidence-based interventions were interventions with research demonstrating their effectiveness. Cora, a kindergarten teacher stated, "I would say evidence-based interventions are something that has research behind it that shows us its effectiveness and that is measureable that you could document the growth that you see happening." Other teachers relied on commercially 89   available products to guide their perception of EBIs. Edith, a first grade teacher said, “it’s like something, it usually has a label to it, like here’s either a strategy or like a particular, what do I want to say? Like a program that you use.” Other teachers relied on their school administration to define EBIs. Anna, a kindergarten teacher explained how EBIs are selected in her school: "[EBIs] comes down from our literacy specialist or our administration that someone has done the study, someone has done the research and said these are effective interventions for children, and they’ve been sanctioned in that way with studies and research behind them, is my understanding." Even though some teachers were willing to accept the word of research passed down to them from administrators, other teachers questioned the validity of research and asked, “what does evidence-based really mean?” Martha, a fourth grade teacher, asked: " I’m not sure about this, but whose evidence are they looking at? Like I might have evidence that a volunteer in the hall listening to a child read could improve their fluency, but if that hasn’t been studied by somebody else who says yes that’s an acceptable intervention, then that’s not a research-based intervention or evidence-based intervention that I can use and say that’s my intervention. You know and it almost feels like it has to be an intervention that some company has created." These quotations explored the broad questions of what does research-based mean? What counts as data? Teachers indicated that they relied on additional types of data than just what’s reported in empirical research studies. Importantly, it appeared teachers relied on personally meaningful data such as their own personal experiences as well as the opinions of fellow teachers (e.g., blogs, personal reviews). Violet, a second grade teacher, brought up the limitations 90   of empirical studies and discussed how research may not be transportable across all populations. " I want more than one person’s study. I want several people’s studies to say it is, does it transfer across areas? And I spent three years teaching on a reservation, what would be evidence-based in the middle of South Dakota might not be evidence-based in New York city, so that’s to me, that I think that there’s, there are lot of programs out there that people say they’re research based or they’re evidence-based and it’s a very select group of students that yes, it has shown to make progress with them. So, like I said there was a few that seemed to have made the cut for several different organization that have done their own independent research." In addition to discussions about the definition of evidence-based interventions, teachers frequently discussed the feasibility of EBIs, especially in terms of implementation barriers. Focus group discussions aligned with prior research suggesting that teachers look for interventions that are feasible, appropriate for their students, and are accompanied by the appropriate materials and professional development (Boardman, 2005). In the present study, teachers also mentioned these three features. Appropriateness for their students was based on perceptions about individual differences of their students including socioeconomic status, the developmental appropriateness, and the instructional level. Mary, a kindergarten teacher, stated: " I was skeptical [about Reading Street]. I mean it’s hard to overcome the population that we have at my school. We’re in a rural community with an extremely high free and reduced lunch rate, you know, which is just an indicator of the at-risk, the huge at riskiness that we are in my building." 91   Edith, a first grade teacher, discussed her views on the appropriateness of the EBI, Peer Assisted Learning Strategies (PALS) for her students: “But, then I also do the additional first grade PALS. And then because I was having a difficult time with like some of the kids too, I found that if I took my lowest six, which they are the ones that usually have a little bit of a harder time working independently anyway, I did teacher directed PALS with them, in a small group, even though that’s supposed to be like two maybe three kids, but it worked for me to do with the six and the rest of the class could kind of do the independent with a partner first grade PALS.” Materials were also discussed in the focus groups, especially in regards to Reading Street. Edith discussed a concern with a lack of access to materials. “And then another big challenge for me is that, a lot of times, I feel like the parts are missing, like it’ll tell you what to teach, but then like what you need to teach it isn’t always necessarily there.” In addition to access to materials, teachers more frequently discussed their displeasure with the quality of Reading Street materials. Teachers expressed dissatisfaction with both the product quality of materials and editorial or print errors within the published materials. Across the focus groups there were 16 mentions of errors within the program, with which teachers expressed frustration. Martha, a fourth grade teacher explained: “Some [videos] are better than others and we laugh about, you know there’s some that they clearly they paid the money to hire some professionals with real instruments or you know. And then there’s the one with the little drum the money to do that one, you know?” Violet, a second grade teacher, discussed the quality of the printed Reading Street 92   materials: "And I don’t, can’t speak for a third grade book or a kindergarten book, but my second grade books are rife, if that’s the right word, with error, full of typos and errors. I mean I sit there, we have a read aloud component that’s a read aloud anthology that you’re supposed to use to help teach a high frequency word or vocabulary word and I’ll sit there with my pencil at the ready because it will have, ‘when she was doing something he bat.’ You know it’s like they used he instead of the. They’ve used ‘there’ wrong, ‘weight’ wrong. They’re missing periods. I mean they’re missing words and sentences and not just in the read aloud in our decodable readers." Finally, professional development was also of concern to these teachers. Teachers discussed the professional development opportunities available for Reading Street and with other evidence-based interventions. Edith, a first grade teacher discussed the opportunities she has utilized: “There’s been workshops that I can go to put on by the ISD. So, like I’ve gone to a couple and just within my own district, they’ve had like, not meetings, but like Monday nights get together and talk about what’s working for you, what are you doing and how are you doing it? Whatever, just to like bounce ideas off of each other and we did it in the fall.” Isobel, a kindergarten teacher from a different school, described how she has had accessed professional development: “I had an RtI coach come down and observe me as I was teaching cause I wanted to see if I was doing it the way I should be doing it and timing it right and she 93   gave me a lot of feedback.” Daisy, a kindergarten teacher, explained her views on the importance of professional development to implementation: “Well even with you saying that about Six Minute Solutions, we’ve implemented that in our third and fourth grade classrooms as well and, you know, it is a good evidence-based intervention and if you’re trained and things like that it can be really impactful if you find a routine for it. You know, we have seen some success for it. I found too that when we’re asking the classroom teachers to add those in, if they’re not getting trained, then I’m not really certain that they’re really being implemented in the way that they’re supposed to be.” Martha, a fourth grade teacher, echoed similar sentiments to Daisy: “It’s very hard to do any of those programs without the right training too. The rest of my team is doing PALS at some point and said here’s the manual you can figure it out I kind of got all lost and said, no you know what? I’m not going to do this if they won’t train me.” Daisy also compared recent professional development opportunities to those she has had in the past: “When I went to training for Sidewalks, we went for one half day training to learn about three programs that are yearlong programs. You know and being a Reading Recovery trained teacher, I spent an entire year in very intensive training to be fully trained and then monthly training sessions after that I was just like baffled, you can train me in three hours on something and think I was just like okay, three hours, three programs that are year long and I’m just going to be able to go 94   change the world.” Teacher perceptions of these three areas, materials, student characteristics, and professional development, provide a possible explanation as to why teachers may have lower motivation to implement an intervention with treatment integrity. When teachers perceive materials as being of low quality or inaccessible, perceive the intervention as not being appropriate for students’ needs, or do not feel as though they have been adequately trained, it seems logical that their expectancies and values are lowered and they would be less motivated to implement the program. Reading Street. Two teacher factors were closely related and together influenced perceptions of Reading Street. These factors included grade taught and teachers’ pedagogical philosophy on reading. Grade taught appeared to be a factor in teachers’ perceptions of Reading Street because teachers at different grade levels brought up differential issues surrounding the program. These grade level differences related to the pedagogical philosophy on the purpose of reading at different grade levels. Sybil, fourth grade teacher, recognized that her view on Reading Street may differ from that of a lower elementary grade teacher: "I think its taken, and I’m fourth grade, so kindergarten and first grade perspective versus upper [elementary], when my kids are sitting there saying, can we just read the story already? I feel like its kind of zaps some of that love of, you know, quality literature and things like that. It’s kind of sucked us dry of that in terms of that... Now they feel like yeah they can read, yeah I can read 115 words correct in a minute, but I just don’t have, they don’t have that same passion.” Ethel, a second grade teacher, agreed with the idea that pedagogically Reading Street does not align with many teachers’ views on the purpose of reading. 95   "So there are things I like, like you say [teacher name], some of the structure is useful as far as it’s already planned for me, but I miss the fact that I feel like I could do a lot more creatively in some areas and still, get kids excited about reading more than this does. This is a lot of watch the screen, do the worksheet, watch the screen, do the worksheet." Two kindergarten teachers both discussed how Reading Street met the needs of the students at their grade level and how that influenced their thoughts toward the program. Anna recognized the benefit of Reading Street’s emphasis on phonics. “I have seen it be very effective in teaching them those beginning phonics skills and I’ve seen the reading and writing off the charts from what it used to be." Isobel discussed how the scope and sequence Reading Street helped her to be more effective with her students. “I have absolutely loved [Reading Street]. This is the first year that I feel my students have made the growth that I knew that they were capable of. It has been a solid foundation and not me piecemealing things together with it being all laid out. I know that I’m not missing anything that they need and to have my lowest children be where my middle kids would usually end up says a lot to me.” Given that Reading Street is heavily focused on phonics, which is often considered a beginning reader skill, it follows that lower elementary teachers would have more positive feelings toward it than the upper elementary teachers. In upper elementary grades, the teachers described the importance of having students start to develop a love of reading through reading whole pieces of literature rather than sections and segments. Martha, a fourth grade teacher, explained: "I find the literature in the book, in the textbook, the stories in fourth grade, I like 96   them. I look forward, oh boy I get to read this part of Winn Dixie again, and I seek out those books and show kids, you can read the whole book, here it is, you know. And there are things that I never would have read on my own. The book, story about the dog and Lewis and Clark, I never would choose a book like that and when I read it and I think this is good, I like this, so it’s good in that respect. And they’re getting a nice mix of fiction and non-fiction in fourth grade. But, then you get to the leveled readers, the leveled readers are so stupid because they have these vocabulary words and they’re hiring writers to write a little dippy book using these words about this topic and they’re just dumb." In addition to these factors, teachers also discussed their emotional responses to the program. Most frequently teachers discussed feeling overwhelmed by Reading Street. These feelings of being overwhelmed may have influenced their beliefs about the programs, which theoretically related to their expectancy for success and their value of the program, which will be discussed further in the next section. Mary, a kindergarten teacher, discussed how she liked (valued) Reading Street even though the program was overwhelming at first. "I like, as an overall description of my experience with Reading Street, it’s very overwhelming and then you dig into it and it’s a little less overwhelming and then you dig into a little more and its a little less." Daisy, a kindergarten teacher, shared similar thoughts on the lessening of anxiety over time: “After you get over the overwhelming kind of components of understanding the structure of the day because when you first read it you’re like ahh, you know trying to figure out how you’re going to make it flow so it doesn’t sound like chop, chop." Expectancy and Value. Generally, teachers agreed that they expected Reading Street to 97   be successful. Although some teachers were initially skeptical, they still endorsed the idea that it would work. Mary, a kindergarten teacher stated, "I guess I expected that it would because the research said that it would. I was skeptical. I mean it’s hard to overcome the population that we have at my school.” The Isobel, also a kindergarten teacher, discussed her initial skepticism and her surprise with the results of Reading Street. " I didn’t expect to have the growth that I received. I knew that it would bring in the pieces that were missing from Book Room teaching of reading because that misses a lot of your phonics and phonemic awareness and that kind of thing. But, I wasn’t prepared for the leaps and bounds that my kids grew. I mean I knew that it was a good program and I was excited, but I also was looking at it going holy cow, these kids have to sit forever." Edith, a first grade teacher, discussed how compared to her colleagues she was excited for the program due to information she had read on the Internet: " So, I think I did have a little bit more of a open mind [than other teachers] I guess going into it just because I was like oh, I read blogs and in the blog world, you hear about the different curriculum and I had heard about Reading Street, so I was kind of excited. And I kind of expected a lot from it." Daisy, a kindergarten teacher, discussed how she attempted to overcome negative feelings but felt as though the climate of her school also influenced her expectations and values for the program: "I had a negative feeling about it going in, but I really tried over the summer to be like okay, I’m going to take this home this is what we’re doing. I want to come back positive. You know, I have to give it a chance before I can totally bash 98   something. I have not taught this, I would not expect somebody who has not taught it to be judgmental about it. I know these things about it whatever. So, really worked hard to come back with a positive attitude about it, but the problem was in our building in general was a very negative feeling about it." In terms of value, there is a substantial amount of overlap between this variable and the teacher factor of pedagogical philosophy and view on the purpose of readings. Teachers’ who value phonics and phonemic awareness placed a higher value on this intervention. Teachers, who view reading more holistically, did not as highly value Reading Street. Daisy explained: "You know in terms of [Kindergarten], the phonics and phonemic awareness components are really strong, there’s a good scope and sequence to it. I have found my kids this year to really have, they are confident to try words, attack words, you know they are problem-solving words visually that I very rarely would see kindergartners do. But, because we are so heavy on the decoding, I’m really worried about the meaning piece of it" Matthew described difficulties he experiences with the program with fourth grade: "Another problem with the way that it was sold I think was that it was sold as having a lot of literature of, excuse me, “award-winning literature”. Now, very first one in fourth grade I think is, (Martha: Winn Dixie), Winn Dixie. It won lots of awards, but Winn Dixie in its entirety won the award, won the awards. You cannot watch, you know, a three-minute excerpt of an Oscar winning movie on the Tonight Show with Jay Leno and say that you have now seen and experienced that movie. So, it’s really I think misleading to say that these are award winning pieces because you take one chapter out of Winn Dixie and that’s not won the award.” 99   In addition to Reading Street not aligning with his views on reading, Matthew’s value for the program also seemed to decrease due to the way the program was presented. In the following quote he discussed how Reading Street was packaged and presented as an anthology but feels as though he was really getting a basal reader. “I think at some point [basal] become a dirty word because when we first were introduced to Reading Street, the what I perceive to be the basal, was presented to us as an anthology, which is a really heady word. But, it’s not an anthology. It has instructional components to it and I actually went to the library and looked up a lot of anthologies… when people were coming in from Pearson and telling us all about this, one of the representatives kept using the term anthology and I thought wow that sounds really impressive, but it’s very misleading because it is a basal…And so this is a basal and no body wants to admit that…maybe there’s a stigma, but well why is that? I guess it’s even less respectful to the teachers and to the students to be calling it something it isn’t then to just say, you know, maybe basals weren’t so bad and we can implement some of what they had.” Similar to this issue of the view on the purpose of reading influencing value for Reading Street, views on the purpose of assessment also influenced the value of Reading Street. Some teachers viewed Reading Street as teaching to a specific task that does not accurately represent reading ability. Violet, a second grade teacher, expressed her views on the purpose of assessment and reading: "And so each time we sit for out data and somebody goes, yep, now they’re reading seventy-five minutes word per minute. Mhmm and are they reading with expression, are they reading with meaning and are they actually pausing or are 100   they reading to beat the timer? " When there is a lack of alignment between teachers’ definition of reading and their views on how reading should be measured, this presents a major obstacle to motivation and integrity. As discussed above, teachers’ pedagogical philosophy about teaching reading underlies their value of Reading Street. According to the expectancy-value model, when value is compromised this will undermine overall motivation. Treatment Integrity. The most commonly discussed factor related to the teachers’ views on treatment integrity was pedagogical philosophy. Teachers often cited their pedagogical philosophy as a reason for not implementing Reading Street with treatment integrity. Mary, a kindergarten teacher, cited her belief in the importance of play as a reason for why she modifies the small group component of Reading Street. "We don’t use the independent chart stations. We use our own centers. When I’m doing reading groups, my other kids are either playing or at intervention. But, that’s okay with me because I feel like my kids need to play." Violet, a second grade teacher, discussed her belief that giving her students books that are at their instructional level in order to engage them is an ethical responsibility as a teacher. "And so I have very quietly in my classroom not had fidelity. I have said okay, I cannot in all conscience as an educator take this child and put this book down in front of them and say this is what you’re allowed to read and this is only what you’re allowed to read, whether or not its above or below [your level] and so I get through those little readers as quickly as I can so I can say yep, covered the readers and then I go looking for the books that I know they’re going to be engaged with." Quotations such as these indicate that teachers are primarily concerned with meeting their 101   students’ needs over meeting the more impersonal system demands such as implementing a program with treatment integrity. This finding is representative of the research to practice gap. In research settings, there have been studies that indicate the importance of treatment integrity on outcomes. However, these findings have little meaning or application in authentic settings if practitioners are unaware of such work or if it contradicts what they perceive as necessary for their students to achieve. Teachers appear to view administration concerns over treatment integrity as a decontextualized demand versus their personalized concern for their specific students’ achievement. Teacher Motivation to Implement Reading Street through Expectancy-Value Theory Teacher motivation for implementing Reading Street was examined through survey items that measured participants’ expectancies and values for the intervention as well as the perceived costs of implementation. The results of the survey revealed that teachers expectations were significantly higher than their perceptions intervention’s value and cost. According to the expectancy-value model, value and cost should have different ratings, as they are theoretically opposed to each other. The more a person values a task or in the present study, an intervention, the perceived cost should be lessened. According to the multiplicative nature of expectancyvalue theory, this framework requires that people must both expect to be successful and value a task to engage in it. The present data partially supports this theory. When looking at motivation to implement the intervention in terms of the number of days per week teachers implemented the program, positive correlations were found between this variable and both expectancy and value. As predicted, this variable and cost were inversely negative. Teachers who perceived the intervention as more costly to implement reported implementing the program less often. 102   In contrast, when looking at the motivation in terms of minutes per day Reading Street was implemented, the relationships were different. The only significant relationship found between this indicator of motivation and the components of expectancy-value theory was with value. Those teachers who reported valuing Reading Street more highly reported implementing it more minutes per day. Focus group findings provided some possible explanations for why these patterns exist. One of the commonly mentioned themes throughout all of the focus groups was teachers’ sense of a lack of autonomy and a sense of a hierarchy. Teachers did not have a choice on whether or not to implement Reading Street. During the focus groups, teachers discussed their experiences with how Reading Street was presented to their schools. “We had a Reading Workshop approach to literacy instruction and we did not even know we were in the market for a new program. It dropped in our lap when we came back and our heads were spinning.” Two teachers in one focus group shared the following exchanging Ethel: “We got it like the day before school stated, first teacher meeting, here it is. And you’re going to the sales pitch meeting and then just teach it.” Matthew: “And it wasn’t just here it is, it was [slams book down] here it is.” Ethel: Fill up your room [with Reading Street materials] These quotations reflect the perception among teachers that they did not have a choice about which reading curriculum they would use. Therefore, it may not have mattered whether or not teachers expected the program to be successful or whether they valued the program; they were expected by administration to implement it just the same. Although the framework of this study was based in expectancy-value, some of the themes that emerged from the focus groups reflected self-determination theory (Deci & Ryan, 2000). Self-determination theory suggests that motivation is based on individuals’ perceptions of 103   competence, relatedness, and autonomy. Across all of the focus groups the theme of lack of autonomy was evident, and this appeared to have affected their motivation. Violet, a second grade teacher, described her original open mindedness to Reading Street and then losing that vision when being told how to implement it: “So, I went through the summer and I read through the student text and I read through the manuals and I thought okay, there are a lot of good pieces in here, I like the idea that your kinda on the same topic for a week. That seems to be good and then when we were being told this is non negotiable, this is non negotiable, you must do this, you must do this, you must not do this, you can’t do this, you can’t do that and I thought okay this is not how I envisioned working with children to get them to be really strong readers and learners.” Similarly, teachers took the mandate to use Reading Street as a slight to their professional credibility, which relates to the idea of competence. Some teachers seemed to interpret the requirement to use a scripted intervention as devaluing their professionalism. Teachers seemed to be expressing frustration that both the administration and Reading Street eroded their freedom to make professional decisions about their instruction. They seemed to have the perception that at both the level of the system and the EBI, they were required to follow a rigid set of procedures without professional choice. Mary, a kindergarten teacher, related Reading Street to her experiences with another manualized curriculum: “I remember when we first got Lucy Calkins, which is a scripted program, you know you could take in your own direction, but there are certain things you’re supposed to say in a certain way. I remember at first we kind of railed against that. The state of Michigan said that I get to make some educational decisions here and 104   this does give me very many decisions. And then it was like Reading Street was that times a billion where it was like you really just say what it says and like I said, read the blue and think the black. And its like, you know, uh anybody could do that. You don’t really need to be a teacher to be able to do that, you know.” Given the systemic factor that teachers did not have a choice in whether or not they could implement Reading Street may have influenced the findings. Future studies exploring teacher motivation to implement evidence-based intervention should focus on the way choice in program impacts implementation. Perceptions of Treatment Integrity (or the “F” Word) Sarah, a third grade teacher, discussed the shift in her school’s perception of treatment integrity, “last year we talked about fidelity, was the dirty little ‘f’ word. (…) But now I feel like in our building the f word is flexibility and fluidity because we can pull in more." Evidence from the focus groups suggests that the perception of treatment integrity varied based on the district in which the teachers worked. Some teachers, like Sarah, discussed how the perception of treatment integrity changed over time. As the teachers implemented the program, administration shifted away from demanding rigid adherence to the program's components to a more flexible model. On the other hand another Mary, a kindergarten teacher, discussed how the district has moved in a reverse pattern. "There used to be more deviating from what Reading Street said in our district and then we had to have big meetings about fidelity, which everybody loves the word fidelity.(….) If its written in Reading Street, other than the writing, you’re saying it and you’re doing it." 105   Teachers discussed treatment integrity, or in their words treatment fidelity, throughout the focus groups. Some questions were designed to elicit responses that would relate to treatment integrity; however, many teachers brought up this topic unprompted. Focus group discussions were coded looking for evidence of teachers endorsing a traditional, adherence focused view of treatment integrity as well as for indications of other aspects of treatment integrity. Interestingly, the aspect of adherence was discussed on a continuum that spanned from vague references to very specific references to the manual. Daisy a kindergarten teacher, broadly discussed the importance of implementing a program fully: "Part of me agrees that it needed to be implemented totally in order for us to see what it’s going to do because if somebody says they’re doing it, then that’s not really showing us what this program can do. And the people that are implementing it fully, and that’s where I’m struggling with that piece is that there are people that aren’t and we were supposed to and you know.” Edith, a first grade teacher, discussed how she very specifically adheres to the manual: “Like I said, now, my plans are my powerpoints. I print those out because they pretty much say everything. I even begin with our learning goals for the week, so it says everything, but it’s for me its’ pretty much to the T from the teachers’ manual because we were told to teach with fidelity." Mary, a kindergarten teacher, described her school’s adherence focused view on treatment integrity as follows: “Fidelity, fidelity, got to do it the right way. So, we found that some teachers were only doing the target things and they were skipping the other things. Some 106   of, one of my co-workers was not doing the conventions piece at all cause she felt she wasn’t going to have time for it. So, there’s been a very big push in our district to do everything with fidelity.” Despite there being evidence that at the system level there was a great concern for rigid adherence to Reading Street, teachers discussed other aspects of treatment integrity throughout the focus group. Participant responsiveness in particular was mentioned as being important when implementing Reading Street and other interventions. Teachers did not a necessarily make a direct connection between their references to participant responsiveness and the relationship to treatment integrity. However, it is clear teachers viewed this concept as important to ensuring their students were learning and benefiting from instruction. Isobel, a kindergarten teacher, described how she does not implement Kindergarten- Peer Assisted Learning Strategies (KPALS) as intended in order to ensure her students are engaged: “I also use K-PALS in my kindergarten classroom, which is a research based and I can’t say I do it with fidelity because with five and six year olds, teaching them to work in partnerships has not worked for me. So, I do it whole group, but they’re very engaged in, I mean they’re still getting it even though I’m not doing it the way they designed it." Most of the aspects of Dane and Schneider's model were naturally brought up in the focus groups; however, two aspects were not as evident in the conversation. Program differentiation was not mentioned at all, and quality of delivery was only mentioned once. Isobel, a kindergarten teacher, recognized the importance of implementing Reading Street in a way that was exciting and engaging to the students: 107   "I mean I knew that it was a good program and I was excited, but I also was looking at it going holy cow, these kids have to sit forever. But, since I teach kindergarten, they didn’t know any different. These kids just came in and because I presented it in an exciting way and I was upbeat about it and I was excited about it, I passed that onto these children that now they’re like I love Reading Street, yay.” Overall, the data collected on treatment integrity suggested that teachers’ perception of it was strongly influenced by their schools and focused on adherence, or in their words, “fidelity.” Teachers also mentioned the importance of participant responsiveness and the quality of delivery when implementing interventions. Despite adherence being the most frequently referenced aspect of treatment integrity, teachers also referenced the value of quality teaching and accommodating their students. This finding that teachers valued meeting their students needs above other aspects of implementation suggest that client characteristics matter to teachers. Client characteristics are one of the fundamental dimensions of evidence-based practice in psychology (APA, 2006), and in this study we see that teachers view student characteristics as vitally important to their profession as well. The teachers in this study balanced their use of evidence-based interventions and Reading Street with their knowledge of their students. Characteristics that were discussed as relevant to consider included grade level, developmental appropriateness, socioeconomic status, and students’ instructional level. Teachers considered these individual differences when using their professional expertise on how to best deliver instruction. Relationship between Teacher Motivation and Treatment Integrity 108   Overall, the data collected in this study did not support the hypotheses that greater levels of motivation would relate to higher levels of treatment integrity. As previously discussed, the one exception to this finding is that the treatment integrity aspect of exposure that did correlate positively with expectancy and value and negatively correlated with cost. Generally, the other measured aspects of treatment integrity, adherence and participant responsiveness did not correlate with teacher motivation. There was a weak correlation found between adherence and value. Suggesting there is a small relationship between how much value teachers placed on the intervention and how closely they adhered to Reading Street’s components. This finding is consistent with Tanol (2010), which also found a weak correlation between treatment integrity and treatment acceptability, a construct that overlaps with variable of value. In the present study this weak relationship between value and adherence may reflect the fact that they did not have a choice in whether or not they used Reading Street. Relationship between Teacher Motivation and Student Outcomes Sarah, a third grade teacher, summarized her motivation to use Reading Street in the following statement, “so, yes I like the basis of Reading Street, but I guess the thing is I don’t teach Reading Street, I teach children and I’m going to do what I need to do to help them learn.” Quotations like this one help explain why quantitative data from this study do not support the hypothesis that increased teacher motivation would be related to increased student outcomes. From the focus group discussions it appears that teachers are not motivated to implement with treatment integrity for its own sake but the focus of their implementation is on meeting the needs of their students. The present study was measuring motivation for teachers to implement Reading Street specifically. As previously discussed, teachers were required to use Reading Street in their classrooms whether or not they expected the intervention to be successful or whether they valued 109   the intervention. Even though some teachers might not have been motivated to use Reading Street, focus group discussions indicated they were still motivated to help their students to achieve. Perhaps, the specific intervention is not necessarily the most important explanatory factor for student outcomes; rather it is the motivation of teachers more broadly to have successful students. This finding that teachers were motivated for their students to achieve more than they were motivated to implement a specific intervention relates to Finnegan and Gross (2007)’s study on the way accountability policies influenced teacher motivation. This study found that teachers’ value for their professional status and for their students’ outcomes increased motivation. Low morale in schools decreased teachers’ motivation. Although the present study did not directly measure teachers’ value for their professional status, their values for student outcomes, or their morale, these issues did surface in the focus groups. Sarah, a third grade teacher, discussed the power of seeing her students make progress. "I think that the powerful thing is then when you can bring the data to your classroom, like when I can see my own evidence, you know when I put my kids on the quadrant to see if you need fluency help or if they need comprehension help. And you know, kids who needed fluency help, they made as much progress in six weeks doing Six Minute Solution as they did in the whole first trimester without it. So, I mean once you can see the evidence in your own classroom, I think it becomes a more powerful tool for you too." Focus group discussions suggested that many teachers were not necessarily motivated specifically to use Reading Street but remained motivated to have students achieve. Therefore, they worked within the context of their schools, which required the use of Reading Street to 110   support students. As Elsie, a second grade teacher explained, “Reading Street doesn’t make a bad teacher good all of a sudden and it doesn’t make a bad teacher good because they have to use it.” Teachers seemed to view Reading Street as a tool they were required to use to promote student growth, which was the overall goal they were trying to achieve. Relationship between Treatment Integrity and Student Outcomes There was insufficient evidence to support the hypothesis that increased treatment integrity would be related to increased student outcomes. This finding contributes to the body of literature suggesting a mixed relationship between treatment integrity and outcomes. There have been many studies that have also failed to connect these two constructs (Lee, Penfield, & Maerten-Rivera, 2009; Perepletchikova & Kazdin, 2005; Sterling-Turner, Watson & Moore, 2003; Wickstrom, Jones, LaFleur & Witt, 1998). In the present study the relationship between outcomes and treatment integrity may not have been significant as curriculum-based measures, which are global measures, may not have been sensitive enough to subtle differences in implementation. Focus group discussions revealed that teachers were focused on meeting the needs of their students to promote favorable outcomes more than on rigidly adhering to the manual of a curriculum. Violet, a second grade teacher explained: “So, I think in a lot of ways, it’s your teaching, it’s the good teaching. I mean I don’t, I haven’t thrown this program to the side, but I look at it and go okay here’s the text that I’m using, here’s the skill, the concept that I’m trying to get across. How am I going to go about doing that with these seven and eight year olds in front of me are going to be able to grasp? What do I need to do as a teacher, how do I have to my classroom set up?” 111   Other teachers echo this sentiment of good teaching and using a program flexibly as being more important than implementation with treatment integrity. Sarah, a third grade teacher stated, “you know, fidelity means that I need to cover what the book says that I need to cover, but it doesn’t mean that I’m a monkey and I’m going to say the same thing as the person who is teaching next to me.” This teacher’s quote captures one reason why it may be difficult to show a relationship between treatment integrity and outcomes. To examine treatment integrity, researchers break interventions down into observable, measurable components. However, there is more to implementation than the observable, measurable components of an intervention can measure. As Martha, a fourth grade teacher stated, “there’s so many variables and they can’t reduce teaching to a script. It’s an art.” To explain the relationship between the implementation of interventions and student outcomes many more variables need to be considered than simply adherence to a checklist or manual. In this study teachers discussed the idea of quality or delivery as being important to implementation. Teachers mentioned times when they felt it was important to stray from the checklist in order to improve the quality of Reading Street for the specific students with whom they were working. This finding was especially evident with upper elementary teachers who were concerned about not only teaching students basic reading skills, but also providing them with the opportunity to experience literature and develop a love for reading. Conclusions Thjs study provided a look at how a specific evidence-based intervention is used in schools. Despite yielding mixed results that confirmed some, but not all of the hypotheses, this research provided an in-depth look at teachers’ motivation and views on using EBIs. Evidencebased practice has moved from the medical field to psychology and now it permeates education. 112   However, evidence-based practice in education settings and in psychological settings appears to function slightly differently. The APA Task Force (2006) identified evidence-based practice in psychology as having three important components: evidence-based interventions, clinical experience, and client characteristics., This study, , suggested however, that school administrations and teachers tended to emphasize different features of evidence-based practice. Administrators focused on the evidence-based intervention,by requiring the use of specific programs with fidelity, whereas, teachers tended to be concerned with the characteristics of their students and their fit with the internveiton. Teachers reported relying on their experiences and taking the characteristics of their students into consideration when implementing interventions. For evidence-based practice to be successful in the school setting all aspects of need to be recognized and valued. Teachers identified a way to do this would be by adapting Reading Street and using it flexibly. Starting with an intervention that has research behind it is important, however, teachers can use their professional expertise to deliver the intervention in a flexible way that meets the needs of their specific students. Allowing teachers to utilize their professional judgment may ensure that their motivation to implement EBIs is not undermined. Requiring strict adherence to a mandated curriculum may threaten teacher autonomy and convey a message that their skills are not valued and that they are not capable of improving student achievement. Future research needs to address how to balance the delivery of interventions with integrity and the use of clinical judgment that also characterizes EBP. School psychologists should consider teacher motivation and the unintended messages that requiring the implementation of EBIs with treatment integrity defined narrowly, can send to teachers. As the evidence-based practice movement grows in education it is important to not only continue to develop improved measures for documenting treatment integrity, but it is equally 113   important to find a way for these measures to become a valued part of the school context and to allow teachers to use their professional expertise to implement EBIs with flexibility in order to meet the needs of their students. Evidence-based interventions are just one component of the larger evidence-based practice movement and school psychologists should not lose sight of the importance of professional expertise and client characteristics. Finding ways to integrate these components of EBP will be critical to supporting the implementation of EBI in schools. When asking teachers to implement EBIs, school psychologists and administrators should listen carefully and honor the teachers voices and opinions. Giving teachers more input intp the selection process of EBIs and input into the process of how treatment integrity data will be collected will show that the school values their expertise. Teachers will also be able to discuss potential implementation barriers that can be addressed prior to EBIs being launched. Involving teachers in this process rather than mandating the use of particular EBIs will support teacher motivation. In addition to addressing teacher attitudes toward EBI, this study raised the even more fundamental question of what is an “evidence-based intervention?” In authentic, school settings EBI are often talked about as specific prepackaged, commercialized programs. However, when looking at the research underlying programs such as Reading Street the evidence is not as strong as publishing companies would like for it to be. The practice of selecting programs based on their packaging is widespread and the supposed ability to be used “right off the shelf” is alluring; however, it could potentially undermine the evidence-based practice movement. By using programs that seem like they could be evidence-based but in reality do not have a strong research base, sends a message to teachers that the EBP movement is another passing fad in education. School psychologists have a responsibility to help their schools make informed decisions about 114   which evidence-based interventions to use and to help their schools incorporate the other components of evidence-based practice by finding ways to include teachers’ expertise and knowledge about students in the process. Limitations There are several limitations inherent in this study including limitations with the conceptual framework, the sample, and the measures. First, in terms of the conceptual framework, the study did not directly assess all five of the aspects of treatment integrity put forth by Dane and Schneider (1998). As discussed in the introduction and literature review, a multidimensional conceptualization is important as the area of treatment integrity continues to develop. Even though constraints of the research setting, did not allow for the measurement of these areas, considering treatment integrity multidimensionally was still a part of the study through the qualitative analysis of teachers’ responses on treatment integrity. In regards to the sampling procedures and data collection there were several limitations. First, the study relied on teachers to volunteer to participate in the survey and focus groups. The researcher was unable to use a random sampling procedure. Without using random sampling the generalizability of the study was lowered. However, given the qualitative analyses that were used, generalizability was not the main goal of this study. Rather the study sought to explore and describe a phenomenon and gave teachers a voice to express their thoughts and views about an important topic in education. Relying on volunteers also may have led to a response bias with some teachers volunteering more readily than other teachers. Teachers that participated may have been strongly opinionated about evidence-based interventions and Reading Street. However, through the focus groups it appeared that there were teachers with positive (“I have absolutely loved it”), negative (“Then I stand alone and hate it. I do. I hate it.”), and neutral feelings toward 115   it (“So, after two complete years, I‘m feeling like I can live with it.”) Relying on volunteers also led to a limited sample size for both the survey and the focus groups. The small sample size for the survey limited the types of quantitative analyses that could be run. The small focus group sizes limited the breadth of thoughts and views in the discussion. There were also not enough focus group volunteers to create groups based on motivation and then compare discussions based on level of motivation. There were some limitations with the measures used in this study. Three of the measures were self-report, which could lead to the social desirability effect. Teachers may have been more likely to answer both the teacher survey measure and the Teacher Activity Log in ways they thought were socially desirable. In addition to the limitations of self-report, constraints of the setting did not allow for independent observations to occur to help offset the potential for report bias. The second limitation of the measures was that they most of them were created specifically for this study. The Teacher Activity Log, the teacher survey, and the focus group questions were all developed for the study. To overcome these limitations the Teacher Activity Log was developed as a collaborative effort and greatly relied on published material from Reading Street. The items on the teacher survey that pertained to motivation were developed based on previous surveys that used an expectancy-value framework to understand the implementation of educational programs. For each factor of the expectancy-value framework (i.e. expectancy, value, and cost), the items that were selected for the present study were adapted from the survey that had achieved the highest Cronbach alpha for that factor. Similarly, the focus group questions were developed for this study. The questions were based on previous research and theory, but they have unknown psychometric properties. There were several limitations to using curriculum-based measures to determine student 116   achievement. First, these assessments are broad and designed to be used primarily for screening purposes. As such, they might not have been sensitive enough to relationships between the variables. Second, using aggregated data instead of individual student data did not account for differences between the classrooms at the beginning of the year. Using analyses that controlled for students’ initial achievement levels would have strengthened this study. Finally, the statistical design limited the conclusions that can be drawn from this research. Given that correlations were calculated, this study cannot speak to causal relationships when considering the relationships between teacher motivation, level of treatment integrity, and student outcomes. These are issues that have been addressed in previous research literature and should continue to be investigated in future research. This study has the potential inform future research based on the relationships that were found and based on the qualitative findings that helped to explain these relationships. 117   APPENDICES 118   Appendix A 1.2 Teacher Motivation Survey: Items Item # Component 1 E 2 E 3 C 4 V 5 E 6 V 7 E 8 E 9 E 10 V 11 E 12 V 13 E 14 E 15 E 16 C 17 V 18 E 19 E 20 V Item If I used cooperative learning Reading Street, the students tend to veer off task. * I understand cooperative learning (Reading Street) well enough to implement it. The costs involved in implementing cooperative learning Reading Street are great. Technology use (Reading Street) increases academic achievement (e.g., grades). The amount of cooperative learning Reading Street training I have received has prepared me to implement it successfully. Technology use does not result in students neglecting important traditional learning resources (e.g., library books). Reading Street does not reduce students’ use of traditional literacy resources (e.g., library books). My students presently lack the skills necessary for effective cooperative group work effectively engaging in Reading Street. * For me to succeed in using cooperative learning with Reading Street requires support from my colleagues. * Using cooperative learning Reading Street is likely to create too many discipline problems. * Technology use Reading Street promotes student collaboration. For me to succeed in using cooperative learning with Reading Street requires support from school administration. * Technology use Reading Street is a valuable instructional tool. My training in cooperative learning Reading Street has not been practical enough for me to implement it successfully. * Cooperative learning Reading Street is appropriate for the grade level I teach. If I use cooperative learning Reading Street, too many students expect other group members to do the work. * It is impossible to implement cooperative learning Reading Street without specialized materials. Technology use Reading Street makes teachers feel more competent as educators. I believe I can implement cooperative learning Reading Street successfully. I have too little teaching experience to implement cooperative learning Reading Street successfully. * Technology use Reading Street gives teachers the opportunity to be learning facilitators instead of information providers. 119   21 C 22 E 23 V 24 E 25 V 26 C 27 E 28 V 29 C 30 V 31 C 32 V 33 E 34 E 35 V 36 E 37 E 38 C 39 V 40 E There is too little time available to prepare students to work effectively in groups with Reading Street. There are too many students in my class to implement cooperative learning Reading Street effectively. * Technology use Reading Street is an effective tool for students of all abilities. My students are resistant to working in cooperative groups with Reading Street. * Technology use Reading Street is necessary because students will not learn computer reading skills on their own, outside of school. Implementing cooperative learning Reading Street requires a great deal of effort. Cooperative learning Reading Street is inappropriate for the subject grade level I teach. Technology use Reading Street enhances my professional development. Cooperative learning Reading Street is an efficient classroom strategy. * Technology use Reading Street helps accommodate students’ personal learning styles. Implementing cooperative learning Reading Street takes too much class time. Reading Street motivates students to get more involved in learning activities. I find that cooperative learning Reading Street is too difficult to implement. * Cooperative learning would not work with my students. * Reading Street does not work with my students. Technology use Reading Street promotes the development of students’ interpersonal skills (e.g. the ability to relate or work with others). If I use cooperative learning Reading Street my classroom is too noisy. * I believe I am a very effective teacher. Implementing cooperative learning Reading Street takes too much preparation time. Technology use Reading Street improves student learning of critical concepts and ideas. The physical set-up of my classroom is an obstacle to using cooperative learning Reading Street. * 1 Expectancy and cost items adapted from Abrami et al. (2004); value items adapted from Wozney (2006) 2 Response scale: 1= strongly disagree to 5= strongly agree 120   Additional Survey Items Reading Street Activity Log Code: Number of years you have implemented Reading Street: 1 2 3 4 5+ Number of years teaching: Highest level of education: Bachelor’s Degree Some graduate courses Master’s Degree Number of days per week I implement Reading Street: 1 2 3 4 Other:_________ 5 On average the number of minutes per day I implement Reading Street: Less than 15 15-30 30-45 60-75 90 or more Please describe some of your experiences with evidence-based interventions in your school. Evidence-based interventions are specific interventions, programs, or instructional strategies that have been studied with research and found to have a positive effect on students’ outcomes. Are you motivated to use evidence-based interventions? Why or why not? How do you know if an evidence-based intervention is successful in your classroom? 121   Appendix B Questioning Route Hello, thank you for coming to this focus group on your experiences with evidence-based intervention and Reading Street. In today’s group I will ask a series of question focusing on your beliefs, perspectives on evidence-based intervention and Reading Street. When we are talking about evidence-based interventions, I am referring to specific strategies, instructional methods, and interventions that have research support for their effectiveness. It is my hope that we will have a casual conversation; don’t feel like you must answer each question, but please give as much input as you want. I’m anticipating today’s session will take about 75-90 minutes. I will also be asking about your implementation of Reading Street. Today’s session will be audio taped; however, only myself and other researchers will listen to the contents. No one from your schools or the ISD will have access to these tapes, nor will I discuss what is said here today except in the write up of the study. Privacy and confidentiality is of utmost importance to me, and in this write up, names or other identifiers will not be used. As a courtesy to the other participants in the group, I ask that you do not discuss what people say today with others. The goal is to make this a safe place to have a productive conversation about your experiences with evidence-based intervention and Reading Street. Does anyone have any questions before we begin? Opening: Tell me your name, how long you’ve been teaching, and how long you’ve been teaching reading. * Introductory: How does the school you teach in incorporate evidence-based interventions into daily practices? Transition: How do you define evidence-based interventions? (RQ1) Key Questions: Do you incorporate evidence-based intervention into your teaching? If so, which ones and how? (RQ1) How would you describe your overall experience using Reading Street? (RQ2) At the beginning of the year did you expect Reading Street to increase your students’ reading performance? Why or why not? (RQ3a) 122   Has this expectation changed over the course of the year? Why or why not? (RQ3a) What are the benefits of using Reading Street? What are the challenges of using it Reading Street? (RQ3b) When planning your Reading Street lessons how do you use the manual in your plans? (RQ4) Sometimes, adaptations are necessary. Have you found times when you need to adapt Reading Street for your students? What adaptations have you made and why? (RQ4) Are there other instructional materials that you use and find helpful to teaching reading? (RQ4) Do you feel you have been successful in implementing Reading Street? How do you know? (RQ4) What reading instruction/programs have you used in the past aside from Reading Street? What reading program would you use if you had a choice? (RQ2) Ending Questions: We have come to the end of focus group. Is there anything I missed or anything you would like to add? Is there anything that you came wanting to say and didn’t get a chance to say? * Do you have any questions for me? *Adapted from Krueger and Casey (2009) 123   Appendix C Reading Street Teacher Activity Logs Reading Street Weekly Activities Kindergarten Directions: Please fill out this survey by marking the days the following Reading Street activities were completed. Please remember that this form is NOT evaluative and is designed to collect data about the implementation of the Reading Street Curriculum. Thoughtful and honest responses will be beneficial to the success of Reading Street in your school! GET READY TO READ Mon. Tue. Wed. Thur. Fri. N/A Concept Talk Questions of the Week Anchored Talk (oral vocabulary) Phonemic Awareness Phonics Handwriting High Frequency Words Phonemic Awareness Review Day Letter Recognition Review Day Blending Concept Wrap Up READING COMPREHENSION Listening Comprehension Comprehension Big Book or Anthology • First Read • Retell • Think, Talk and Write nd rd Big Book or Anthology- 2 and 3 Read Comprehension Target Skill Mon. Tue. Wed. Thur. Fri. N/A LANGUAGE ARTS Conventions Writing Mini-Lessons Mon. Tue. Wed. Thur. Fri. N/A PROGRESS MONITORING Word Reading (Day 1) Word Reading (Day 2) High-Frequency Words/ Re-Tell (Day 3) Fluency (Day 4) Oral Vocabulary (Day 5) Mon. Tue. Wed. Thur. Fri. N/A SMALL GROUP ACTIVITIES Completed Yellow Page Activities/Small Group Time Mon. Tue. Wed. Thur. Fri. N/A 124   Other students are actively engaged in independent activities or practice stations (rate 1-5 with 5 being the highest) Reading Street Weekly Activities Grade 1 Directions: Please fill out this survey by marking the days the following Reading Street activities were completed. Please remember that this form is NOT evaluative and is designed to collect data about the implementation of the Reading Street Curriculum. Thoughtful and honest responses will be beneficial to the success of Reading Street in your school! ‘ GET READY TO READ Concept Talk Questions of the Week Anchored Talk (oral vocabulary) Phonemic Awareness Phonics Spelling Oral Vocabulary- Amazing Words Concept Wrap Up Mon. Tue. Wed. Thur. Fri. N/A READING COMPREHENSION High-Frequency Words Listening Comprehension Comprehension Story Words Vocabulary Build Background st nd Main Selection- 1 and 2 Read Paired Selection Mon. Tue. Wed. Thur. Fri. N/A LANGUAGE ARTS Conventions Writing Mini-Lessons Mon. Tue. Wed. Thur. Fri. N/A PROGRESS MONITORING Word Reading (Day 1) Word Reading (Day 2) High-Frequency Words/ Re-Tell (Day 3) Fluency (Day 4) Oral Vocabulary (Day 5) Mon. Tue. Wed. Thur. Fri. N/A SMALL GROUP ACTIVITIES Completed Yellow Page Activities/Small Group Time Other students are actively engaged in Mon. Tue. Wed. Thur. Fri. N/A 125   independent activities or practice stations (rate 1-5 with 5 being the highest) Reading Street Weekly Activities Grade 2 Directions: Please fill out this survey by marking the days the following Reading Street activities were completed. Please remember that this form is NOT evaluative and is designed to collect data about the implementation of the Reading Street Curriculum. Thoughtful and honest responses will be beneficial to the success of Reading Street in your school! ‘ GET READY TO READ Concept Talk Questions of the Week Anchored Talk (oral vocabulary) Phonemic Awareness Phonics Spelling Oral Vocabulary- Amazing Words Concept Wrap Up Mon. Tue. Wed. Thur. Fri. N/A READING COMPREHENSION High-Frequency Words Listening Comprehension Comprehension Story Words Vocabulary Build Background st nd Main Selection- 1 and 2 Read Paired Selection Mon. Tue. Wed. Thur. Fri. N/A LANGUAGE ARTS Conventions Writing Mini-Lessons Mon. Tue. Wed. Thur. Fri. N/A PROGRESS MONITORING Word Reading (Day 1) High-Frequency Words (Day 2) Re-Telling (Day 3) Fluency (Day 4) Oral Vocabulary (Day 5) Mon. Tue. Wed. Thur. Fri. N/A SMALL GROUP ACTIVITIES Completed Yellow Page Activities/Small Group Time Other students are actively engaged in independent activities or practice stations Mon. Tue. Wed. Thur. Fri. N/A 126   (rate 1-5 with 5 being the highest) Reading Street Weekly Activities Grade 3 Directions: Please fill out this survey by marking the days the following Reading Street activities were completed. Please remember that this form is NOT evaluative and is designed to collect data about the implementation of the Reading Street Curriculum. Thoughtful and honest responses will be beneficial to the success of Reading Street in your school! ‘ GET READY TO READ Concept Talk Questions of the Week Anchored Talk (oral vocabulary) Word Analysis Oral Vocabulary- Amazing Words Comprehension Check (Day 3) Concept Wrap Up Mon. Tue. Wed. Thur. Fri. N/A READING COMPREHENSION Comprehension Skill-Strategy Lesson Vocabulary Vocabulary Skill-Strategy Main Selection: Option 1 & Option 2 • Build background • Pre-reading strategies • Retelling • Think critically Paired Selection Mon. Tue. Wed. Thur. Fri. N/A LANGUAGE ARTS Conventions Writing Mini-Lessons Spelling Mon. Tue. Wed. Thur. Fri. N/A PROGRESS MONITORING Oral Vocabulary (Day 1) Word Reading (Day 2) Re-Telling (Day 3) Fluency (Day 4) Oral Vocabulary (Day 5) Mon. Tue. Wed. Thur. Fri. N/A SMALL GROUP ACTIVITIES Completed Yellow Page Activities/Small Group Time Other students are actively engaged in independent activities or practice stations Mon. Tue. Wed. Thur. Fri. N/A 127   (rate 1-5 with 5 being the highest) Reading Street Weekly Activities Grade 4-6 Directions: Please fill out this survey by marking the days the following Reading Street activities were completed. Please remember that this form is NOT evaluative and is designed to collect data about the implementation of the Reading Street Curriculum. Thoughtful and honest responses will be beneficial to the success of Reading Street in your school! ‘ GET READY TO READ Concept Talk Questions of the Week Anchored Talk Oral Vocabulary- Amazing Words Comprehension Check (Day 3) Genre (Day 4) Concept Wrap Up • Amazing ideas • Review of target skills Mon. Tue. Wed. Thur. Fri. N/A READING COMPREHENSION Comprehension Skill & Strategy Lesson Lesson Vocabulary Vocabulary Skill & Strategy Lesson Main Selection: Option 1 & Option 2 • Retelling • Think critically Paired Selection Mon. Tue. Wed. Thur. Fri. N/A LANGUAGE ARTS Conventions Writing Mini-Lessons Spelling Mon. Tue. Wed. Thur. Fri. N/A PROGRESS MONITORING Oral Vocabulary (Day 1) Re-Telling (Day 2) Re-Telling (Day 3) Fluency (Day 4) Oral Vocabulary (Day 5) Mon. Tue. Wed. Thur. Fri. N/A SMALL GROUP ACTIVITIES Completed Yellow Page Activities/Small Group Time Other students are actively engaged in independent activities or practice stations Mon. Tue. Wed. Thur. Fri. N/A 128   Appendix D Aimsweb and DIBELS Cut-Scores for Determining At-Grade Level Aimsweb Cut off Scores for Reading at Grade Level Grades K: Phoneme Segmentation Fluency Grade Fall Fall SD Fall Winter Winter Winter Spring Spring Spring Mean Cut Mean SD Cut Mean SD Cut Score Score Score K ---29 19 10 46 20 26 DIBELS Next Cut off Scores for Being at Grade Level Grades K:Phoneme Segmentation Fluency Grade Fall Fall SD Fall Winter Winter Winter Spring Spring Spring Mean Cut Mean SD Cut Mean SD Cut Score Score Score K ---35 19 16 49 16 33 Aimsweb Cut off Scores for Being at Grade Level Grades 1-6:Oral Reading Fluency Grade Fall Fall Fall Cut Spring Spring Bench Standard off Score Benchmark Standard Mark Deviation Mean Deviation Mean 1 1 1 11 71 40 47 36 2 64 37 27 106 38 3 89 40 49 125 42 4 109 39 70 140 42 5 122 40 82 152 42 6 140 39 101 166 41 1 Spring Cut off Score 31 68 83 98 90 125 Winter benchmark mean and standard deviation as this is the first time Aimsweb assesses ORF st for 1 grade. DIBELS Next Cut off Scores for Being at Grade Level Grades 1-6: Oral Reading Fluency Grade Fall Fall Fall Cut Spring Spring Spring Bench Standard off Score Benchmark Standard Cut off Mark Deviation Mean Deviation Score Mean 1 1 1 7 60 33 27 37 30 2 3 4 5 6 62 33 29 95 38 57 80 36 44 107 38 69 93 38 55 123 38 85 112 38 74 132 39 93 130 33 97 143 37 106 1 Winter benchmark mean and standard deviation as this is the first time the district measured st DORF for 1 grade. 129   Appendix E Recruitment Letters Dear (insert name of superintendent), My name is Amanda Hall, and I am a fourth year doctoral student in Michigan State’s school psychology program. I am working on a dissertation research project that examines teachers’ experiences with implementing evidence-based interventions. I am interested in using the data the Ingham County Intermediate School District has been collecting with the Reading Street Teacher Activities Log as well as collecting some additional information on the teachers’ experiences with Reading Street. The additional data I would like to collect includes a teacher survey and teacher focus groups. The teacher survey would ask teachers questions about their views on the importance and success of Reading Street and their views on evidence-based intervention. The focus groups would go into more depth on these same issues. The curriculumbased measures would be used to examine the relation between teachers’ views and student outcomes. The over goal of this study is to understand the implementation of evidence-based intervention from the perspective of teachers. With your permission, I would like to contact the principals and teachers at the schools in your district to invite them to participate in this study. Teachers that participate in the study will receive a gift card in appreciation for their time. I am also requesting permission to access the curriculum-based measures that are collected in the schools. These data will be accessed from the ISD and student data will be de-identified before it is given to me. Similarly, teachers will not be identified by name, but rather the teacher codes that are used for the Reading Street Teacher Activity Logs will be used to identify teachers. I would be happy to answer any questions about the study and provide more details about the study. As part of Michigan State’s Institutional Review Board (IRB) process, letters of support need to be submitted from the potential sites where data will be collected. Once this study has be reviewed by the IRB, you will be presented with documentation that it has been approved. By signing this document, you are indicating that you support this study being conducted in your district. Thank you for your time and consideration and I look forward to hearing from you. Sincerely, Amanda Hall, M.A. Hallama3@msu.edu (717) 903-9140 I,____________________________ support research to be conducted in my district. ______________________________ Signature ________________________ Date 130   Dear (insert name of principal), My name is Amanda Hall and I am a fourth year doctoral student in Michigan State University’s school psychology program. I am working on a dissertation research project that examines teachers’ experiences with implementing evidence-based interventions, specifically, Reading Street, which has been adopted in your school district. I would like to collect data that includes a teacher survey and teacher perspectives through focus groups. The teacher survey would ask teachers questions about their views on the importance and success of Reading Street and their views on evidence-based intervention. The focus groups would go in more depth on these same issues. The over goal of this study is to understand the implementation of evidence-based intervention from the perspective of teachers. I have received a letter of support from the curriculum director to conduct this study in your school. I am asking if you would be willing to support this research as well and allow me to recruit participants and provide access to data. To carry out this study, I am would need support in three areas. First, I would like to ask for 10 minutes at a staff meeting to introduce the study to the teachers. Second, I would ask you to forward an email containing information about the study to the teachers in your school. Third, I would ask for access to a conference room or other meeting space after school on two days to hold the focus groups. I appreciate your consideration of this request and I would be happy to answer any questions. I look forward to hearing from you. Sincerely, Amanda Hall, M.A. Hallama3@msu.edu (717) 903-9140 I,____________________________ support research to be conducted in my district. ______________________________ Signature ________________________ Date 131   Name of Study: Teacher Motivation and Perspectives on Implementing a Tier-1 Reading Intervention with Treatment Integrity Institution: Michigan State University, East Lansing, Michigan Principal Investigator: Dr. Evelyn R. Oka, Associate Professor, School Psychology & Educational Psychology Research Coordinator: Amanda Hall, M.A. hallama3@msu.edu, (717) 903-9140 Hello! Thank you for taking the time to read about my study. This study is a part of my dissertation project for my doctoral program in school psychology at Michigan State University. The goal of my study is to understand teachers’ experiences with and motivation for implementing the evidence-based intervention, Reading Street. To accomplish this goal, I believe it is important to invite teachers to share their views about these topics. The study will use three pieces of data. First, there will be a teacher survey. Any teacher who is implementing Reading Street is invited to take this 20 minute online survey. Teachers who complete the survey will receive a $15.00 online gift card to Amazon.com. Secondly, at the conclusion of the survey you will have the opportunity to indicate willingness to participate in a focus group. These focus groups will involve questions about your experiences with Reading Street. The focus groups will take place after school in your building. The groups will last approximately 7590 minutes and light refreshments will be served. Teachers who participate in the focus groups will receive a $30.00 gift card from a selection of stores, such as Target, Kroger, Starbucks or Barnes and Noble. Finally, as part of this study, I am interested in looking at classroom level DIBELS/Aimsweb data. All of the data that are collected will be kept confidential with data stored in locked and secure locations. No names will be used in the presentation of the data. School administrators at your school or at the ISD will not be informed of who did or did not participate in the study. If you are interested in participating in this study, please watch your e-mail over the next few days. I will be sending out an email with a link to the online survey. This study was approved by the Michigan State University Institutional Review Board. If you have any questions about the study that you feel the researchers cannot answer, please contact them at Michigan State University's Human Research Protection Program at 517-355-2180, Fax 517-432-4503, or e-mail irb@msu.edu or regular mail at 207 Olds Hall, MSU, East Lansing, MI 48824 Thank you in advance for your help. If you have any questions about this study, please contact me at (717)903-9140 or Hallama3@msu.edu . Sincerely, 132   Amanda Hall Hallama3@msu.edu (717)903-9140 133   Appendix F Consent Forms CONSENT FOR PARTICIPATION IN RESEARCH Purpose of Research: The purpose of this research is to better understand the implementation of evidence-based interventions (EBIs) in school settings. There are many studies that have identified interventions that have empirical support; however, there are fewer studies that have examined issues of the implementation of EBIs in everyday practice. Understanding issues of implementation are important because of differences between studying an EBI in highly controlled research settings and using one in everyday practice. This study seeks input from teachers on EBIs to better understand what their experiences in using an EBI in an authentic setting. Understanding teachers’ experiences with and motivation for using EBIs is part of the process of learning how to better support teachers in their use of EBIs in their classrooms. Your Role: As a part of this study, you will be asked to complete an online survey. The survey consists of 40 items about your views on of Reading Street. There are three additional shortanswer questions about research-based practices in general and your experience specifically with Reading Street. In addition, the study will collect existing reading achievement data from students’ reading curriculum-based measures. These data will be accessed by the researcher from the Intermediate School District and will not create additional work for you. Potential Benefits and Risks: While there is no direct benefit to you, the opportunity to reflect on your experiences with Reading Street may benefit teachers and students by increasing our understanding of research-based practices. The main risk is that this may elicit thoughts and feelings about your practice, some of which may be negative or critical. Privacy and Confidentiality: Privacy and confidentiality are of utmost importance. No administrators at the school or ISD will be informed of your participation in the study. Names will not be collected for the survey, and therefore, responses will be anonymous. For those consenting to participate in the study, the researcher will be given classroom level with curriculum based measure data without teachers’ names. In order to conduct one of the analyses for the study, it is necessary to be able to connect teachers’ survey responses to teachers’ student data. This connection will be made through using the code system for the Reading Street Teacher Activity Log. The classroom data will be given to the researcher with this code rather than identifying information such as name or email address. After the completion of the study all raw data will be destroyed. Your Right to Participate or Withdraw: Participation in this study is completely voluntary. Furthermore, you have the right to withdraw from the study at any time. Costs and Compensation for Participating: There are no financial costs associated with participating in this study. It is anticipated that the survey will take approximately 30 minutes to 134   complete. As a token of the researchers’ appreciation for your you will receive a $15.00 online gift certificate for Amazon.com. Contact Information for Questions and Concerns: If you have concerns or questions about this study, please contact the researchers: Amanda Hall, M.A. by phone (717)903-9140 or email: hallama3@msu.edu or Evelyn Oka, Ph.D., by phone: 517-432-9615; email: evoka@msu.edu; 439 Erikson Hall, East Lansing, MI, 48824. (http://www.educ.msu.edu/cepse/SchoolPsychology/faculty.asp) If you have any questions or concerns about your rights as a research participant or want to offer a complaint, please contact the Michigan State University's Human Research Protection Program at 517-355-2180, Fax 517-432-4503, or e-mail irb@msu.edu or regular mail at 207 Olds Hall, MSU, East Lansing, MI 48824. DOCUMENTATION OF INFORMED CONSENT  Yes, I _______________________________consent to participate in this research study.  No, I __________________________________do not participate in this research study. ________________________________________ Signature 135 ____________________________ Date   CONSENT FOR PARTICIPATION IN RESEARCH Purpose of Research: The purpose of this research is to better understand the implementation of evidence-based interventions (EBIs) in school settings. There are many studies that have identified interventions that have empirical support; however, there are fewer studies that have examined issues of the implementation of EBIs in everyday practice. Understanding issues of implementation are important because of differences between studying an EBI in highly controlled research settings and using one in everyday practice. This study seeks input from teachers on EBIs to better understand what their experiences in using an EBI in an authentic setting. Understanding teachers’ experiences with and motivation for using EBIs is part of the process of learning how to better support teachers in their use of EBIs in their classrooms. Your Role: As a part of this study, you will be asked to participate in a focus group of three to five other teachers that are also implementing Reading Street. The focus group will center around your understanding of research-based practices, your experiences with research-based practices, and your experiences with Reading Street. The focus group will be guided by a set of predetermined questions and participants will be asked to have a discussion around each of the questions. Potential Benefits and Risks: While there is no direct benefit to you, the opportunity to reflect on your experiences with Reading Street may benefit teachers and students by increasing our understanding of research-based practices. The main risk is that participation may elicit thoughts and feelings about your practice, some of which may be negative or critical. Privacy and Confidentiality: Privacy and confidentiality are of utmost importance. No administrators at the school or ISD will be informed of your participation in the study. The focus groups will be audio recorded for analyses; however, only the researchers will access these recordings. After the completion of the study all raw data will be destroyed. Although every precaution will be taken to respect privacy and confidentiality, in a focus group confidentiality can only be guaranteed at the group level. Group members will take on responsibility to keep each other’s information confidential. Your Right to Participate or Withdraw: Participation in this study is completely voluntary. Furthermore, you have the right to withdraw from the study at any time. Costs and Compensation for Participating: There are no financial costs associated with participating in this study. It is anticipated that the focus group will last for approximately 75-90 minutes. Light refreshments will be provided during the group. As a token of the researchers’ appreciation for your you will receive a $30.00 gift certificate for your participation. 136   Contact Information for Questions and Concerns: If you have concerns or questions about this study, please contact the researchers: Amanda Hall, M.A. by phone (717)903-9140 or email: hallama3@msu.edu or Evelyn Oka, Ph.D., by phone: 517-432-9615; email: evoka@msu.edu; 439 Erikson Hall, East Lansing, MI, 48824. (http://www.educ.msu.edu/cepse/SchoolPsychology/faculty.asp) If you have any questions or concerns about your rights as a research participant or want to offer a complaint, please contact the Michigan State University's Human Research Protection Program at 517-355-2180, Fax 517-432-4503, or e-mail irb@msu.edu or regular mail at 207 Olds Hall, MSU, East Lansing, MI 48824. DOCUMENTATION OF INFORMED CONSENT  Yes, I _______________________________consent to participate in this research study.  No, I __________________________________do not participate in this research study. ________________________________________ Signature 137 ____________________________ Date   BIBLIOGRAPHY 138   BIBLIOGRAPHY Abrami, P. C., Poulsen, C., & Chambers, B. (2004). Teacher motivation to implement an educational innovation: Factors differentiating users and non-users of cooperative learning. Educational Psychology, 24, 201-216. APA Presidential Task Force on Evidence-Based Practice (2006). Evidence-based practice in psychology. The American psychologist, 61(4), 271-85. doi:10.1037/0003-066X.61.4.271 Atkinson, J. W. (1957). Motivational determinants of risk-taking behavior. Psychological Review, 64, 359-372. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/13505972 Baker, C. N., Kupersmidt, J. B., Voegler-Lee, M. E., Arnold, D. H., & Willoughby, M. T. (2010). Early Childhood Research Quarterly Predicting teacher participation in a classroom-based, integrated preventive intervention for preschoolers. Early Childhood Research Quarterly, 25, 270-283. doi:10.1016/j.ecresq.2009.09.005 Bandura, A. (1997). Self-efficacy. New York: W.H. Freeman and Company. Baraldi, A. N., & Enders, C. K. (2010). An introduction to modern missing data analyses. Journal of school psychology, 48(1), 5–37. doi:10.1016/j.jsp.2009.10.001 Battle, E. S. (1966). Motivational determinants of academic competence. Journal of Personality and Social Psychology, (6), 634-642. American Psychological Association. Retrieved from http://psycnet.apa.org/journals/psp/4/6/634/ Bender, W. N., & Shores, C. (2007). Response to Intervention. Thousand Oaks: Corwin Press. Benner, G. J., Nelson, J. R., Stage, S. a, & Ralston, N. C. (2010). The influence of fidelity of Implementation on the reading outcomes of middle school students experiencing reading difficulties. Remedial and Special Education, 32, 79-88. doi:10.1177/0741932510361265 Biggs, B. K., Vernberg, E. M., Twemlow, S. W., Fonagy, P., & Dill, E. J. (2008). Teacher adherence and its relation to teacher attitudes and student outcomes in an elementary school-based violence prevention program. School Psychology Review, 37, 533–549. Retrieved from http://www.nasponline.org/publications/spr/pdf/spr374biggs.pdf Boardman, A., Arguelles, M., Vaughen, S., Hughes, M., & Klingner, J. (2005). Special education teachers’ views on research-based practices. Journal of Special Education, 39, 168-180. doi:10.1177/00224669050390030401 Brandon, P. R., Taum, a K. H., Young, D. B., Pottenger, F. M., & Speitel, T. W. (2008). The complexity of measuring the quality of program implementation with observations: The case of middle school inquiry-based science. American Journal of Evaluation, 29, 235250. doi:10.1177/1098214008319175 139   Brantlinger, E., Jimenez, R., Klingner, J., Pugach, M., & Richardson, V. (2005). Qualitative studies in special education. Exceptional children, 71, 195–207. CEC. Retrieved from http://cec.metapress.com/index/HXQ1648763406W17.pdf Brophy, J. (2010). Motivating Students to Learn. New York: Routledge. Burns, M. K., & Ysseldyke, J. E. (2009). Reported prevalence of evidence-based instructional practices in special education. The Journal of Special Education, 43, (1), 3-12. Retrieved from http://sed.sagepub.com/content/43/1/3.short Cave, A., & Mulloy, M. (2010). How do cognitive and motivational factors influence teachers; degree of program implementation?: A qualitative examination of teacher perspectives. National Forum of Educational Administration and Supervision Journal, 27(4), 1-26. Retrieved from http://www.nationalforum.com/Electronic Journal Volumes/Cave, Agnes A Qualitative Examination of Teacher Perspectives NFEASJ V27 N4 2010.pdf Chambless, D. L., & Ollendick, T. H. (2001). Empirically supported psychological interventions: controversies and evidence. Annual review of psychology, 52, 685-716. doi:10.1146/annurev.psych.52.1.685 Cochrane, W. S., & Laux, J. M. (2008). A survey investigating school psychologists’ measurement of treatment integrity in school-based interventions and their beliefs about its importance. Psychology in the Schools, 45, 499–507. Wiley Online Library. doi:10.1002/pits Crabtree, B.F., & Miller, W.L. Doing qualitative research. Research methods for primary care, Vol. 3.(pp. 3-28)Thousand Oaks, CA, US: Sage Publications, In Crabtree, Benjamin F. (Ed); Miller, William L. (Ed), (1992). xvi, 276 pp. Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: are implementation effects out of control? Clinical psychology review, 18, 23–45. Elsevier. Retrieved from http://www.sciencedirect.com/science/article/pii/S0272735897000433 Deci, E. L. (1975). Intrinsic Motivation. New York: Plenum Press. Durlak, J. A. Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: a meta-analysis of schoolbased universal interventions. Child development, 82, 405-32. doi:10.1111/j.14678624.2010.01564.x Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. American journal of community psychology, 41, 327-350. doi:10.1007/s10464-008-9165-0 140   Eccles, J. S. (1983). Expectancies, values, and academic behaviors. In J. T. Spence (Ed.), Development of Achievement Motivation. San Francisco: W.H. Freeman and Company. Finnigan, K. S., & Gross, B. (2007). Do Accountability Policy Sanctions Influence Teacher Motivation? Lessons From Chicago’s Low-Performing Schools. American Educational Research Journal (Vol. 44, pp. 594–630). doi:10.3102/0002831207306767 Fixsen, D., Blase, K. A., Horner, R., & Sugai, G. (2009). Scaling-up evidence-based practices in education. Scaling-up Brief. Chapel Hill. Retrieved from http://www.swcompcenter.org/common_core_workshop/Scaling-UpReadinessforChangeBrief.pdf Foley, L. S. (2011). Exploring K-3 teachers’ implementation of comprehension strategy I instruction (CSI) using expectancy-value theory. Literacy Research and Instruction, 50, 195-215. doi:10.1080/19388071.2010.505680 Fuchs, Douglas, Mock, D., Morgan, P. L., & Young, C. L. (2003). Responsiveness-toIntervention: Definitions, Evidence, and Implications for the Learning Disabilities Construct. Learning Disabilities Research and Practice, 18, 157-171. doi:10.1111/15405826.00072 Gatti, G. G. (2003). Scott Foresman Reading effect size study. Evaluation report to Pearson Education. Good, R. H., & Kaminski, R. A. (1996). Assessment for instructional decisions: Toward a proactive/prevention model of decision-making for early literacy skills. School Psychology Quarterly, 11, 326-336. Retrieved from http://psycnet.apa.org/journals/spq/11/4/326 Graves, S. L., & Frohwerk, A. (2009). Multilevel modeling and school psychology: A review and practical example. School Psychology Quarterly, 24, 84-94. doi:10.1037/a0016160 Gresham, F.M. (1989). Assessment of treatment integrity in school consultation and prereferral I intervention. School Psychology Review, 18, 37-50. Retrieved from http://psycnet.apa.org/psycinfo/1989-27782-001 Gresham, F.M., Gansle, K. A., & Noell, G. H. (1993). Treatment integrity in applied behavior analysis with children. Journal of Applied Behavior Analysis, 26, 257-263. Society for the Experimental Analysis of Behavior. Retrieved from http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1297745/ Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33, 665-79. doi:10.1007/s10802-005-7646-2 141   Hansen, D. A. (1989). Lesson evading and lesson dissembling: Ego strategies in the classroom. American Journal of Education, 97, 184-208. doi:10.1086/443921 Henggeler, S. W., Melton, G. B., Brondino, M. J., Scherer, D. G., & Hanley, J. H. (1997). Multisystemic therapy with violent and chronic juvenile offenders and their families: the role of treatment fidelity in successful dissemination. Journal of Consulting and Clinical Psychology, 65(5), 821-833. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/9337501 Higa, C. K., Chorpita, B. F., Steele, R. G., Elkin, T. D., & Roberts, M. C. (2008). EvidenceBased Therapies: Translating Research into Practice Handbook of Evidence-Based Therapies for Children and Adolescents. In M.C. Roberts (Ed.), (pp. 45-61): Springer US. Hirschstein, M. K., Van Schoiack Edstrom, L., Frey, K. S., Snell, J. L., & MacKenzie, E. P. (2007). Walking the talk in bullying prevention: Teacher implementation variables related to initial impact of the Steps to Respect program. School Psychology Review, 36, 3. Retrieved from http://www.nasponline.org/publications/spr/pdf/spr361hirschstein.pdf Jussim, L., Robustelli, S. L., & Cain, T. R. (2009). Teacher expectations and self-fulfilling prophecies. In K. R. Wentzel (Ed.), Handbook of motivation at school. New York: Florence: Routledge. Kratochwill, T. R., & Stoiber, K. C. (2002). Evidence-based interventions in school psychology: Conceptual foundations of the Procedural and Coding Manual of Division 16 and the Society for the Study of School Psychology Task Force. School Psychology Quarterly, 17, 341-389. doi:10.1521/scpq.17.4.341.20872 Kukla, A. (1972). Foundations of an attributional theory of performance. Psychological Review, 79, 454-470. American Psychological Association. Retrieved from http://psycnet.apa.org/journals/rev/79/6/454/ Lee, O., Penfield, R., & Maerten-Rivera, J. (2009). Effects of fidelity of implementation on science achievement gains among English language learners. Journal of Research in Science Teaching, 46, 836-859. doi:10.1002/tea.20335 Marshall, C., & Rossman, G. B. (1999). Designing Qualitative Research (3rd ed.). Thousand Oaks: Sage Publications. McIntyre, L. L., Gresham, Frank M, DiGennaro, F. D., & Reed, D. D. (2007). Treatment integrity of school-based interventions with children in the journal of applied behavior analysis 1991-2005. Journal of applied behavior analysis, 40(4), 659-72. Retrieved from http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2078573&tool=pmcentrez&r endertype=abstract 142   Mowbray, C. T., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24, 315-340. Sage Publications. Retrieved from http://aje.sagepub.com/content/24/3/315.short National Association of School Psychologists. (2010). Model for Comprehensive and Integrated School Psychological Services, NASP Practice Model Overview. [Brochure]. Bethesda, MD: Author. Neves de Jesus, S., & Lens, W. (2005). An integrated model for the study of teacher motivation. Applied Psychology, 54, 119–134. Wiley Online Library. Retrieved from http://onlinelibrary.wiley.com/doi/10.1111/j.1464-0597.2005.00199.x/full Newman, D., & Jaciw, A. P. (2005). Effectiveness of Scott Foresman’s Links to Reading First as an intervention for struggling readers. Evaluation report to Pearson Education. Noell, G., & Gresham, F. (2002). Does treatment integrity matter? A preliminary investigation of Instructional implementation and mathematics performance. Journal of Behavioral Education, 11, 51-67. Retrieved from http://www.springerlink.com/index/5CUTN3KGF538NPQB.pdf O’Donnell, C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K-12 curriculum intervention research. Review of Educational Research, 78, 33-84. doi:10.3102/0034654307313793 Perepletchikova, F., & Kazdin, A. E. (2005). Treatment integrity and therapeutic change: Issues and research recommendations. Clinical Psychology: Science and Practice, 12, 365–383. Wiley Online Library. doi:10.1093/clipsy/bpi045 Power, T. J., Blom-Hoffman, J., Clarke, A. T., Riley-Tillman, T. C., Kelleher, C., & Manz, P. H. (2005). Reconceptualizing intervention integrity: A partnership-based framework for linking research with practice. Psychology in the Schools, 42, 495-507. doi:10.1002/pits.20087 Ransford, C., Greenberg, M., Domitrovich, C. E., Small, M., & Jacobson, L. (2009). The role of teachers' psychological experiences and perceptions of curriculum supports on the implementation of a social and emotional learning curriculum. School Psychology Review, 38, 510-532. Retrieved from http://www.prevention.psu.edu/people/documents/spr384ransford.pdf Retelsdorf, J., Butler, R., Streblow, L., & Schiefele, U. (2010). Teachers’ goal orientations for teaching: Associations with instructional practices, interest in teaching, and burnout. Learning and Instruction, 20, 30–46. Elsevier. doi:10.1016/j.learninstruc.2009.01.001 Ringwalt, C. L., Ennett, S., Johnson, R., Rohrbach, L. A., Simons-Rudolph, A., Vincus, A., & Thorne, J. (2003). Factors associated with fidelity to substance use prevention curriculum guides in the nation’s middle schools. Health Education & Behavior, 30, 375-391. Sage Publications. doi:10.1177/1090198103253627 143   Ringwalt, C., Vincus, A. A., Hanley, S., Ennett, S. T., Bowling, J. M., & Rohrbach, L. A. (2009). The prevalence of evidence-based drug use prevention curricula in U.S. middle schools in 2005. Prevention Science, 10, 33–40. Springer. Retrieved from http://www.springerlink.com/index/B11K48726NJM58V7.pdf Sackett, D. L., Rosenberg, William, M., Gray, J. M., Haynes, R. B., & Richardson, W. S. (1996). Evidence based medicine: what it is and what it isn’t. British Medical Journal, 312(7172). Retrieved from http://www.bmj.com/content/312/7023/71.short Sanetti, L. M. H., & Kratochwill, T. R. (2009). Toward Developing a Science of Treatment Integrity  : Introduction to the Special Series. School Psychology Quarterly, 38, 445-459. Sanetti, L. M. H., & Kratochwill, T.R. (2009)b. Treatment integrity assessment in the schools: An evaluation of the Treatment Integrity Planning Protocol. School Psychology Quarterly, 24, 24 -35. Educational Publishing Foundation. doi:10.1037/a0015431 Schunk, D. H., Pintrich, P. R., & Meece, J. L. (2008). Motivation in education: Theory, research, and applications. Upper Saddle River: Pearson. Shinn, M. R., Good, R. H., Knutson, N., Tilly, W. D., & Collins, V. (1992). Curriculum-based measurement of oral reading fluency: A confirmatory analysis of its relation to reading. School Psychology Review, 21, 459-479. Retrieved from http://www.leasesped.org/Northern IL Partnership/CBM validity/CBM of ORF.pdf Shinn, MR, & Shinn, MM. (2002). Aimsweb training workbook: Administration and scoring of reading curriculum-based measurement (R-CBM) for use in general outcome measurement. , Inc. Available at: www. aimsweb. com/. Retrieved from http://www.kunaschools.org/resources/documents/AdminandScoringRCBM09292005.pdf Skinner, C. H., McLaughlin, T., & Logan, P. (1997). Cover, Copy, and Compare  : A selfmanaged academic intervention effective across skills , students, and settings. Journal of Behavioral Education, 7, 295–306. Springer. Retrieved from http://www.springerlink.com/index/x3xn344127lm0635.pdf Snell, M. E. (2003). Applying Research to Practice: The More Pervasive Problem? Research and Practice for Persons with Severe Disabilities, 28(3), 143-147. doi:10.2511/rpsd.28.3.143 Spence, J. T., & Helmreich. (1983). Achievement-related motives and behaviors. In J. T. Spence (Ed.), Development of Achievement Motivation. San Francisco: W.H. Freeman and Company. Stein, M. L., Berends, M., Fuchs, D., McMaster, K., Sáenz, L., Yen, L., Fuchs, L.S., et al. (2008). Scaling up an early reading program: Relationships among teacher support, fidelity of implementation, and student performance across different sites and years. 144   Educational evaluation and policy analysis, 30, 368–388. SAGE Publications. doi:10.3102/0162373708322738 Sterling-Turner, H. E., Watson, T. S., & Moore, J. W. (2002). The effects of direct training and treatment integrity on treatment outcomes in school consultation. School Psychology Quarterly, 17, 47-77. doi:10.1521/scpq.17.1.47.19906 Sugai, G., & Horner, R. R. (2006). A promising approach for expanding and sustaining schoolwide positive behavior support. School Psychology Review, 35, 245-259. National Association of School Psychologists. Retrieved from http://www.nasponline.org/publications/spr/pdf/spr352sugai.pdf Tanol, G. (2010). Treatment Fidelity: Relation to Treatment Acceptability and Change over Time. University of Minnesota. Vaughn, S., & Fuchs, Lynn S. (2003). Redefining Learning Disabilities as Inadequate Response to Instruction: The Promise and Potential Problems. Learning Disabilities Research and Practice, 18, 137-146. Wiley Online Library. doi:10.1111/1540-5826.00070 Weiner, B. (1985). An attributional theory of achievement motivation and emotion. Psychological Review, 92, 548-573. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/3903815 Wentzel, K. R., & Wigfield, A. (1998). Academic and social motivational influences on students’ academic performance. Educational Psychology Review, 10(2), 155–175. Springer. Retrieved from http://www.springerlink.com/index/X6R675P747513738.pdf Wickstrom, K. F., Jones, K. M., LaFleur, L. H., & Witt, J. C. (1998). An analysis of treatment integrity in school-based behavioral consultation. School Psychology Quarterly, 13, 141154. Guilford Publications. Retrieved from http://psycnet.apa.org/journals/spq/13/2/141/ Wigfield, A., & Eccles, J. (2000). Expectancy-Value Theory of Achievement Motivation. Contemporary educational psychology, 25, 68-81. doi:10.1006/ceps.1999.1015 Woolfolk, A. H., Hoy, W. K., & Davis, H. A. (2009). Teachers’ self-efficacy beliefs. In K. R. Wentzel (Ed.), Handbook of motivation at school. Florence Routledge. Wozney, L., Venkatesh, V., & Abrami, P. C. (2006). Implementing computer technologies: Teachers’ perceptions and practices. Journal of Technology and Teacher Education, 14, 173-207. ASSOCIATION FOR THE ADVANCEMENT OF COMPUTING IN. Retrieved from http://doe.concordia.ca/cslp/cslp_cms/sites/all/themes/jframe/downloads/PDF/wozneyetal jtte141.pdf Yeaton, W. H., & Sechrest, L. (1981). Critical dimensions in the choice and maintenance of successful treatments: strength, integrity, and effectiveness. Journal of Consulting and 145   Clinical Psychology, 49, 156-167. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/7217482 Ysseldyke, J., & Bolt, D. M. (2007). Effect of technology-enhanced continuous progress monitoring on math achievement. School Psychology Review, 36, 453-467. National Association of School Psychologists. Retrieved from http://questgarden.com/76/55/0/090212083730/files/progress monitoring math.pdf 146