DISTRICT RESPONSE TO MICHIGAN’S TEACHER EVALUATION LAWS By Kevin Mowrer A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of K-12 Educational Administration- Doctor of Philosophy 2014 ABSTRACT DISTRICT RESPONSE TO MICHIGAN’S TEACHER EVALUATION LAWS By Kevin Mowrer Between 2008 and 2011, Michigan legislators passed a series of laws mandating changes to school districts’ teacher evaluation systems. The state offered little guidance to school districts implementing these changes. Districts around the state responded in many different ways. Through an analysis of online surveys and interviews of teachers and administrators, as well as artifacts from the implementation of the laws, this dissertation investigates the relationship of district response to teacher perceptions in six school districts in southeastern Michigan. Copyright by KEVIN MOWRER 2014 TABLE OF CONTENTS LIST OF TABLES .......................................................................................................................... vi LIST OF FIGURES………………………………………………………………………………………………………………………… viii CHAPTER 1: INTRODUCTION .................................................................................................... 1 CHAPTER 2: LITERATURE REVIEW…………………………………………………………………….……………………. OVERVIEW OF NATIONAL AND MICHIGAN SCHOOL REFORM 1994-2011….………….………………… MICHIGAN TEACHER EVALUATION LAW PA 101, 102, AND 103 OF 2011……………….……………….. TEACHERS: A PROFILE OF THE PEOPLE AND THEIR MOTIVATION…………………..………………………… BUSINESS SECTOR/HUMAN RESOURCE AND PERFORMANCE APPRAISAL RESEARCH…………….. IMPLEMENTATION OF PERFORMANCE APPRAISAL SYSTEMS…………………………………………………. TEACHER PERFORMANCE APPRAISAL RESEARCH……………………………………………………………………. DISCUSSION OF PERFORMANCE APPRAISAL RESEARCH……………………….………………………………… SYNTHESIS OF LITERATURE…………………………………………………………………………………………………….. 6 6 9 13 14 16 21 22 22 CHAPTER 3: RESEARCH DESIGN AND METHODS…………………………………………………….……………….. 24 PARTICIPANT DISTRICTS………………………………………………………………………………………………………….. 27 PHASE ONE (A, B, C): IMPLEMENTATION COHERENCE….………………………………………………………… 27 Phase One A: Electronic Surveys…………………………………………………………………………………. 28 Sample Selection for Phase One A…………..……………………………………………………. 28 Rationale for Phase One A…………………………………………………………………………… 29 Data Collection for Phase One A………………………….………………………………………. 31 Data Analysis for Phase One A………………………………………………………………………31 Phase One B: Documents/Artifacts………………………………………………………….................... 32 Rationale for Phase One B……………………………………………………………………………..32 Data Collection for Phase One B……………………………………………………………………. 32 Data Analysis for Phase One B……........………………………………........................... 33 Phase One C: Interviews with Administrators…………………..…..……………………………………. 33 Sample Selection for Phase One C…………………..………………………………………….. 33 Rationale for Phase One C…………………………………………………………………………… 33 Data Collection for Phase One C…………………………………………………………………… 34 Data Analysis for Phase One C……………………………………………………………………… 34 Phase Two: Interviews with Teachers……….………………….…………………………………………….. 35 Sample Selection for Phase Two…………………….………………………………………………36 Rationale for Phase Two……………………………….……………………………………………….36 Data Collection for Phase Two…….……………….………………………………………………. 37 Data Analysis for Phase Two……………………….……………………………………………….. 37 CONFIDENTIALITY AND CONSENT……………………………………….…………………………………………………. 37 INTERNAL RELIABILITY AND VALIDITY……………………………….…………………………………………………… 38 EXTERNAL RELIABLITY AND VALIDITY……………………………….……………………………………………………. 39 COMPLICATIONS IN DESIGN AND METHODS……………...........………..……………………………………….. 39 CHAPTER 4: DATA ANALYSIS…………………………………………………………………………………………………… IMPLEMENTATION COHERENCE ACROSS DISTRICTS ……………………………………………………………….. TEACHER AND ADMINISTRATOR PERCEPTIONS BY POSITION TYPE………………………………………….. Most Important Part of Implementation…………………………………………………………………….. What Should Have Been Included in Implementation…………………………………………………. How Implementation Influenced Teachers’ Perceptions……………………………………………… Teacher Interview Responses about Noe’s Attributes of Effective Evaluation Systems.. Reliability…………………………………………………………………………………………………………………… Relevance……………………………………………………………..……………………………………………………. Specificity………………………………………………………………………………………………….................. Strategic Congruence…………………………………..…………………………………………………………….. iv 41 41 52 54 55 56 57 61 61 63 65 Acceptability…………………………………………………………………………………………….……………….. 66 ACCEPTABILITY AND MORALE BY POSITION TYPE…………………………………………………………………… 68 IMPLEMENTATION COHERENCE AND PERCEPTIONS BY DISTRICT……………………………………………. 72 District 1: Large Suburban District……………………………………………………………………………… 72 District 1: Implementation…………………………………………………………………………… 72 District 1: Teacher and Administrator Perceptions….....………………………………… 77 District 1: Relationship between Implementation and Perception………………… 80 District 2: Medium Town Fringe…………………………………………………………………………………. 84 District 2: Implementation…………………………………………………………………………… 84 District 2: Teacher and Administrator Perceptions….....……………………………….. 89 District 2: Relationship between Implementation and Perception………………… 94 District 3: Suburb Large……………………………………………………………………………………………… 97 District 3: Implementation……………………………………………………………………………. 97 District 3: Teacher and Administrator Perceptions….....………………………………… 104 District 3: Relationship between Implementation and Perception………………… 110 District 4: Medium Rural Fringe………………………………………………………………………………….. 114 District 4: Implementation…………………………………………………………………………… 114 District 4: Teacher and Administrator Perceptions….....………………………………… 120 District 4: Relationship between Implementation and Perception………………… 127 District 5: Small Rural Fringe……………………………………………………………………………………… 130 District 5: Implementation………………………………………………………………………….… 130 District 5: Teacher and Administrator Perceptions….....………………………………… 136 District 5: Relationship between Implementation and Perception………………… 143 District 6: Large Rural Fringe……………………………………………………………………………………... 145 District 6: Implementation…………………………………………………………………………... 145 District 6: Teacher and Administrator Perceptions….....…………………………….… 151 District 6: Relationship between Implementation and Perception….....…………. 156 FORMAL AND INFORMAL COMMUNICATION ABOUT KEY ISSUES OF IMPLEMENTATION…………. 159 CHAPTER 5: SYNTHESIS OF FINDINGS…………………………………………………………………………… …………. Implementation Steps Districts Followed Closely………………………………………………………… Implementation Steps Districts Did Not Follow Closely or At All…………………………………. Rationale for Lack of Implementation Coherence………………………………………………………. Challenges with Grote’s Framework…………………………………………………………………………… Findings About All Districts and The Differences Between Districts……………………………. 171 172 173 175 177 177 CHAPTER 6: CONCLUSIONS AND IMPLICATIONS FOR PRACTICE……………………………………………… 186 APPENDICES…………………………………………………………………………………………………………………………… Appendix A: Admin/Union Electronic Survey Questions…………………………………………….. Appendix B: Non-Union Representative Electronic Teacher Surveys…………………………… Appendix C: Administrator Interview Questions…………………………………………………………. Appendix D: Teacher Interview Questions………………………………………………………………….. Appendix E: Methodology- Notes from the field…………………………………………………………. 193 194 200 205 207 210 BIBLIOGRAPHY……………………………………………………………………………………………………………………….. 215 v LIST OF TABLES Table 1: Michigan Laws Passed in Response to Race to the Top………….………….………….………………..………….…………………… 9 Table 2: Changes to Michigan’s Teach Evaluation Law with Possible Problems It Meant To Address…………….………..…… 11 Table 3: Grote’s Components of Performance Appraisal Implementation…………………………….......…….………..………………… 20 Table 4: Phases of Research………….………….………….………..………….………….………….………..………………..…….………..……………….. 26 Table 5: Participating Districts by Size and Type…….………..………….………….………….………..…………………….………..……………….. 27 Table 6: Implementation Step Summary By District……………………………………………………………………………………………………… 44 Table 7: Views of Evaluation Implementation, by District and Position Type ………………………………………………………………. 53 Table 8: Teacher Views of Evaluation System Attributes, by District …………………........................................................... 59 Table 9: Summary of Perception Responses by Position Table…………………………………………………………………………………... 70 Table 10: District 1 Implementation Survey Reponses…………………………………………………………………………………………………. 73 Table 11: D1 Teacher/Administrator Survey Perception of Implementation’s Influence………………………………………… 77 Table 12: District 1 Interview Perception Response Summary ……………………………………………………………………………………. 78 Table 13: District 1 Interview Perception Response Summary based on Noe’s Attributes……………………………………………… 79 Table 14: District 1 Implementation and Influence on Perception of the System………………………………………………………… 83 Table 15: District 2 Implementation Survey Reponses …………………………………………………………………………………………………. 85 Table 16: D2 Teacher/ Administrator Survey Perceptions of Implementation’s Influence…………………………………… 90 Table 17: District 2 Interview Perception Response Summary ……………………………………………………………………………………… 92 Table 18: District 2 Interview Perception Response Summary based on Noe’s Attributes…………………………………………….. 93 Table 19: District 2 Implementation and Influence on Perception of the System…………………………………………………………… 96 Table 20: District 3 Implementation Survey Reponses…………………………………………………………………………………………………… 98 Table 21: D3 Teacher/ Administrator Survey Perceptions of Implementation’s Influence…………………………………………….. 105 Table 22: District 3 Interview Perception Response Summary………………………………………………………………………………………. 107 Table 23: District 3 Interview Perception Response Summary based on Noe’s Attributes…………………………………………….. 108 Table 24: District 3 Implementation and Influence on Perception of the System………………………………………………………….. 113 Table 25: District 4 Implementation Survey Reponses………………………………………………………………………………………………….. 115 Table 26: D4 Teacher/ Administrator Survey Perceptions of Implementation’s Influence……………………………………………. 121 vi Table 27: District 4 Interview Perception Response Summary ……………………………………………………………………………………... 123 Table 28: District 4 Interview Perception Response Summary based on Noe’s Attributes……………………………………………. 124 Table 29: District 4 Implementation and Influence on Perception of the System………………………………………………………….. 129 Table 30: District 5 Implementation Survey Reponses……………………………………………..…………………………………………………… 131 Table 31: D5 Teacher/ Administrator Survey Perceptions of Implementation’s Influence…………………………………………….. 137 Table 32: District 5 Interview Perception Response Summary …………………………………..………………………………………………….. 140 Table 33: District 5 Interview Perception Response Summary based on Noe’s Attributes……………………………………………… 141 Table 34: District 5 Implementation and Influence on Perception of the System……..…………………………………………………… 144 Table 35: District 6 Implementation Survey Reponses……………………………………………..…………………………………………………… 146 Table 36: D6 Teacher/ Administrator Survey Perceptions of Implementation’s Influence……………………………………………. 152 Table 37: District 6 Interview Perception Response Summary ……………………………………………………………………………………… 154 Table 38: District 6 Interview Perception Response Summary based on Noe’s Attributes…………………………………………….. 155 Table 39: District 6’s Implementation and Influence on Perception of the System……………………………………………………….. 158 Table 40: Formal and Informal Communication about Key Implementation Issues…………………………………………………….. 161 Table 41: Degrees of Administrator Distance from the State’s Teacher Evaluation Mandate……………………………………….. 171 Table 42: Teacher Survey Perceptions of Acceptability and Morale at the Building Level…………………………………………….. 178 Table 43: Teacher Interview Perceptions of Acceptability …….……………………………………………………………………………………. 179 Table 44: Number of Teachers Who Declined Participation By District…………………………………………………………………………… 212 vii LIST OF FIGURES Figure 1: Conceptual Map of Teacher Response to Michigan’s Teacher Evaluation Laws……………………………………… viii 172 CHAPTER 1: INTRODUCTION Teacher evaluation is a hotly debated topic in the State of Michigan and across America. Education reforms identify teacher performance appraisal as a key lever for improving schools, beginning nationally with the No Child Left Behind Act of 2001 and continuing through the Race to the Top initiative in 2008. This is based on theoretical and empirical research which highlights teachers and their instructional practices as the most important school-based factor in student achievement (Nye, Konstantopoulos, & Hedges, 2004). Therefore, the evaluation of teacher effectiveness has become increasingly important to practitioners and researchers. There are many competing voices in this debate. Unions, professional groups, politicians, policy-makers, private evaluation developers, and researchers want a say in how teachers are evaluated. So, it is challenging to reach consensus on the best steps to improve schools and student performance by utilizing teacher evaluations. In July of 2011, the State of Michigan passed several laws that prescribe how school districts and schools must evaluate teachers. Under the new state law, teacher evaluations must include student learning growth data (starting out at 25% and moving up to 40% of the evaluation in the 2014-2015 school year), relevant instructional training (taken by teachers outside the school day and school year), and significant accomplishments and contributions (not clearly defined) above the normal expectations for teachers in their peer groups. According to the law, the evaluations must be used to establish rankings of teachers (highly effective, effective, minimally effective, and ineffective). The evaluations and rankings must also be used in setting teacher compensation and in lay-off and tenure decisions. The state law, however, did not specify the evaluation frameworks, tools, or systems for evaluating teachers, though a pilot and a set of recommendations were promised. The state merely identified four categories for the final teacher ratings. Observations for failing and inexperienced teachers were mandated. Evaluation procedures were not to be negotiated with unions in teacher contracts. Significantly, the law specified that the state would develop or identify an observation framework and observation tools which would be used in schools across the state; but, until such evaluation tools and procedures were developed or identified, local districts were to devise their own frameworks, tools, and systems. Michigan’s legislation was framed as an attempt to take advantage of Race to the Top funding, awarded to districts that best met guidelines and requirements set by the U. S. Department of Education. Michigan did not win any funds, but legislators used both the competition and the lack of “winning” funding to pass the teacher evaluation laws in the state. 1 At this point, it is important to define a few key terms regarding teacher performance appraisal systems. In this project, an “observation framework” is the rubric that districts use for observational assessments of teacher behavior. Typically, districts use frameworks developed or influenced by an outside expert. For years these frameworks served as the foundation for teacher evaluations in Michigan. The state set up an advisory group, the Michigan Council on Educator Effectiveness (MCEE), to pilot several of these frameworks. The four frameworks chosen by the state were as follows: Charlotte Danielson’s Framework for Teaching, Marzano Teacher Evaluation Model, The Thoughtful Classroom, or 5 Dimensions of Teaching and Learning. A version of Danielson’s framework was used by all six districts in this study. Many of these frameworks, however, did not have defined methods to provide feedback to teachers in a timely way and use that feedback as a basis for performance appraisal. The electronic/online tools used to get feedback to teachers and to encourage teacher reflection are referred to as “observation tools” in this study. Finally, the overall system, including the observation framework, observation tool, definitions of relevant training and significant contributions, measurement of student growth data, and points or weight attributed to each domain/section of effective teacher behaviors are referred to as the “teacher evaluation system.” All school districts in Michigan were put in a precarious position. Despite a mandate to not formally and collectively negotiate the system, districts did what they could. They did not have steps to follow. They did not have clear definitions of the measures in the state law. They did not have guidance on tools or framework. They still do not. Districts in the state and in the study had to move forward, using their best judgment to navigate what state legislators wrote as law. Existing research on teacher evaluation utilizes a variety of approaches. There are several waves of research, some of it based in other countries. The American empirical work, though limited in comparison to the international, is situated in the 1980’s and 1990’s. U. S. research focuses on behavioral tools, which attempt to define what good teaching looks like. It is also based on classroom evaluations. Research in the last five years that incorporates test scores and value-added measures remains limited and contested, with some controversy as to whether available measures are stable and reliable. There is little research on how school districts implement their evaluation tools and procedures. Michigan’s law currently stipulates some features of teacher evaluation systems that districts must utilize. The law also specifies that the state shall provide districts with a template evaluation framework. As of this writing, a state committee had piloted four observational frameworks, but the legislature had not identified any as “the one” for districts to use. Until an observational framework and an overarching evaluation system are made available, school districts have found it necessary to improvise. They have either purchased observational frameworks and tools from commercial vendors or developed their own, perhaps in 2 collaboration with other districts. In addition to developing or purchasing observational frameworks and tools, districts have had to create an overall system, which was supposed to include student growth data, relevant training, and significant, relevant contributions. All of these areas were to be brought together in a system that accurately rated teachers by rankings set by the state. A few terms used above must be defined, as they are defined by 2011’s Revision of the School Code, Section 380.1248. The first is “student growth data.” According to the law, this is evidence that students grew in the individual teacher’s class and subject matter during the given school year. If state mandated standardized tests are available, those must be used. If they are not available, pre-tests and post-tests are to be used to measure student growth. There is no explanation of the validity or reliability necessary to allow locally developed pre and post-tests to be used as a key factor in teacher ratings. The second is “relevant training.” Relevant training is defined this way in the state law: “This factor shall be based on completion of relevant training other than the professional development or continuing education that is required by the employer or by state law, and integration of that training into instruction in a meaningful way” (p.1). The manner in which districts are to measure this relevant training and its integration has not been defined. The third is “significant, relevant contributions.” The law defines these contributions this way: “This factor shall be based on whether the individual contributes to the overall performance of the school by making clear, significant, relevant contributions above the normal expectations for an individual in his or her peer group and having demonstrated a record of exceptional performance” (p.1). As with the previous two terms, there is no clear guidance form the state on how to judge if a teacher’s contributions are “above the normal expectations for an individual in his or her peer group.” Upon this backdrop of new terms, unclear guidelines, and vague direction, districts were asked to react. However districts developed or obtained their tools, frameworks, and systems, they all had to implement them to meet the requirements of state law. The method of implementation was standard. When compared to research from the fields of business and human resources, many important steps were omitted or under-utilized. Because of the consistency in implementation, the manner in which districts undertook those steps varied. The way that districts implemented their evaluation systems shared many common shortcomings; therefore, teacher perceptions of the evaluation procedures also showed much general consistency across districts. In particular, districts’ responses offer an opportunity to observe how district implementation correlates with the perceptions of teachers and administrators regarding the evaluation system’s acceptability and motivational power among employees. 3 My research project analyzes district implementation of Michigan’s teacher evaluation law (PA 101, 102, and 103 of 2011). In my literature review, I will define what the law says, trace federal and state action that led to the law’s passage, and analyze the theories of action embedded in the law. I will review the literature on teachers and their motivation. I will then review the literature on theoretical and empirical research on performance appraisal and its implementation from business and human relations fields. This study’s empirical work is based on six school districts in a medium-sized suburban county in southeast Michigan. It examines how these six districts implemented their teacher evaluation systems and teachers’ perceptions of the evaluation system and its implementation. The project analyzes the coherence of districts’ implementation of the new law, using Grote’s (1996) framework for employee evaluation implementation. The State of Michigan contracted with researchers to develop reliable frameworks and tools to evaluate teacher performance in future years. Those researchers concluded their study and made recommendations. Currently, the state legislature has not taken any action to implement the researchers’ conclusions in state law. As in other states, Michigan lawmakers hope to improve teacher quality. The theory of action embedded in PA 101, 102, and 103 seems to be to reduce job security and to use student growth data, ranging from undefined to tightly-defined in subject disciplines, as well as poorly described significant training and relevant contributions by the teachers. These evaluations are required to be used in determining teacher compensation and layoffs. The legal requirement of the performance appraisal, and mandated rankings, has major consequences for teachers. There is the possibility of a flawed implementation in the absence of a clear teacher evaluation system or a well-aligned student assessment device, defined contributions, or vetted professional development in place. The implementation steps and the extent to which they were fully used seemed to relate directly to the way teachers perceived the new teacher evaluation system. The implementations and perceptions differed slightly among districts, but greatly among teachers and administrators. The importance of several steps in the implementation became clear. The project’s primary goal was to provide Michigan school administrators with suggested procedures for implementing the state’s evaluation system when it becomes available. It also sought to discover how variations in districts’ teacher evaluation implementation influence teachers’ perceptions of the evaluation system. The research, however, found only modest variation in implementation across the six sample districts. Accordingly it also found much consistency in teacher perceptions, based on usage, communication, and method of the implementation steps. The project did not evaluate different teacher observation frameworks or tools. In fact, in all six sample districts used the same observation framework and similar tools. Though the teacher observation frameworks or tools may influence teacher 4 perception, they are not the focus of the study. It is also not meant as a criticism of districts or the State of Michigan. As the literature review will reveal, the state and the districts had limited choices, as the national education reform movement pushes for changes in exchange for the chance of more funding. In a time of dwindling resources, the state and districts may have felt they must comply, and do so immediately. 5 CHAPTER 2: LITERATURE REVIEW District implementation of teacher evaluation framework, tools, and system in American schools is limited. State lawmakers historically restrained themselves from legislating guidelines for teacher evaluation systems (other than in tenure laws, which dictated classroom observations and timelines), so districts have generally not had any established evaluation guidelines. In some cases, large cities have implemented evaluation systems, but little research describes these cases of actual implementation. Most of the research relates to the evaluation frameworks, tools, and practices for effective teaching, rather than the application or integration of these systems. Therefore, the research base for this project is limited and fragmented. The literature review will cover the existing research on these disconnected parts. The review begins by describing the history, context, and specific wording on Michigan’s teacher evaluation law. I then will review a profile of teachers, their attitudes, and their motivations. I will explain and synthesize research and standards from human resources and educational research on performance appraisal. I will compare these recommended procedures (specifically from the field of human relations) to guidelines for teacher performance appraisal set forth by the new law. Finally, I will synthesize these literatures to put forth a framework for evaluating Michigan districts’ responses to the teacher evaluation law. The literature review will cover the existing research on these disconnected parts. The review begins by describing the history, context, and specific wording of Michigan’s teacher evaluation law. I then will review a profile of teachers, their attitudes, and their motivations. Next I will explain and synthesize research from human resources and educational research on performance appraisal. I will compare recommended procedures from the field of human relations to the guidelines for teacher performance appraisal embodied in Michigan’s new law. Finally, I will synthesize these literatures to put forth a framework for evaluating Michigan districts’ responses to the teacher evaluation law. OVERVIEW OF NATIONAL AND MICHIGAN REFORM 1994-2011 Proposal A centralized funding to the state level, tying all education funding to enrollment and directly to the students the system would serve. Its central selling point to voters was a reduction in property taxes. The measure was approved after the legislature repealed many property taxes earlier in its 1993-1994 session. Proposal A replaced property taxes as the chief funding source and increased sales and sin (tobacco, alcohol, lottery) taxes. Therefore, all monies for property taxes were sent to the state capital, Lansing, and then sent back to local districts. Districts whose residents paid higher property taxes were rewarded by higher funding levels, but a minimum level of funding was assured for districts. Rather than levying, collecting, and keeping taxes, districts were required to send monies to the state, greatly increasing state control of local districts, but were promised more equality in funding, guaranteeing a base foundation allowance. Proposal A also 6 included protections against districts requesting millages for operating expenses. In addition, Proposal A’s educational reform initiative focused on school choice. Public charters, as well as inter-district school choice, became the law of the state. Much rhetoric, from the governor of that era, John Engler, centered on the equality of funding, the reduction of property taxes, the introduction of school choice, and the elimination of local district authority to levy taxes to cover operating expenses. Under Proposal A, the state controlled all operational funding, while allowing students and parents to “take their funding with them” to charter schools and other districts via schools-of-choice. With the state in control of funding, dictating other portions of the state’s system seemed within reach of Lansing. Initially, test scores from the Michigan Educational Assessment Program (MEAP) became more important points of pride for school districts. To encourage all high schools, both public and private, to participate in the MEAP, the state developed the Michigan Merit Award, which was given to graduating seniors between 2000 and 2009. This program granted up to $3,000 to students who performed exceptionally well on the Michigan Educational Assessment Program tests. It was discontinued in 2009 due to declines in available state revenues; it also established the state as opportunity providers for students. Over time, the state transformed initial public recognition or shaming transformed into state intervention if standardized test scores were low enough for a school or a district. The Improving America’s Schools Act of 1994, an amendment to the Elementary and Secondary School Act (ESEA), passed around the same time as Proposal A. It required states to develop and implement challenging standards, Goals 2000. Title I schools were required to make Adequate Yearly Process (AYP) with the students receiving Title I services. Those Title I schools without adequate progress would be forced to develop action plans (Wright, Wright, and Heath, 2004). Another major move by the state was introducing state standards, as dictated by Goals 2000, a national initiative advanced by President Bill Clinton. The first set of Michigan standards were introduced in 1997. The standards, both nationally and in Michigan, were positioned as methods to help teachers and administrators improve student standardized test scores. Though not directly tied to funding in Michigan, one might anticipate that schools and districts with higher test scores would attract more students, while schools with lower test scores would see their students leave, which could provide a financial incentive to align curriculum and instruction to the new standards. Educators scrambled to meet the new standards in their efforts to improve schools and districts (Koch, 1999). States set standards and then allowed local districts and educational units to figure out how to meet them. Tests, set by most states, would measure the progress of schools and districts toward the goals and standards set. This happened in 7 Michigan, centralizing control of what was taught in schools and holding schools accountable through standardized tests, insofar as district employees understood and implemented the standards. In 2001, Congress amended the Elementary and Secondary Education Act (ESEA) with No Child Left Behind (NCLB). The foundation of the law was that every American student would be proficient in reading and math by 2013-2014. Schools that did not make adequate yearly progress would face sanctions. The law stipulated that subgroups of students, including students of different ethnicities, disabilities, English language learning skills, and low-income status, must improve. NCLB also featured new measures to improve accountability, choice, and proposed flexibility for states, districts, and schools (Wright, Wright, and Heath, 2004). On April 20, 2006, Governor Jennifer M. Granholm signed into law a rigorous new set of statewide graduation requirements called the Michigan Merit Curriculum. This greatly “raised the bar” from former state standards, which only included government and health courses. The new curriculum included four years of English and math, including an Algebra II requirement. Three years of science were also mandated, including either chemistry or physics. The merit curriculum also included two years of a world language, starting with the class of 2016. In addition, the Michigan Merit Exam (MME) was launched, largely comprised of the ACT, a college admissions test. Schools had to adjust their own graduation requirements to meet the state’s new requirements (Ackley, 2010). Through these requirements, the state set the classes by which districts would be required to deliver the standards. President Obama’s first formal education policy announcement came in July of 2009 with the Race to the Top funding initiative. States could compete for portions of $4.35 billion, called in the press release “a historic investment” (Gibbs, 2009) for adopting reforms in the following areas: designing and implementing rigorous standards and high-quality assessments, attracting and keeping great teachers and leaders in America’s classrooms, supporting data systems that inform decisions and improve instruction, using innovation and effective approaches to turn-around struggling schools, and demonstrating and sustaining education reform (Gibbs 2009). In response to RTTT Michigan’s Department of Education (MDE) and Governor Granholm, Michigan’s legislature passed five educational reform bills in December of 2009 (Lane, 2010). The bills, House Bills 4787-4788 and House Bill 5596, and Senate Bills 926 and 981, included the following measures:  A process for the state to identify the lowest-achieving 5 percent of public schools and place them under supervision of a state school reform officer. Schools that are identified would submit redesign plans. The school reform officer, who is to be hired by the state, could also appoint a CEO to take control of multiple public schools if it is advantageous to achieving better educational results. 8     An alternative process for teacher certification, in which a person could receive an interim teaching certificate if they are participating in a state superintendent-approved alternative teaching program, hold degrees that meet required grade points, and pass examinations for the pertinent subject area. Allowing additional charter schools and two K-12 “cyber schools” that would provide full-time online instruction. Raising the dropout age from 16 to 18, effective for the high school class of 2016. Requiring an annual evaluation of teachers and administrators, in part using data on student growth (Lane, 2010). MICHIGAN TEACHER EVALUATION LAW PA 101, 102, and 103 of 2011 At a time of desperately tight state budgets and declining sales, income and property tax collections, Michigan’s lawmakers recognized that they needed to pass laws in order to have a chance to receive RTT funding. Legislators, under Governor Granholm, passed the following laws: Table 1: Michigan Laws Passed in Response to Race to the Top Date 1/4/2010 Public Act # PA 201 (of 2009) PA 202 (of 2009) PA 203 (of 2009) 1/4/2010 1/4/2010 1/4/2010 PA 204 (of 2009) 5/19/2010 PA 75 Description Forces bids from union on outsourced services Allows alternative certification routes Provides funding for "schools of excellence" as public school academies, creates a teacher identifier system for collection of data, establishes a system to ensure basic instructional supplies, clarifies pupil instruction hours for certain schools, and appropriates certain federal funds Legislates take-over of low-performing schools, raises mandatory age for dropouts to 18 (for the class of 2016), and expands personal curriculum for high school students. Sets retirement incentive; mandates employees who stay have to pay 3% toward retirement; sets a hybrid system for new employees Michigan’s focus for the second round of funding then was on collaboration and legislative action. Those applications were due on June 1, 2010 and included PA’s 201 through 204, listed above. The only major change, according to Boyd (2010,) in the formula and requirements for round 2 of Race to the Top (RTTT) funding, was a tightening of budgetary suggestions into requirements (based on the size of the state). At the same time as Round 2 applications were organized and filed, Governor Granholm, a Democrat, was term-limited out, so little was done, legislatively, leading up to the election. The state’s department of education, however, still made an effort to meet the requirements of the second round and gain the funding. 9 Most of those efforts focused on collaboration. According to a press release from the MDE (May 2010), announcing the Round 2 application, more collaboration was achieved by procuring Memoranda of Understanding from many groups. The groups included 87% of local school districts, with 737 traditional districts and public school academies, to participate and be eligible for funding, if the state was chosen. Furthermore, letters of support were attached from over 80 organizations, including the Michigan Association of School Boards; Michigan Association of School Administrators; Michigan Education Association; American Federation of Teachers-Michigan; Michigan Association of Public School Academies; Michigan Association of Secondary School Principals; Detroit Regional Chamber of Commerce; Ford Motor Company; Michigan ParentTeacher Student Association; post-secondary universities and community colleges; and Early Childhood Investment Corporation (MDE Memo 2010). Mike Flanagan, in a summary of the application, titled “Accelerate Michigan”, said this: “We all have real opportunity here and now, to show the nation that Michigan is leading the way in having innovative schools that develop the highest quality graduates in the world. “ Despite the rhetoric, collaboration, and legislation, Michigan’s effort to “win” funds fell short. In fact, according to the U. S. Department of Education’s press release from August 24, 2010, Michigan fell from 21 st place to 23rd place. An independent, non-partisan analysis (from New America’s Jennifer Cohen Kabaker) of the Phase 2 results indicates that the major area of improvement and failure of all states was in the area of “Great Teachers and Leaders.” Other improvements apparently were not enough, and Michigan leaders learned they would not be eligible for Phase 3 of Race to the Top, as only finalists from Phase 2 would be eligible. Winners of Phase 2 included Massachusetts, New York, Hawaii, Florida, Rhode Island, Maryland, Georgia, North Carolina, and Ohio. Those eligible for Phase 3 were Arizona, Colorado, Illinois, Kentucky, Louisiana, New Jersey, and Pennsylvania. A short while later, in November of 2010, Rick Snyder, a Republican, was elected governor of Michigan. Under Governor Granholm, Michigan failed to reach the upper echelon of school reform, partially because of a lack of clarity and legislative action, evaluating great teachers and principals, but also because of inadequate data systems. Governor Snyder was inaugurated in January 2011. He espoused the goal to improve two of those areas: evaluating great teachers and inadequate data systems. Snyder also knew the Michigan economy was very limited. So, he proposed and mandated changes that improved some of the areas identified by the U. S. Department of Education as lacking in Michigan’s educational system. While doing this, he ensured that districts had “financial freedom” to enact changes, and he reduced overall employee compensation to allow districts more freedom. 10 Below is a chart of Snyder’s legislative efforts in the area of teacher evaluation listed beside possible problems the laws were meant to address. Table 2: Changes to Michigan’s Teacher Evaluation Law with Possible Problems It Meant To Address Public Act Number PA 100 PA 101 Portion of Legislation Allows unpaid suspension of 20-40 days; removes "reasonable and just cause" as the standard for discharge; holds that two evaluations of "ineffective" is proof of ineffective teaching; stops pay for tenure teachers 90 days after an appeal is filed. Ties completion of probation to evaluations that emphasize student achievement; prohibits bargaining over evaluation. PA 101 Allows taking away tenure and a new probation tied to student achievement PA 101 Adds six more subjects that would be prohibited from collective bargaining, including (1) placement of teachers; (2) personnel decisions when conducting a reduction in force, a recall, or when hiring; (3) performance evaluation systems; (4) the discharge or discipline of employees; (5) the format or number of classroom observations conducted during performance evaluations; and (6) the method of performance-based compensation. Amends the school code; prohibits bargaining over evaluation and lay-off standards. PA 102 PA 102 Amends school code to mandate the following factors in teacher evaluation: significant, relevant accomplishments and contributions, student growth, and relevant special training 11 What possible problem could it mean to address? Tenure overprotects teachers from the job action based on ineffectiveness Completion of probationary job term is too easy. Teachers are not required to show they are effective and improve student achievement. Tenured teachers are too protected and not allowed to be placed on probation if they do not improve student achievement Teacher’s job security is too negotiable through collective bargaining Teacher’s job security is too negotiable through collective bargaining Teachers are not increasing student growth, obtaining training outside the school day/year, and not making significant contributions to achieve seniority and increases in pay Table 2 (cont’d) Public Act Number Portion of Legislation PA 102 Prohibits seniority as a factor in assignments and makes "effectiveness" (rather than seniority) the determining factor when a workforce reduction is necessary. Requires each school district's performance evaluation system to rate teachers as "highly effective," "effective," "minimally effective," or "ineffective." Revises tenure hearing procedures. PA 102 PA 102 PA 102 Establishes the permissible grounds for the discharge or demotion of teachers on continuing tenure. PA 102 Limits the length of time a teacher's salary continues during a suspension. PA 102 Establishes and requires a mutual consent policy for teacher placement. Amends PERA to make the following prohibited subjects: policies and standards for RIF; decisions, content and standards for performance evaluation; decisions, standards and procedures regarding discipline or discharge of any employees; decisions and standard on how merit pay is awarded; decisions about classroom observations; placement in innovative programs under Sec. 1247 of the school code. PA 103 What possible problem could it mean to address? Teacher’s job placement is too tied to seniority and not performance District’s ratings systems vary too much Teacher tenure is too strong and protects ineffective teachers Teacher tenure is too strong and protects ineffective teachers Teachers get paid when they are suspended for performing poorly and are suspended Teachers have too much power in their job placement Teacher evaluation is too limited by collective bargaining The changes to evaluation law have not been challenged directly by employees or the MEA. One possible explanation is that districts, while threatening to “use” the rankings and annual evaluations against teachers, are not doing so. In a recent Ed Trust paper, Sarah Lenoff (2012) surveyed ten large districts in Michigan, finding that 99.4% of the teachers were judged to be “effective” or “highly effective” in 2010-2011. Whether districts’ teacher evaluations included a higher proportion of less favorable ratings in subsequent years is thus far unknown. Legal opinions on Michigan’s teacher evaluation law are limited. That does not necessarily mean that the law will not be challenged in the future. Clearly new legal rulings regarding the laws could modify district evaluation procedures, but such litigation has not yet surfaced. Michigan has been moving toward centralized control of its public schools over the last twenty years. Therefore, the state’s teacher evaluation law is not unexpected. The law, however, definitely complicated districts’ teacher evaluation procedures. Initially, the state provided very little information on how districts should implement the new law. With no tool or process and a lack of useful state-wide data (MEAP and ACT scores are not well-designed to measure teacher growth), the 12 implementation becomes even more important. Even if the tools and process had been defined, an evaluation system should motivate teachers clearly to do what is best for students. To understand how evaluations can do this, one must understand employee and teacher motivation. TEACHERS: A PROFILE OF THE PEOPLE AND THEIR MOTIVATION Michigan’s new teacher evaluation law seeks to motivate teachers to do their best for students by rewarding them for this effort. In order to know what might motivate teachers, it is useful to consider the profile of teachers. The following section will detail who teachers are and their overall perceptions of work. It will then delve into the broader topic of employee motivation in general, before discussing teacher motivation. The description of public school teachers that follows will cover the following areas: demographics, mobility, job perceptions. It relies on national data, which are similar to those pertaining to Michigan. The national numbers are not disaggregated in more recent surveys of teacher job satisfaction. The Met Life National Survey on Teachers last disaggregated by state in 1995. At that point, Michigan’s numbers were very similar to national numbers in major demographic areas. The national profile is generally stable over the last 30 years, except that teachers on average have gotten younger, as baby boomers retired. Education Week sponsored a review of teachers’ job satisfaction, and their reactions to current reforms. Its results are revealing and somewhat surprising for those who speculated that changes in education negatively affect teacher’s job engagement. The author, C. Emily Feistritzer (2011), surveyed 2,500 teachers, of which 1,076 responded. The survey tracked teachers’ age, years of experience, grade levels taught, and school setting (city, rural, suburban). One way to measure a teacher’s job satisfaction and engagement is to ask, what will you be doing in five years? Feistritzer (2011) founds that 89% of her respondents were “somewhat satisfied” or “very satisfied” with their teaching jobs in 2011, compared to 83% in 2005. Only 11% were “somewhat dissatisfied” or “very dissatisfied” in 2011. Similarly, 86% were “somewhat satisfied” or “very satisfied” with their working conditions in 2011, compared to 81% in 2005. In the area of “tests of student achievement,” 62% of respondents were “somewhat satisfied” or “very satisfied”, compared with 44% in 2005. One can interpret these results in several ways, but it seems that teachers are adapting to the changes NCLB brought and are more satisfied with their teaching jobs, their working conditions, and the achievement of their students on standardized tests. The satisfaction teachers feel about their jobs and working conditions is positive, according to the survey. Opponents of educational reform may find these results surprising, as it is sometimes purported that teachers are losing motivation and 13 engagement in their careers. Even in the area of student achievement tests, the satisfaction improved by 18 percentage points over six years. Feistritzer’s results suggest that teachers are generally satisfied with their schools and work lives. Sharon Conley and Naftaly S. Glasman (2008) write extensively about teachers’ reactions to experience toward evaluations. They highlight teachers’ fears. Their findings, a meta-analysis of past research on fear and performance, show background for why teachers may fear evaluations in a general sense. Their first fear is the “conflicting priorities” of their work (Educational Research Service, 2000). A second fear is the “boundedness” of teachers’ work. There is always something to be improved, and teachers feel there is not enough time to improve all the areas that need to be addressed. Pressures, from inside and outside the school, cause teachers fear as well. It seems easy to see why state/district take-over, which can include for a teacher losing one’s job, one’s teaching assignment, one’s colleagues, and one’s principal, is intimidating to teachers. Teachers are looking for how to best do their job: help students learn. They want to know what they need to do and how to do it. Should they teach to get students ready for standardized tests? Should they give more homework and grade more papers? In their pursuit of their own job excellence, should they reduce the amount of time they spend helping individual students, in exchange for doing what is best for the greatest number, to improve test scores for their whole class? Districts are charged with both developing effective teacher behavior, while evaluating the teachers for rankings and lay-offs. Negotiation on these issues is now illegal under state law. If a district is to follow the law, these various motivations, questions, and behaviors must be identified and scrutinized without formal discussion with the teachers and their representatives. With so many challenges and uncertain answers to the above questions and motivations, performance appraisal systems for teachers at least need to be implemented effectively and have as their goal acceptability from teachers. Research from business and human resources provide a framework to, at a minimum, define the implementation and goal for the system, if a district can determine the behaviors they want to emphasize and evaluate with teachers. The following sections on implementation of performance evaluation will cover what is known about performance appraisal and the implementation of performance appraisal systems from a business human resources field for insights that it may provide for school operations. Performance appraisal in business has been extensively discussed. Hundreds of books have been written on the subject. After surveying this literature, I will consider its relevance for teacher evaluation. BUSINESS SECTOR/HUMAN RESOURCE AND PERFORMANCE APPRAISAL RESEARCH Research on performance appraisal in business differs from that in education. Most of the literature doesn’t recommend one best practice way to evaluate employee performance. One surety in all the work is the centrality of the job 14 analysis and description. There are several methods for job analysis, including task analysis inventory, which identifies the different tasks performed on a job. This can be accomplished using the CODAP method (Primoff, 1975), an automated system that organizes tasks into dimensions. There are other examples, such as the Fleishman Job Analysis System. All methods of job analysis are complicated and nuanced. Interestingly, job analysis and the conversation about what teachers’ jobs are have never happened explicitly or transparently at the state level before PA 101, 102, and 103. With such a focus on “business” practices on the financial side of districts, the most basic of business practices in evaluating job performance was skipped. Job design, fit, and placement are protocols emphasized in the human resources field (Ilgen and Hollenbeck, 1991). Most analysis of jobs in business assumes that the worker is the right fit for the job. If s/he is not, the employee would be moved. None of these job analysis areas is a part of Michigan’s teacher evaluation law, but all are highly integrated in the area of business and human resources (Brocka and Brocka, 1992). There is no perfect solution to performance evaluation. Research indicates that there are different approaches to it based on the purpose of the organization, the time available to accomplish it, and the capacity of the rater. The effectiveness of performance appraisal is determined by several variables which impact the acceptability to the employees being evaluated (Noe, 2006). Relevance - the extent to which a performance measure assesses all of the relevant, and only the relevant, aspects of performance. Relevance is concerned with the overlap between actual job performance and the measure of job performance. To be relevant, the performance appraisal cannot include irrelevant aspects of job performance(Noe, 2006). It also must include all relevant aspects of performance, leaving none out of the appraisal (p. 338). Reliability - the consistency of the performance measure. This includes inter-rater reliability, or the consistency among individuals who evaluate the employee’s performance. Evidence indicates that most subjective supervisory measures of job performance exhibit low reliability. Reliability has to measure performance consistently across time. Noe (2006) indicates the use of evidence as essential for evaluators to maintain reliability (p. 339). Specificity - the extent to which a performance measure tells employees what is expected of them and how they can meet those expectations. Employees understand expectations and also know how to improve their own performance (Noe, 2006). The employee must understand the terminology, sequence, and communication of the evaluation. 15 Strategic Congruence - the extent to which a performance management system elicits job performance that is congruent with the organization’s strategy, goals, and culture (Schuler and Jackson, 1987). To achieve strategic congruence, the organization and the performance appraisal system must be flexible enough to change as the vision and goals of the organization change. Strategic congruence also should attempt to translate the organization’s goals into the job description and performance of each employee. Acceptability - whether the people who use the performance measure accept it. Noe (2006) explains that many elaborate performance measures are valid and reliable, but they take so much time for the managers to complete that the managers do not use them. Gillard and Langdon (1998) address their acceptability by employees. Nathan, Mohrman, and Millman (1997) note that performance measurement systems that are perceived to be unfair by employees are likely to be legally challenged, to be used incorrectly, and to decrease employee motivation. Acceptability is likely to be conditional on the evaluation system’s validity, reliability, specificity, and strategic congruence. These variables of measurement combine and work together (or against one another) in any performance appraisal system. Almost all performance evaluation systems are complex and their execution is complicated. In order to satisfy all of these variables of effectivness, an organization must implement its evaluation system with great care. This research study will use Noe’s (2006) criteria to assess districts’ employee evaluation implementation and its impact on employee motivation. Acceptability matters greatly for motivating employees and their managers. If employees do not find the performance appraisal system acceptable, then they will not improve or even maintain their current performance. Performance appraisals must also be used for job-based decisions. If those decisions are not acceptable to those in the organization, there will be much contention leading to wasted time and energy. This variable of acceptability will be central to the study. IMPLEMENTATION OF PERFORMANCE APPRAISAL SYSTEMS Organizational leaders in both business and education often overlook implementation. If all performance evaluation systems have positive and negative attributes, then the implementation of any system will allow stakeholders to facilitate transparency, openness, and buy-in, with the hope of motivating employees more effectively. In the sections that follow, implementation of performance appraisal systems will be explored and discussed, merging two major works on the subject into a coherent set of guidelines for implementing a system. The main areas of implementation include implementation teams, pilots, communication, training, and evaluation of the system. 16 Dick Grote (1996), in his book The Complete Guide to Performance Appraisal, writes extensively about the development and implementation of a performance appraisal systems. Grote builds upon Thomas and Bretz (1994) and other work on performance appraisal from the 1990’s. Grote also cites his consulting worg with numerous companies as the basis for his ideas. The empirical basis for Grote’s framework, therefore, comes from his personal experience and a meta-analysis of previous writers on performance appraisal. Though his ideas have a limited empirical basis, they do offer an attractive conceptual framing of key elemenets of performance appraisal implementation Grote does not make claims about which of the implementation components are most important, despite the fact that some may be more important than others. In Grote’s analysis, effective implementation is conceived as a step-by-step sequence of actions. For the purposes of this study, they serve as framework to describe district’s steps to implement their system. To begin, Grote (1996) recommends implementation teams comprised of stakeholders from all levels of the organization, especially from both bottom and top levels of the employee structure. Mohrman, Resnick-West, and Lawler (1989) contribute the most to what the implementation team must do. The team should do empirical research on needs assessment, evaluation tools, pilots, communication, training, evaluation, and adaptation/adjustment of appraisal system. They also note the importance of top-management buy-in and accountability for managers and employees. Each component of implementation is discussed below, with a concentration on ideas most applicable to school districts. Grote compares an implementation team to a task force formed to implement a new product at a company. All divisions and departments are involved, especially marketing. Marketing is key because the success or failure of the product (or performance appraisal system) is determined by the way consumers (or employees) perceive and accept the product (or process of appraisal). Surveying stakeholders on the current system and their hopes for future systems is Grote’s first step, to determine what each level of the organization needs and expects from it. With this “baseline” data collected, the implementation team should “build organization-wide understanding, support, and acceptance among system users” (p. 253). This could be termed a “needs assessment” and it is carried out by the implementation team. A pilot phase, according to Grote (1996) should involve one or two selected departments or units, so kinks can be worked out before the system is taken to the whole organization. Grote (1996) says “the purpose of the pilot test is to gain 17 information from users on how the system works in practice” (p. 225). After the pilot, the recommendations of all users should be integrated to improve the system and the fit for the organization (Grote, 1996). The pilot-tested system and the actual system, according to Mohrman, Resnick-West, and Lawler (1989), must have a clear communication plan. They state the plan should answer the following questions:  What is the purpose of the communication?  What is to be said?  Who is to do the communicating?  When is it to be said, and through what channels?  Who is the desired audience?  How will the results of the communication be measured? (p. 133) In considering these questions, the implementation team must overcome traditional barriers to communication. As Mohrman, Resnick-West, and Lawler (1989) see them, these barriers are “selective exposure, selective perception, and selective retention” (p. 133). Selective exposure is the general propensity of all people to expose their minds to ideas/concepts/content with which they already know or have seen. Selective perception is the way people perceive information in the way they understand and like it. Selective retention refers to individuals only remembering things with which they previously agreed (Mohrman, Resnick-West, and Lawler, 1989, p. 133). Mohrman, Resnick-West, and Lawler (1989) identify steps to overcome these barriers. They include participation (the more people involved, the more accepting those people will be, the more the organization will understand the message/s), reward (people understand how it will benefit them), first is best (people will be able to form beliefs if they do not have beliefs in existence), facts don’t speak for themselves (people need more than facts; they need rationale and explanation of those facts), reinforcement (people understand communication best when it is reinforced through several channels and is consistent), print-credibility (print is more powerful/lasting then spoken word), and credible source (people must trust the source of communication). Mohrman, Resnick-West, and Lawler laid out ambitious and thoughtful goals and objectives for communication. Even if all of these barriers are overcome, however, the proof of the communication will be in the actualization and evaluation of the system. The actualized is defined by the training given to all levels of an organization. 18 According to Mohrman, Resnick-West, and Lawler (1989), training should encompass all members of the organization. This includes top management (oversight and usage), managers (usage), and grass-roots employees (informational). The training needs to be completed fully before the system is implemented. The implementation team is charged with defining and ensuring participation at all levels. As a corollary, it is interesting to note that Thomas and Bretz (1994) found only 22% of Fortune 500 sample managers were evaluated on how they conduct performance appraisals. Training, then, must be linked to evaluation of managers who evaluate (Mohrman, Resnick-West, and Lawler, 1989). As a part of the implementation process, Grote (1996) strongly recommends an appeal process. Michigan’s PA 101, 102, and 103 of 2011 do allow for an appeal, up to the superintendent. Grote (1996) emphasizes the appeal process as a part of an effective system: “one of the most accurate indicators of an ineffective performance appraisal system is the absence of complaints about it” (p. 264). Therefore, upper management implementing a new performance appraisal system should expect and welcome appeals of appraisals. In fact, Mohrman, Resnick-West, and Lawler (1989) state that “since 80% of the individuals entering appraisal situations feel that they have been performing above average, most people will receive information they do not want to hear” (p. 149). The appeal process is a part of the legislation, so it is not included in the study because its existence is mandated. Both natural feelings of disagreement about the new system and a need to organizations to improve continuously lead to the evaluation of the system and the implementation. Mohrman, Resnick-West, and Lawler (1989) suggest the use of feedback surveys. Ideally, a baseline survey would be taken before the new system is implemented (as suggested above). A pre-survey or needs assessment would allow the organization to take stock in the effect of its system. The mechanics of the system are important, but also the system’s overall impact on culture and climate Evaluation process and tools should be malleable and adjusted (as needed), according to Grote and Mohrman, Resnick-West, and Lawler), to improve consistency and validity. This evaluation and adaption of the system improves the system for managers and employees. There are bound to be issues in the evaluation system that need to be tailored to the individual needs of the organization, according to Grote (p. 262). That additional work needs to be undertaken by the organization to ensure the evaluation system’s acceptability to both employees and managers. Grote maintains that buy-in from top management is essential (p. 256). Regular meetings with a representative of top management enable the implementation team to win top management’s buy-in, but also gain feedback on the process as a whole. If the new performance appraisal system is a priority for the top executives of a company, they are more likely to communicate this importance to their employees (supervisors), who will make it an integral part of their jobs. If top 19 management is evaluating managers on the way the managers appraise employee job performance, then the managers likely will give their appraisals more careful and thorough consideration. The other area that may be outside of the implementation team’s purview is the accountability of those practically implementing employee evaluation. According to Grote (1996), only 22% of sample managers in Fortune 500 companies are evaluated on their performance appraisals of their employees. Top management’s buy-in should encompass evaluating their direct reports on their ability to appraise employees in an acceptable, reliable, and valid way. Grote emphasizes the importance of implementation teams, needs assessments, tools, pilots, communication, training, accountability, management buy-in, evaluation, and adaptation throughout his work on implementing a performance appraisal system (p. 252-279). Those components are listed below in Table 3. Table 3: Grote’s Components of Performance Appraisal Implementation Implementation Component Description of Implementation Component Implementation Team Involves members from all levels of organization. Their task is to implement the performance appraisal system completely. An instrument designed to define the strengths of the current performance appraisal system and identify needs for the new system. Also serves a baseline against which to measure new system after pilot and initial implementation. Top management is involved in the implementation and holds managers accountable for their employee performance appraisals. The development or selection of a tool and methods to be used in the new performance appraisal system. A trial of the new evaluation system with one department or group of employees in order to provide feedback and improve system before wide implementation begins. A plan for communicating, first to the pilot, and then to the organization the goals, terminology, methods, and outcomes of the new evaluation system Training all members of organization, including top management (oversight and usage), managers (usage), and nonmanagerial employees (informational) before system is implemented. A system to hold managers and employees accountable for the results of the performance appraisal system. Ongoing feedback from managers and employees on system to gain consensus on ways to improve system continuously. Ongoing changes made to the system to improve it. Needs Assessment Top Management Buy-In Selection/development of tool and methods Pilot Communication Plan Training Accountability for Managers and Employees Evaluation of System Adaptation/adjustment of appraisal system Each of the implementation components identified by Grote are likely to relate to acceptability. Top management must be prepared to defend the decisions. They also want the appraisals to move the company forward. Motivating employees through performance appraisal assumes that the employees will accept the managers’ accounts of their 20 performance and continue to perform well, or change their performance to fit what their managers identify as growth areas. Managers need to find the system acceptable, too, because they have to convince employees to use the evaluation to improve their performance. Pilots, communication, training, management buy-in, evaluation of the system, and adaption of the system all center on the employees and their perceptions of the system. While much writing on performance appraisals deals with the ideal, implementation is key. For the optimal motivation of all employees, there must be attention to how the system is developed, implemented, and evaluated and revised. No matter how well or how badly an organization appraises the performance of its employees, the process is complex and nuanced, as is clear from the flaws in all systems, from the complexities in the best systems, and from the ongoing discussion of the subject. The next section deals with how that complexity is multiplied when teaching and learning are the focus of perf0rmance appraisal. TEACHER PERFORMANCE APPRAISAL RESEARCH Evaluating teacher performance is a topic of discussion throughout the world. Nations from Europe to Asia have made attempts throughout the last thirty years to define how best to appraise the performance of their teachers. There are several challenges. They include student motivation, which can be affected by many factors outside the school; the changing and sometimes conflicting nature of what is expected by teachers from society; the resultant varying expectations of students and their performance; and a lack of administrative time and skill both to run a school and spend adequate time truly appraising teacher performance. Much of the research is on the nuts and bolts of the systems and the important aspects of teacher behavior. This section reviews research on teacher evaluations. Medley, Coker, and Soar (1984), in their book Measurement-Based Evaluation of Teacher Performance, address many of the issues discussed in educational reform circles today. The authors survey and synthesize available empirical research on teacher evaluation systems available at the time, and they offer helpful examples of observation and evaluation tools and procedures Chapters include inadequacy of existing models, evaluating teachers by classroom behavior, defining dimensions of teaching, structured observation systems, defining tasks to be performed, obtaining the record, and scoring the record. Although some of the examples and expectations are dated, newer works echo much of what Medley, Coker, and Soar (1984) review. Cohen’s chapter in Teacher Assessment and the Quest for Teacher Quality (2010) explains much of why American educational leaders have it so difficult. There is a lack of standards for students. There is a lack of standards for teachers. 21 Cohen doesn’t describe a lack of training or proper implementation of teacher evaluations because he shows that the proper infrastructure is not in place. Mary Kennedy (2010), editor of the volume, reiterates this point, from a different perspective. She reviews the variance in teacher evaluations through the years. She also shows how conflicting the standards are for teachers. Teachers are to be flexible and rigorous. They are to be strict in discipline, but also capable of forming productive relationships with students. Kennedy says that, given even the most ideal outlook on teacher evaluations, they still lack a focus on intellectual rigor and curiosity that is so essential to student success. DISCUSSION OF PERFORMANCE APPRAISAL RESEARCH Good performance appraisal is challenging in business and perhaps even more so, in education. Michigan’s teacher evaluation law (PA 101, 102, and 103 of 2011) forced districts to respond to this sticky issue with haste, but provided no meaningful guidance. Districts, without much knowledge outside of their own experiences, were required to choose or develop and implement a new performance appraisal system in less than a year. The steps they followed were devised by employees with little knowledge about how to implement a performance appraisal system. Despite these daunting circumstances, all districts complied and came up with systems. They took steps, as best they knew how, to implement their systems well. Their efforts to do so are important signs of what they value and how they function as organizations. The performance appraisal implementation can also be seen as a pilot for future state mandates. These initial district responses are worth studying in order to improve evaluation implementation in the years ahead. Grote’s (1996) performance appraisal implementation components can be used to structure districts’ narratives about their work on the state law. His components form an ideal and thought-out approach to implementation. Noe’s work (2006) on evaluations and performance appraisal gives this research a vantage point from which to view districts’ implementation. SYNTHESIS OF LITERATURE Districts had to implement Michigan’s eacher evaluation law quickly. Districts around the state had until their labor agreement expired to develop and implement a new performance appraisal system. The state did not provide a tool or implementation guidance, so districts had to figure it ouy on their own. From the above literature review of teachers and their motivations, it seems teachers have varying motivations. As a group, they are conscientious and achievement-oriented. They are scared. They feel their work is never done. Many profess 22 entering the professional to help students. They generally want to reform education and prepare students for college and work. They are willing to work hard to improve standardized test scores. They even want to rid the profession of unmotivated peers. Districts had to keep in mind all of these factors, while being mindful of their own specific contexts. The history and context of Michigan’s educational system over the last twenty years mixed with the current financial realities of a deep recession. The result in the county studied and throughout the state were varying levels of implementation coherence, based on Grote’s framework of performance appraisal implementation. From those responses, perceptions of teachers and administrators were formed about the teacher evaluation system in each district. These perceptions, in terms of motivation, can be chiefly measured against Noe’s attributes of effective evaluations. By using these business/human resources approaches, the study can describe the relationship between implementation coherence and teacher/administration perceptions in each district. In the districts studied, there was little variance between districts in implementation steps. There also was some variance in teacher perception. Overall, the clear narrative that emerged from the study is how many teachers negatively perceived the teacher evaluation sytsem and its implementation and how that relates to the implementation steps the districts followed and communicated. 23 CHAPTER 3: RESEARCH DESIGN AND METHODS The research questions are as follows: 1. How did districts implement Michigan’s 2011 Teacher Evaluation Law?  To what extent did it follow the principles of effective evaluation system implementation from the field of human resources? 2. How do district administrators and teachers perceive the new teacher evaluation system’s acceptability?  How do these perceptions relate to a district’s evaluation system implementation? The independent variable in the study is the extent of congruence between school district practices and Grote’s framework of evaluation system implementation. The dependent variable is teacher and administrator perceptions of the evaluation system. The study relies primarily on Grote’s framework for effective evaluation system implementation. The coherence of a district’s implementation will be assessed by the extent to which its implementation incorporates the elements of Grote’s framework. Coherence also will be measured by the manner in which districts used the steps of Grote’s framework. Grote’s components are listed below (1996): 1. Formation of an implementation team (with members from all levels of the organization) 2. Top management buy-in 3. Needs assessment 4. Tool for selection/development of evaluation 5. Pilot 6. Communication plan 7. Training 8. Accountability for managers and employees 9. Evaluation of the appraisal system 10. Adaptation/adjustment of the appraisal system Teacher perceptions were organized by Noe’s (2006) attributes of effective performance appraisal systems: reliability, relevance, specificity, strategic congruence, and acceptability. The research also examines what participants regarded as the most important part of the implementation, what they believed should have been done (but was not), and the relationship between the implementation and teacher morale. 24 These research questions were answered by online surveys completed by teachers, administrators, and school board members; by interviews with a subset (from the first group of participants) of teachers and administrators; and by an analysis of district documents. To answer question 1, employees in key district roles were surveyed, and district documents were analyzed. The selection of these roles was guided by the goal of securing input from people who were well positioned to assess their district’s implementation with respect to the elements in Grote’s framework. These surveys were analyzed to determine the usage of Grote’s components. To answer question 2, there were separate interviews with teachers and administrators. In the surveys, participants also were asked about the acceptability of the district’s teacher evaluation system and the overall influence of the implementation on teacher morale in their building and in the district. The interviews with teachers were typically longer and had a wider range, investigating implementation, perception of implementation, and overall perception of the teacher evaluation system, using Noe’s attributes of acceptable evaluations as a basis. The interviews with administrators were generally shorter, filling in the gaps of description on implementation coherence and briefly describing teacher perception of the district’s evaluation system. The district documents were collected from district-level actors in the teacher evaluation system implementation. Below, in Table 4, are the phases of research. 25 Table 4: Phases of Research Phases HS office provision of emails/phone numbers One A Description Participants 1 building level administrator Method Notes HS Principals Admin survey 1 central, 1 building administrator, 1 building union representative Online survey Teacher survey 2 core teachers, 1 non–core teacher Online survey Central office administrators were superintendents, assistant superintendents, executive directors, or building administrators with district responsibilities. All participants were in the district during the implementation Randomly Selected One B Document analysis District documents Narrative analysis of documents One C Admin interview Personal interview Two Teacher interview 1 building administrator 1 union rep teacher (core subject area teacher or non-core subject area teacher), 1 teacher (core subject area teacher or non-core subject area teacher), 26 If the union representative was a core subject area teacher, the other teacher was in a non-core subject area. If the union representative was a non-core subject area teacher, then the other teacher was a core subject area teacher. PARTICIPANT DISTRICTS The participating districts were from a medium sized county in southeastern Michigan. Of the nine districts in the county, six chose to take part in the study. Each district was designated by a number. Table 5 displays some basic demographic information about the districts. Districts with total enrollments of 1,500 students or less are categorized as “small.” Districts with total enrollment between 1,501 and 3,000 students are categorized as “medium.” Districts with a total enrollment of more than 3,001 students are categorized as “large.” District type classifications from the National Center for Education Statistics are also displayed in Table 5 below. Table 5: Participating Districts by Size and Type District Number District Size (small, medium, or large) District Type (as defined by NCES) 1 Large Suburb: Large 2 Medium Town: Fringe 3 Large Suburb: Large 4 Medium Rural: Fringe 5 Large Rural: Fringe 6 Small Rural: Fringe The districts varied somewhat in size and community make-up. Further demographic information is not included in order to preserve districts’ and participants’ anonymity, in view of the sensitivity of the topic studied. Participants offered their honest perceptions of district action. Therefore, further identifiers are not included. The selected districts are different. All districts had either “focus” or “priority” schools during the 2012–2013 school year, as identified by the Michigan Department of Education. The socioeconomic backgrounds of the students, faculty, and administration vary from one another as well. They represent a fairly wide range of communities, with distinct histories and perspectives. PHASE ONE (A, B, C): IMPLEMENTATION COHERENCE The goal of this phase of research is to gain teachers ‘ and administrators’ descriptions of the implementation of their district’s teacher evaluation system as well as their perceptions of its acceptability and impact on employee morale. It is accomplished in three ways: electronic surveys (A), analysis of district documents (B), and interviews with building-level 27 administrators (C). Each is detailed below. The multiple data sources will facilitate triangulation of the findings during the data analysis. Phase One A: Electronic Surveys Electronic surveys were emailed to six employees in each district. These included: o two core area high school teachers with more than five years’ experience o one high school elective teacher with more than five years’ experience o one high school building union representative o one district administrator with more than three years’ experience The survey questions for Phase One A are listed in Appendix A. Sample Selection for Phase One A Eight districts in the county were approached to take part in the study. There are nine districts in the county, and one was exempted. The exempted district was the researcher’s home district. Using this district would not allow the district personnel to participate honestly without the fear of adverse consequences based on their answers. Two districts did not agree to take part in the survey. I obtained informed consent from each district superintendent. In all participating districts, there is a single traditional high school. To obtain names and e-mail addresses of building-level employees, I worked with the high schools to obtain lists of core subject and non–core subject teachers with more than five years’ experience. From those lists, participants were chosen randomly. Random selection took place by assigning a number to each core and non–core subject area teacher. The one with the highest number was the first teacher approached to take part in the research. When choosing the teacher to take part in the interviews, I used the highest number of the two teachers selected to take part in the interview. Potential participants were sent at least three e-mails, soliciting their participation. If they either did not respond or responded negatively, I moved on to the next teacher numerically. I used the same random selection process to choose a union representative. 28 All of the central office and building administrators were approached through email and a follow-up phone call. The building principal and central office administrator in each sample district agreed to participate once emailed. At the central office level, the participants were superintendents, assistant superintendents, executive directors, or building administrators with central office responsibilities. Rationale for Phase One A The rationale for the six survey participants was that they represent key actors in the district’s implementation of the teacher evaluation system. A central office administrator and building-level administrator should be familiar with the district’s overall implementation plans and actions. A building-level administrator and union building representative should have key insights regarding the actual implementation at the ground level. Presumably, teachers would also generally know about the implementation process but also have the key perspective of understanding the perceptions of those being evaluated. Given their formal duties, union representatives had a different vantage point compared with other teachers, and teacher unions play key roles in district employment relations. The minimum of five years’ experience ensures that participants was employed by their current districts before the passage of Michigan’s 2009 Teacher Evaluation Law. A teacher with less experience would not have had the opportunity to experience teacher evaluations prior to the Race to the Top legislation and Michigan’s legislative response. A second rationale for this decision is that teacher tenure was granted after four years for teachers hired in the 2011–2012 school year. Participants in the study would likely have a tenured status. So they would have greater job protection than those with less than five years’ experience. They would feel more comfortable in their responses with a minimum of one year of teacher tenure protection. I used high schools as the site for all building-level employees to participate for several reasons. First, as a practicing high school principal, I know all of the high school principals in the county. I spent time with them in meetings, conferences, and professional development opportunities at the county’s intermediate school district. I also know many teachers in the county, so I am known to them. High schools are also my area of expertise. I have worked in high schools for fifteen years. I am familiar with the vocabulary, dispositions, and modes of operation for high schools. Therefore, I used my professional knowledge to relate and to understand the participants. 29 The selection of core and non–core subject teachers was based on the different ways the Teacher Evaluation Law affects these teachers. The Michigan Merit Exam (MME) only assesses math, reading, social studies, English/writing, and science. Therefore, non–core subject teachers do not have a state-mandated test to assess their performance. Their perspectives on the implementation process, the teacher perceptions of the implementation, and overall perception of the evaluation system may vary based on their teaching assignments. The intention for the inclusion of core subject and non–core subject teachers then accounts for both groups: those directly affected by MME testing in their area and those not directly affected by MME testing. Special education teachers were exempted from the study. Special education issues are complicated in almost every area of teacher evaluation, including student growth, effective teaching techniques, and proving compliance to other special education state mandates. Therefore, the implementation would be hard to isolate because so many special education issues are unsolved and could skew the perceptions collected. Core subject area teachers were defined as those who teach the majority of their schedules in the areas of English language, arts, science, social studies, world language, and math. World language was included because for two years world language has been a part of the Michigan Merit Curriculum. Non-core subject area teachers were defined as those who taught a subject area outside of English language, arts, science, social studies, world language, and math. There is no state-mandated standardized test in these subject areas. The reason for the selection of administrators with more than three years’ experience in administrative roles is to capture the perceptions of those who were a part of the implementation of the new evaluation system. Those with less than three years’ experience would be unaware of the implementation process and would only be able to comment on the perceptions of the current system. An essential component of building-level administrator participation is their ability to detail and understand the implementation coherence of the new evaluation system. Therefore, only those with three years’ experience are included. In all six districts, the building administrator was the head principal, though some of the principals had different roles during the implementation of the teacher evaluation system. I chose to use one building, at the same grade level (high school), per district to reduce the potential influence of different school grade-level on building cultures. Focusing on high schools allows the relationships between implementation and perception to be explored under a single school-level leadership team in each district. It also abstracts from potential difference in the dispositions of elementary and middle school teachers towards teaching and student learning. 30 All Phase One A participants were informed that they could be asked to participate in the interview portion of the study. They had to consent to being in both parts of the study in order to participate. Participants chosen for Phase Two would be interviewed. Interviewees were chosen for Phase Two based on their having the highest randomly assigned number in the initial assignment. Data Collection for Phase One A Appendix A and Appendix B display the survey questions. The survey poses several closed-ended or multiple-choice questions designed to elicit information regarding specific aspects of districts’ implementation process. Several open-ended questions were designed to elicit participants’ perceptions of the acceptability of the teacher evaluation system and how its implementation influenced teacher morale. The initial participants were approached with an e-mail and a phone call. If the potential participant did not respond, a second and a third attempt were made to gain their consent and participation in the study. If a potential participant declined, the next participant in the random selection list was used. Common reasons for declining were a lack of time to participate (especially in the interview section, which was possible if participants completed the surveys) or what teachers called “personal reasons.” When the survey was completed, I e-mailed the participant an Amazon gift card and a “thank you” for their participation. The gift cards served as remuneration for his/her participation in the study. Financial support for this remuneration was provided by a small grant from Michigan State University’s Department of Educational Administration. Data Analysis for Phase One A The data analysis focused on two areas: implementation coherence (according to Grote’s framework) and participants’ perceptions of the evaluation system’s acceptability and its influence on employee morale. The multiple choice responses are fully summarized in the district-by-district description in Chapter 4. Any “other” responses in which participants described a district action outside of the multiple choice answers are fully described. The analysis established descriptions of each district’s evaluation implementation and its coherence as well as descriptions of participants’ general perceptions of those procedures. The survey also contained one general question about the communication about the new teacher evaluation system, and one question about how well teachers understood expectations. These questions will be used in comparison with Phase 31 Two interviews that pursue teacher and administrator perceptions more intensively. Each participant’s response is fully quoted within the district-by-district descriptions. Positional analysis also was used in Chapter 4. As one might imagine, district administrators and building union representatives had almost identical responses. The variation, in both interviews and online surveys, seemed to exist in the building administrator and teachers without official union positions answers. Respondents seemed to give the most detail and be the most critical on the online surveys. In fact, in two non-union position teachers, the teachers said they found the teacher evaluation system “acceptable” in the interviews, but “unacceptable” on the online surveys. The open-ended questions about morale and acceptability on the online surveys revealed many of the differences in both implementation and teacher perception the study tried to describe. Lining up response to the online survey questions side by side seemed to highlight the differences between districts. This need for this type of analysis was clear after examining each district’s data separately. Phase One B: Documents/Artifacts Phase One B of the research asked each of the six sample districts to submit agendas, minutes, communications, statements, or definitions of evaluation procedures as well as other related documents related to the district’s implementation of Michigan’s Teacher Evaluation Law from central office administration to building administrators. The documents were used to verify the steps of the implementation. The districts submitted as many as twenty documents and as few as three. A district administration with more than five years’ experience and involvement with the district’s implementation of the teacher evaluation system was identified to assist the researcher in the collection of the documents. In most cases, this was a superintendent, an assistant superintendent, executive director, or a building administrator with district-level responsibilities. Rationale for Phase One B The documents provided a text-based record of each district’s teacher evaluation implementation and allowed the researcher to triangulate findings with from the survey and interviews. Participants received a $25 Amazon gift card as remuneration. Support for this compensation was provided by a grant from Michigan State University’s College of Education. Data Collection for Phase One B 32 Districts were approached to identify one key player at the district level to obtain and send agendas, minutes, communications, and other documents related to the implementation to the researcher through e-mail. This request was made through phone call or a physical visit to the district by the researcher. Once the participant was identified, an e-mail was sent. If the participant did not respond to the e-mail, a phone call was made to the person. All of the district participants agreed to participate in the study. All of the documents were kept and analyzed electronically. Data Analysis for Phase One B The analysis of the documents was based on the verification of different steps of implementation coherence. If there were agendas, e-mails, or communications, submission of those written artifacts was noted on district-by-district analyses in Chapter 4. Phase One C: Interviews with Administrators Phase One C of the research used interviews with building-level administrators who were involved in the implementation of the district’s teacher evaluation system. Some of the administrators did not have three years’ experience in their particular districts or positions, so they had to start their descriptions when they came into the district or from the positions they occupied at the time. All of the administrators implemented the teacher evaluation system, but several were not present for the choice and planning of the implementation. The questions focused on the implementation, what could have been improved, what was important, general acceptability, and influence on teacher morale. These questions are listed in Appendix C. Sample Selection for Phase One C Interviewees consisted of the high school–level administrators who participated in Phase One A. These were all of the high school principals in the buildings. All districts would be included in Phase One C. Rationale for Phase One C The rationale for this phase of the research was to provide a building- /district-level view of the district’s implementation process. Building administrators were well situated to fill in possible description gaps with more detail about the components of implementation because they attend district-level and building-level meetings and discussions. 33 Their perspective was limited because they are not teachers and had to rely on what teachers communicated to them. Teachers may have been overly optimistic with administrators because the administrators evaluate them and ultimately decide on their employment futures. This was why they were not asked, component by component, about the perceptions of the evaluation system and implementation. All of the administrators interviewed were a part of the implementation. Several of them entered the district’s implementation with many early decisions already made. Administrators’ perspectives were useful to validate the descriptions of the components of district implementation and to detail areas that may be unclear or unknown to teachers. Participating administrators received a $25 Amazon gift card as remuneration for the participants’ time. Data Collection for Phase One C Administrators were sent a copy of the questions that they were going to be asked in the interviews in a reminder email sent before the interview. They were encouraged to read through the questions before the interview. The interviews took place in the administrator’s office. All of the interviews took place during a school day, and administrators were unable to get away to meet at a non-district location. The interview questions were brief and sent to the participants in advance of the interviews, as mentioned above. The total interview time should be between fifteen and forty minutes. The interviews were recorded and transcribed. Administrators who participate in this phase of the research received a $25 Amazon gift card as remuneration for the participants’ time. This money was provided through a small grant from Michigan State University’s Department of Educational Administration. Data Analysis for Phase One C Analysis of the interviews will be undertaken from a narrative analysis perspective. This approach is detailed briefly above, based on the work of Cortazzi (1993) and Reissman (1993). Responses about implementation coherence, important parts, areas that should have been included, influence on morale, and general acceptability of the system are clearly answered and provided triangulated data on the topics. 34 Since the questions about Noe’s attributes were of a direct variety, each answer is given to the direct questions (e.g. “Was your district’s teacher evaluation system acceptable to you?”) and then a quote that seemed to summarize the rationale for their answer is given in the next column. Beside the district by district comparison, I compared all of the districts to one another on each question I asked. The differences between districts became clearer as I laid out each district next to each other. I saw there were more positive teacher perceptions in Districts 1 and 2. I then re-analyzed all of the administrators’ descriptions of implementation. That is where I found some of the subtle differences between how the districts used Grote’s steps (1996). As administrators described how they communicated and approached the new system, the differences were able to be defined by my own summaries and administrators’ quotes on how districts implemented. Phase Two: Interviews with Teachers The goals for Phase Two of the research were twofold: first, to provide a check on the Phase One data on implementation coherence; second, to capture teacher and administration perceptions of the district’s teacher evaluation system and its acceptability, including validity, reliability, specificity, and strategic congruence. The questions are available in Appendix D. Sample Selection for Phase Two All districts were included in Phase Two. The districts were represented in Phase Two. Phase Two participants were a subset of the same as those in Phase One participants, including one building/area union representative/teacher and one core area high school teacher. If the union representative taught a core subject area, the other interview was with a teacher from a non–core subject area. If the union representative taught a non–core subject area, the teacher with the highest randomly assigned number was chosen to be a part of the interview portion of the study. The union representative for Phase One automatically participates in Phase Two. Phase Two participants was randomly selected from the Phase One A participants. The high school building administrator was automatically selected for Phase Two. The two core area high school teachers were placed on a Microsoft Excel spreadsheet and assigned a number. The teacher with the highest number was asked to participate in Phase Two. This occurred before Phase One A, so it was possible to inform participants of the level of their involvement in the study. 35 Participants in Phase Two will be sent a $30 Amazon gift card as remuneration for their time and participation in the study. This remuneration is provided by Michigan State University’s Department of Educational Administration in a small grant. Rationale for Phase Two Each participant in Phase Two represented a different level of teacher perception. The building/area union representative/teacher represented a building and district perspective because building/area representatives must be aware of what is happening across each district through union meetings and their own experience within the building and relationships across the district. Using both a core subject area and a non–core subject area teacher allowed both core subject and non–core subject area teachers to give their perceptions. Core area teachers were chosen because the state-mandated standardized test, the MME, used to measure student learning growth directly covers their content area. Student learning in the courses of non–core subject area teachers, meanwhile, is not tested by the state; therefore, their performance evaluation cannot include this key component. In addition, in times of tight district budgets, teachers of non–core subjects may be more vulnerable to program cuts and layoffs. Whether these factors are associated with differences in the perceptions of core and non–core subject teachers is of interest. The rationale for interviews, rather than online surveys, for this portion of the study was that describing the perceptions of teachers and relating them to the district’s implementation coherence can be personal and possibly stressful. Interviews allowed the participants to become comfortable with the interviewer and give honest responses to the questions. Once a date was set, teachers were sent a copy of the questions in a reminder e-mail. They were encouraged to read through the questions before the interview. The rationale for this practice was so that teachers will optimally consider the questions and their perceptions before the actual interview. Data Collection for Phase Two The interviews lasted 30 to 45 minutes and were held in the building where teachers worked, either after school or during their preparatory period for teachers with after-school commitments. The interview included three sets of questions. The first gave participants the opportunity to describe their district’s teacher evaluation implementation. The second set focused on their perceptions of the evaluation system implementation. The third set of questions focused on their views regarding the evaluation system’s acceptability, corresponding to Noe’s (2006) framework for effective evaluations. 36 The interviews were recorded by the interviewer and then transcribed. Participants were aware that the interviews would be taped and agreed to this method. Data Analysis for Phase Two The analysis of the interviews were analyzed using used a basic form of a narrative analysis perspective. Cortazzi (1993) emphasizes this form of analysis because it “can allow us to study teachers’ culture and thinking” (p. 1). Furthermore, Cortazzi says, “Narrative analysis can be seen as an opening to the mind, or, if we are analyzing narratives of a specific group of tellers, as opening a window of their culture” (p. 2). This “opening of the mind” about the implementation of teacher evaluation systems is the goal of the study. The interview questions were straightforward. For example, one question asked, “What was not included in the implementation of the system that should have been?” Another question asked, “Another way to assess performance evaluation systems is to measure if they encompass all of the relevant, and only the relevant, aspects of performance. Do teachers in your building feel that the current evaluation system (implemented in the 2011-2012 school year) is valid in assessing all the relevant and only the relevant aspects of performance? Why or why not?” All of the interviewees answered the questions directly. All of these answers are described in the district-by-district analysis in Chapter 4. I analyzed each text for quotes that seemed to describe why the participants answered in the way they did. In interviews, I often asked why? Teachers and administrators revealed their rationales by describing their experiences. As I moved into deeper analysis of the texts, I highlighted key answers, rationales, and quotes that described the teachers’ and administrators’ perceptions of their districts’ teacher evaluation systems and their perceptions of the system and its implementation. This analysis and synthesis is described in Chapter 4. As I analyzed districts individually, as well as side by side, I began to notice there were slight differences in how the districts approached communication and decisions about their implementation of the state law. I reviewed all of the transcripts and began to find subtle themes in teachers’ descriptions, especially around the topics of positionality to the mandate, communicated confidence, clarity of expectations and “highly effective” status, and the increased stakes of performance pay and teacher evaluation. A more complete description of this secondary analysis is available in Chapter 4, Section 5. CONFIDENTIALITY AND CONSENT Reliability was checked in the study in several key areas. First, the description of the district’s implementation coherence was addressed in both Phase One and Phase Two. Participants were asked to describe the steps of implementation 37 in both phases, in closed-ended and open-ended questions. This permitted triangulation of the findings--will provide an opportunity to checking for consistency between the surveys, the interviews, and the documents. It resulted in a wide-ranging view, rich in detail, of each district’s teacher evaluation implementation of its teacher evaluation system. INTERNAL RELIABILITY AND VALIDITY Reliability was checked in the study in several key areas. First, the description of the district’s implementation coherence was addressed in both Phase One and Phase Two. Participants were asked to describe the steps of implementation in both phases, in closed-ended and open-ended questions. This will provide an opportunity to check for consistency between the surveys, the interviews, and the documents. It will result in a wide-ranging view, rich in detail, of the district’s implementation of its teacher evaluation system. Internal validity was checked during both Phase One and Phase Two by comparing the online surveys to the interview results. The questions asked encouraged participants to describe the implementation process and their perceptions. All of the participants had an answer to these questions. They either implied or directly stated the relationship between the implementation and their perceptions of the system. Using Grote’s framework (1996) as a framework for implementation and then Noe’s attributes (2006) as a framework for participants to describe their perceptions, teachers and administrators provided detailed descriptions of each. Participants, in most cases, directly described the implementation, their perceptions, and the reasons for their perceptions. Their straightforward responses allowed a careful analysis of the implementation and participants’ perceptions. Because of the nature and sensitivity of the topic, a larger study with more nuanced approaches to the variables may reveal more relationships between implementation and perceptions. In summary, the interviews, surveys, and documents will serve as basis to triangulate the findings about implementation coherence. 38 EXTERNAL RELIABLITY AND VALIDITY External reliability will be sustained in the study through a careful description of methods and analysis, to allow future researchers on to approach the subject of teacher evaluation implementation coherence and the effect on teacher perception of the initiative. Focus will be given to following the same protocols with approaching and conducting surveys and interviews, to allow future researchers, either in the state of Michigan elsewhere or beyond, to replicate, test and extend the study’s and its findings. The questions in this study directly asked participants about the implementation and their perceptions. Future studies may be able to ask broader questions about the implementation and bring into the discussion other intervening variables influencing implementation and perceptions. Given the strong link between the focus of the study’s and prevailing laws and circumstances in Michigan, external validity beyond Michigan is likely limited in this study. Michigan’s laws, contexts, and circumstances limit the study’s external validity. COMPLICATIONS IN DESIGN AND METHODS Teachers and administrators were so busy that they needed the interviews to take place in their school. All teacher interviews took place in teachers’ classrooms after school or during the teachers’ preparation period. All administrator interviews took place in their offices during or after school. The initial design called for interviews to take place in a “neutral” location. In the end, that was not possible. It is unknown whether this had any influence on teacher or administrator responses. A second complication was the time needed to take part in the interview. Several randomly selected teachers did not participate because they said they simply did not have the time to dedicate to the study. Their responses, therefore, were not included in the study. The selection bias of those who “had the time” may influence the findings of the study. A third complication was that all participants knew that the researcher was a principal in the county in which the study took place. Therefore, some teachers expressed concern about their responses being relayed to their administrators and therefore impacting their standing in their schools and their districts. A fourth complication was that many of the teacher participants related their responses to their performance appraisals they received. Therefore, there was a personalization of perceptions that influenced the findings. Those teachers 39 who received high ratings, especially if they did not hold official positions in the union, generally found the system acceptable because of what happened to them, even if they had concerns or questions about the implementation and the system. Teachers were not asked about their individual performance ratings, so it is difficult to discern how much a teacher’s own evaluation rating influenced his/her perceptions about the implementation and the system. A fifth complication was defining the word “acceptable.” In Grote’s work (1996), acceptable meant that both the evaluators and those being evaluated accepted the results as valid. In the study, acceptable came to mean more than that. It seemed to encompass accepting some of the assumptions the state made about teachers and administrators as valid. It seemed to mean to teachers that standardized test scores were a valid way to measure their performance. Teachers did not want to accept this. Therefore, when asked about acceptability of their district’s teacher evaluation system, teachers seemed to be “neutral” at best. Even if the district implemented the system well, the system still was mandated by the state. “Neutral” came to mean teachers understand expectations and could perform them. Few teachers said they could accept the system as a whole because of the assumptions and realities of the state mandate. Districts’ implementation was viewed through the lens of the state’s actions. This made isolating the variable of the district’s implementation challenging. Despite these complications, many teachers expressed what seemed to be realistic and honest responses about the implementation and their perceptions. Teachers generally seemed to want the evaluation systems to be reliable, relevant, specific, strategically congruent, and acceptable. Their answers to questions about effective performance appraisal attributes seemed to belie a sense of wanting the system to be better. They hoped, generally, this project would help their districts improve their systems. Further complications are described in Appendix E. 40 CHAPTER 4: DATA ANALYSIS The data analysis is organized in five sections. The first section describes and discusses implementation steps across districts, using coherence to Grote’s implementation steps as a framework (1996). The second section describes and discusses teacher and administrator perceptions of the implementation of the teacher evaluation system and the evaluation system itself across districts. The third section describes and discusses acceptability and morale by position type across districts. Each of the first three sections sets a context and presents a summary view of districts that allows readers to contextualize the fourth section. In that section, implementation coherence, together with teacher and administrator perceptions, are described by district, one at a time, through the lens of all districts presented in the previous sections. Those district-by-district descriptions are more detailed than the summaries in the first three sections. They contain quotations, labeled by participant. The need for the fifth section emerged out of the first four. It describes and details each district’s formal and informal communication about consistent follow-through on the district’s processes; about administrator confidence and honesty regarding the evaluation system and implementation; about “highly effective” status and performance pay; about the district’s position relative to the state law; and about how well teachers understood expectations under the new evaluation system. The fifth section also reviews possible variables that were considered, but did not seem to show a strong relationship to positive teacher perceptions. All of these descriptions will be discussed further in the synthesis of Chapter 5. Given limits on Because of the amount of information that can be displayed on a single page, and considering that there were six districts in the sample, this method seems to be the most effective method to display responses. The reader can see responses across districts and then analyze each district individually. The variations among districts and within districts from the study sample are nuanced. The length and detail of the analysis in this chapter are intended to describe these variations fully. It should be noted that Grote’s implementation steps were not known by districts while they were implementing their responses. In this study, the implementation coherence metric was not provided or explained to any district before it participated in the study.. IMPLEMENTATION COHERENCE ACROSS DISTRICTS Section 1 of data analysis first seeks to identify each district’s coherence to Grote’s implementation steps (1996). Grote’s steps are first reviewed, in summary, as they were described in the earlier literature review. 41 1. Formation of an implementation team (with members from all levels of the organization): The organization forming a team was comprised of all its stakeholders, especially from both the bottom and top levels of the employee structure. In this study, the implementation team is typically referred to as the “teacher evaluation team.” 2. Top management buy-in: Regular meetings and/or involvement by/with a representative of top management enables the implementation team to win top management’s buy-in, but also to gain feedback on the process as a whole. In this study, top management’s involvement in reviewing teacher evaluation results. 3. Needs assessment: Surveying stakeholders or a group of stakeholders on the current system and their hopes for future systems is Grote’s first step, to determine what each level of the organization needs and expects from it. 4. Tool for selection/development of evaluation: The team chooses or develops a tool based on needs assessment. 5. Pilot: A test of the selected tool and system is conducted, involving one or two selected departments or units, so that obstacles can be overcome before the system is taken to the whole organization. 6. Communication plan: A plan of communication is put in place to overcome all barriers to successful implementation. 7. Training: Informing and equipping all members of the system are accomplished to optimize the use of the performance appraisal system. 8. Accountability for managers and employees: Management and staff, from top to bottom, are held responsible for the system used to appraise employees in an acceptable, reliable, and valid manner. 9. Evaluation of the appraisal system: Assessing the strengths and weaknesses of the system and its implementation is deemed a necessary process for adjusting the system to improve it. 10. Adaptation/adjustment of the appraisal system: Continuous improvement of the system for managers and employees is considered the goal of its implementation over time. Variations and similarities among districts are presented in Table 5 below, followed by a brief description of findings. Participants’ Position (job type)s by district are also indicated by the position’s initials:  DA -District Administrator  BA -Building Administrator 42  UNCSAT -Union representative Non-Core Subject Area Teacher  UCSAT - Union representative Core Subject Area Teacher  All- All participants Agreed 43 Table 6: Implementation Step Summary By District Implementation Step Evaluation Team Formed Decisions by Evaluation Team District 1 District 2 District 3 District 4 District 5 District 6 3/3 –TeamBuilding and district admins and teacher and union teacher reps (All) 3/3Cooperativel y (All) 3/3 –TeamBuilding and district admins and teacher and union teacher reps (All) 3/3 TeamUnion teacher reps and building and district admins (All) 3/3 TeamUnion teacher reps and building and district admins (All) 3/3 TeamUnion teacher reps and building and district admins (All) 3/3 TeamUnion teacher reps and building and district admins (All) 1/3 Cooperatively, with stakeholders on team (DA) 3/3 Team discussed, but final decision was made by administrati on (All) 2/3 Cooperativel y with stakeholders on team (DA, UCSAT) 2/3- Team discussed, but final decision was made by admins. (BA, UCSAT) 1/3- Team discussed, but final decision was made by administratio n (UNCSAT) 1/3 Team discussed, but final decision was made by admins (BA) 1/3- Dictated to district by Michigan Department of Education (DA) 1/3- Selected a tool from external consultant, with input from team (DA) 1/3 Team discussed, but final decision was made by administration (UCSAT) 1/3 Selected a tool from external consultant, with input from team (BA) 1/3- Other (please describe)“discussed and developed our own based on Danielson model” (BA) Notes: DA - District Administrator BA - Building Administrator All - All participants agreed UNCSAT - Union representative Non-Core Subject Area Teacher UCSAT - Union representative Core Subject Area Teacher 44 Table 6 (cont’d) Implementation Step Central Office Involvement Focus group/needs assessment completed Pilot Notes: District 1 District 2 District 3 District 4 District 5 District 6 1/3- They led the process (DA) 1/3- They were informed of the process regularly in a private setting (BA) 1/3- They were informed of the process regularly in a public setting (UCSAT) 1/3- Yes (DA) 1/3- No (UCSAT) 1/3- Not sure (BA) 2/3- They led the process (BA, DA) 1/3- Not sure how they were involved (UCSAT) 2/3- They led the process (DA, UNCSAT) 1/3- They were informed of the process (BA) 2/3- Led the process (DA, BA) 1/3- Were informed of the process (UCSAT) 1/3- They led the process (UCSAT) 2/3- They were informed of the process regularly in a private setting (DA, BA) 1/3- Yes (DA) 1/3- No (UCSAT) 1/3- Not sure (BA) 1/3- Yes (DA) 2/3- No (BA, UNCSAT) 3/3- No (All) 3/3- No (All) 3/3 No (All) 2/3- No (BA, DA) 1/3- Not sure (UNCSAT) 1/3- Yes (BA) 1/3- No (DA) 1/3- Not sure (UCSAT) 3/3- No (All) 1/3- They led the process (DA) 1/3- They were informed of the process regularly in a public setting (board meeting, committee meeting) (UNCSAT) 1/3- Not sure how involved they were involved (BA) 1/3- Yes (BA) 2/3- No (DA, UNCSAT) DA - District Administrator BA - Building Administrator UNCSAT - Union representative Non-Core Subject Area Teacher UCSAT - Union representative Core Subject Area Teacher 45 3/3- No (All) 3/3- No (All) Table 6 (cont’d) Implementation Step Communication Modes Notes: District 1 District 2 District 3 District 4 District 5 District 6 3/3Administrators explained in faculty meetings (All) 3/3Administrators explained in faculty meetings (All) 2/3Administrators explained in faculty meetings (DA, BA) 1/3- Building administrator explained at faculty meetings. (UCSAT) 1/3- Building administrator explained at faculty meetings. (UCSAT) 3/3-Emails to all certified faculty and staff (All) 3/3 Building and/or district leaders explained in union meetings (All) 1/3- Emails to all certified faculty and staff (BA) 1/3- Building and/or district leaders explained in union meetings (BA) 3/3Administrators explained in faculty meetings and district meetings (All) 1/3- Emails to all certified faculty and staff (BA) 1/3- Through emails with evaluation procedures and tools attached (BA) 1/3- Through emails with evaluation procedures and tools attached (BA) 1/3- Central office administrators explained in district wide meetings (DA) 1/3- Central office administrators explained in district wide meetings (BA) 1/3- Other “It is basically the same format that we've been using--the Charlotte Danielson Model” (BA) 1/3- Other (please explain)Teacher stakeholder input meeting (BA) 1/3- OtherQuote 1: “Building faculty meetings and/or individual private meetings with teachers” (DA) 1/3- OtherQuote 1: “Building faculty meetings and/or individual private meetings with teachers” (DA) DA - District Administrator 1/3- Emails to all certified faculty and staff (DA) 1/3- Building and/or district leaders explained in union meetings (BA) 2/3- Central office administrators explained in district-wide meetings (BA, UNCSAT) BA - Building Administrator UNCSAT - Union representative Non-Core Subject Area Teacher UCSAT - Union representative Core Subject Area Teacher 46 2/3- Building and/or district leaders explained in union meetings (BA, UCSAT) 2/3- Central office administrators explained in district-wide meetings (DA, BA) Table 6 (cont’d) Implementation Step Communication: How well was new system communicated to teachers? Modes of training for administrators District 1 District 2 District 3 District 4 District 5 District 6 1/3 Very well (BA) 2/3 Well (DA, UCSAT) 1/3 Very well (DA) 1/3 Well (BA) 1/3 Not well (UCSAT) 2/3-Through training by outside vendors (BA, DA) 2/3 Well (BA, DA) 1/3 Not well (UNCSAT) 3/3 Well (All) 3/3 Not Well (All) 1/3 Well (DA) 2/3 Not well (BA, NCSAT) 2/3-Through training by district employees and outside vendors (BA, DA) 3/3- Through providing administrators with written documents that described the new procedures (All) 2/3Through training through outside vendors (DA, UCSAT) 1/3- Through training from district employees (DA) 3/3Through written procedures (All) 2/3- Through answering questions administrators had about the procedures (DA, BA) 1/3- Through training by outside vendors (DA) 2/3 Through training by district employees (BA/DA) 2/3- Through providing administrators with written documents that described the new procedures (BA/DA) 3/3- Through answering questions administrators had about the procedures (All) 1/3- Through training by district employees (BA) 2/3- Through providing administrators with written documents that described the new procedures (BA, DA) 1/3-Through answering questions administrators had about the procedures (BA) 2/3- By answering questions administrato rs had (DA, UNCSAT) 1/3- Other (please describe) “left to sink or swim” (UNCSAT) 1/3- Other (please describe) Quote: “ I'm not sure how administrators were trained.” (UCSAT) Notes: DA - District Administrator 1/3- Through training by district employees (DA) 1/3- Through training by outside vendors (DA) Other: Quote 1: “Routine meetings aimed at creating a balanced approach amongst administrators” (BA) Quote 2: “Read Danielson Book” (DA) BA - Building Administrator UNCSAT - Union representative Non-Core Subject Area Teacher UCSAT - Union representative Core Subject Area Teacher 47 1/3Through answering questions admins had about the procedures (DA, BA) 1/3- Through providing administrator s with written documents that described the new procedures (DA) 1/3- Through answering questions admins had about the procedures (DA) Other: Quote 1: “training by the ISD” (UNCSAT) Quote 2: “No training was provided about implementati on/use of the tool” (BA) Table 6 (cont’d) Implementation Step Teachers Understand How to Perform District 1 District 2 District 3 District 4 District 5 District 6 3/3 Well (All) 3/3 Well (All) 2/3 Well (BA, DA) 1/3 Not well (UNCSAT) 3/3- Yes (All) 2/3- Yes (BA, DA) 1/3- Not sure (UCSAT) District solicited feedback on system 1/3- No (BA) 2/3- Not sure (DA, UCSAT) 2/3- Yes, with a focus group (BA, DA) 1/3- No (UCSAT) 1/3- Yes (BA) 1/3- No (DA) 1/3- Not sure (UNCSAT) 1/3- Yes, with a survey (DA) 1/3- Yes, with a focus group (BA) 1/3- No (UNCSAT) Adjustment to system based on feedback 3/3- No (All) 1/2- Changed language/descrip tions in tool (DA) 1/3 Well (DA) 2/3 Not well (BA, UCSAT) 1/3- Yes (DA) 2/3- No (BA, UCSAT) 1/3- Yes, with a focus group (DA) 2/3- No (BA, UCSAT) 1/1- Not sure (DA) 1/3 Well (DA) 2/3 Not well (BA, UNCSAT) Evaluators evaluated on their evaluations 2/3 Well (BA, DA) 1/3 Not well (UCSAT) 1/3- Yes (DA) 2/3- No (BA, UCSAT) 1/2- Other (please explain)Quote: “Used the informal feedback to finetune the logistical process (not the evaluation as a whole)” (BA) 1/2- Provided more training for administrators/ev aluators on performing evaluations (DA) 1/2- Not sure (UNCSAT) 2/3- Yes, with a focus group (DA, BA) 1/3- No (UCSAT) 1/2- Provided more training for administrators/ evaluators on performing evaluations (BA) 1/2- Other (please explain): “Changed process slightly” (DA) Notes: DA - District Administrator BA - Building Administrator UNCSAT - Union representative Non-Core Subject Area Teacher All - All participants agreed UCSAT - Union representative Core Subject Area Teacher 48 3/3 No (All) 2/3- No (BA, DA) 1/3- Not sure (UNCSAT) 0/0 Answered questions because no feedback was gained. Table 6 shows that among Grote’s ten recommended elements for implementing an employee evaluation system, the sample districts generally incorporated the same subset of elements. These included forming an implementation (evaluation) team, that included relevant stakeholders, enlisting central office involvement, selecting an evaluation tool, performing an informal needs assessment (by having a team provide feedback on the system), communicating plans, and training personnel on evaluation system’s use. All six of the districts followed these implementation steps by Grote (1996) closely. Variation among districts occurred in how they implemented those steps. Areas of difference among districts were found in how the evaluation team was chosen, how central administration was involved, how the tool was developed/adapted, how thorough the needs assessment was, how the communication used, and how the training was introduced and completed. Those differences are described district-by-district in section 4. It should be noted that none of the districts was familiar with Grote’s implementation steps prior to their implementation. Districts worked with their teachers, and with their understanding of the state law, to do the best they could in the time they had. Their practical knowledge of the new state legislation varied greatly among districts. Districts implemented while waiting for the state to provide a framework and a tool. At the time of this writing, the state had provided neither. Three years later, the state has not supplied either the training or the funding for a framework or a tool. One administrator described the concern about putting something into place if the state was going to provide resources to implement a mandated system. “The whole other concern is, is the state going to give us something? We kind of have been spinning our wheels for a couple years, because the state said, we're going to give you a tool. So, we said, well, why are we going to reinvent the wheel?” Districts were required to implement changes to their teacher evaluation system when their collective bargaining agreements with teachers and administrators expired. For Districts 1, 2, 3, 5, and 6, the implementation year was 2010-2011. For District 4, the implementation year was 2011-2012 because their collective bargaining agreement went for an additional year. According to Grote (1996), ideal timelines for implementation of performance appraisal systems is between three and six years. Most districts had approximately one school year to prepare and implement their systems. District 5 had two years. Implementation coherence was lacking in the area of a formal needs assessment. None of the districts involved general teachers (teachers at large), who were not involved on the teacher evaluation team. Districts typically did work with union representation, which makes sense as collective bargaining is the tradition of school districts and their employees. The discussions and rationale for decisions on the selection of the framework, tools, and system were unknown to the general teacher in the building, unless they had a personal connection to a member of the team or were themselves a member of the 49 team. None of the districts developed and utilized a communication plan (that was submitted or discussed in interviews/surveys) for how to introduce and implement the system. None of the districts used a pilot. All districts except District 4 seemed to hold their evaluators only informally accountable (in administrators’ evaluations). District 4 included the way that administrators evaluated teachers on evaluators’ own evaluations. Conversations and coaching about the quality of performance appraisals may have occurred with evaluators. All of the evaluation and adaption of the system was done informally. This technique of informal conversations and small changes, neither of which was clearly communicated to general teachers, seemed to shut out large portions of the general teacher population from understanding the system or feeling that their input could or would improve the system. During the 2013-2014 school year, two districts made major changes to their frameworks, tools, and systems, but those changes did not come from any formal evaluation of the teacher evaluation system in previous years. The changes were not linked to teacher needs or feedback. Only District 2 re-convened its teacher evaluation team to discuss how the system went and how to improve it. The omission of these steps seems to have a relationship to the perceptions that follow. In the key areas of morale and acceptability, the highest score teachers gave the implementation and system was “neutral.” The issue of what is acceptable, of course, is subjective and defined by participants in the study. The framework came from Noe’s research on effective evaluation systems which presumes that adherence to the recommended steps will enhance acceptability among employees. In the interviews and open-ended questions, teachers seemed to use the term “acceptable” to mean that the teacher evaluation system was reliable, relevant, congruent to district goals, and specific. Concerns in any of these areas seemed to lead teachers to say the system was not “acceptable.” The same question gave participants the option of “neutral” in the area of acceptability. “Neutral” in the study seemed to mean that the participants understood what the system really is, something mandated by the state. It had to be implemented. Whether or not teachers found the system to be reliable, relevant, congruent to district goals, and specific did not seem to matter. If it existed and teachers generally understood the system’s expectations and knew how to perform, they seemed to indicate “neutral.” “Unacceptable” came to be defined as participants understanding the system to be mandated, and this is where the study found the district doing a poor job of implementation. There were too many concerns to feel neutrally about the system and its implementation. To those interviewed, it seemed to mean that the district implemented the system ineffectively. Thus, 50 the system could not be accepted by the teachers, despite the fact it had to be implemented. One should keep these developed definitions in mind as one reads the perceptions that follow. The summary of responses that follows describes the perceptions of teachers across districts. 51 TEACHER AND ADMINISTRATOR PERCEPTIONS BY POSITION TYPE Teacher and administrator perceptions are broken into two tables. Table 6 records how teachers and building administrators perceived implementation of the evaluation system in response to concrete questions about it. These questions began the interview section about perceptions. Participants were asked to describe the most important part of the implementation, how it influenced teacher perceptions, and what should have been included (but was not) in the implementation. Such questions were posed to interviewees at the outset to afford them opportunity to organize their perceptions in terms of what was important, how implementation influenced their perceptions, and what should have been included. As participants answered these questions, they evaluated and assessed the system’s implementation wholly. Beginning with these questions was meant to wet the well to facilitate an assessment of the implementation and system by Noe’s attributes (2006). They also provided valuable data about perceptions of the implementation. Their perceptions of implementation, across districts, are related to implementation in the discussion that follows Table 7. In Table 7, the first participant responses are from the teacher union building representative. The second response is from the teacher without an official union position. The third is from the building administrator. 52 Table 7: Views of Evaluation Implementation, by District and Position Type (1: Building Union Representative, 2: Teacher, 3: Administrator) Questions about implementation Most Important Part District 1 District 2 District 3 District 4 District 5 District 6 1. Being prepared and ready to roll out to teachers 1. Communicati on from administrato rs on how it impacted us. 2. On-going training and communicati on from administratio n. 3. Have enough time to describe system to teachers. Teachers had time to work with one another. 1. More about relationship than implementat ion because the system is so flawed. 1. First initial sit-down with administrator where they told you what you were going to need to do. 1. Understanding what the target is. 1. Getting everybody on board. 1. Consistency 2. Explaining self-evals were going to determine rankings 3. Have enough time to work with teachers Implementation influence perception 1. It caused fear. 2. Do not know if it mattered. 3. The teachers lacked confidence in administrat ors. 2. Lot of anxiety at first. It’s gotten better as people have learned the system. 2. Training so all administrator s are on the same page. 2. Teachers understanding what they would be evaluated on. 2. Walkthroughs to ensure everyone knew what administration was looking for 3. Making it a part of the school improvement plan to communicate it to faculty. 3. Making sure all administrators were looking for the same thing. 1. Hurt morale. People took it personally. 1. Yes. Teachers did not know what the target was. 1. Negatively, because of lack of communication 2. Paranoia because the criteria (including Eportfolios) was not clear. 2. Yes. You are accountable to show students are growing. 2. It was stressful because any change is stressful. 3. It was okay. Teachers say it was not a “gotcha” system. 3. It was complicated because the district was implementing two things at the same time. 3. Faculty meetings with descriptions and explanations. 3. Hard to say. There is a lack of trust. 3. Difficult at first. Communicati on has brought understandin g and comfort with system. 53 2. Preobservation conference (meeting between administrator and teacher) 3. Clear understanding of the goal 1. Confusing 2. Not at all 3. Yes, positively Table 7 (cont’d) (1: Building Union Representative, 2: Teacher, 3: Administrator) Questions about implementation District 1 District 2 District 3 District 4 District 5 District 6 Should have been included, but was not 1. Have things prepared 1. A follow-up at the end of the year. 1. Clarity on “highly effective 1. More input from teachers 1. Informatio n 2. Communica te importance of different parts of the system 2. Nothing 2. More time to implement things 1. Clarity on what teachers were supposed to do. 2. More time to better understand what was happening 2. Difference between effective and ineffective ness 3. More opportunities to run trial and error on the system. 3. Clarity on “highly effective” 3. Some sort of template for teachers to know how to meet expectation s 2. How to show student growth 3. Consistency amongst evaluators 3. Professional development so teachers could understand what was expected 3. Time Most Important Part of Implementation When asked about the most important part of implementation, teachers and administrators responded with four main areas: time, communication, consistency, and training. The most important part was not necessarily what the district did but what the teachers and administrators believe could or should have been the most important parts of implementation. Time, as teachers described it, really was aligned with communication. In almost all districts in the study, as teachers got more time to talk to administrators and other teachers, they began to understand how the system would evaluate their performance. Communication regarding the system specifics through performance indicators took place throughout the implementation. Communication and clarity seemed to take time. Teachers seemed to speak more positively if they were given time to learn from each other with the administrator. A key step in implementation and communication happened in bits and pieces throughout the six districts, without a comprehensive communication plan used by any of the districts. This intermittent, yet important, communication led to perceptions influenced by each building’s leader or leadership team’s communication skills. 54 . Training and consistency also seem to be related, especially in the minds of teachers. Teachers described a need for each teacher and each department/teaching subject area to receive the same opportunity to achieve a rating that was commensurate with the effort and performance of those teachers, regardless of subject area or track of students taught. Although all districts did perform training for their administrators and ongoing tuning to improve consistency, that fact was only communicated clearly to some teachers. Without communication, teachers did not recognize the effectiveness or presence of training. One teacher described it this way: “I think we were given the information in a timely manner and so forth, and there was some discussion. They did listen to our concerns. My concern is that the people who are evaluators have not been trained. We're being evaluated on criteria when our evaluators themselves have not all been trained on how to do that. That's one thing, that we are all given the criteria, but then how were they trained on evaluating us? That was something that I think we have, one administrator who is very good at that, and the rest are learning. They realize that, that they needed more training, and so they're working on that, but that doesn't change the past.” The teacher’s statements are particularly interesting because in his/her district, administrators were provided with extensive training and met on an ongoing basis to improve consistency. Whether those efforts were unsuccessful or were unknown to most teachers, many teachers without official union positions did not know what their districts did to train and to provide consistency. This appeared to undermine teachers’ respect for the administrators as well as their assessment of the evaluation system’s legitimacy. Related to consistency was the legitimacy of the principal. Part of legitimacy seemed to be the classroom experience of the principal. Legitimacy also seemed to be related to the amount of time a principal spent listening to his or her staff. This listening could take place individually, in groups, or through union representation. Following through on the communication also seemed to relate to the consistency. Principals in Districts 1 and 2 seemed to listen to their teachers and to address teachers’ concerns. Legitimacy is foundational to the trust and collaboration present in acceptable performance appraisal systems (Noe 2006). What Should Have Been Included in Implementation When asked about what should have been included in implementation, several teachers replied that nothing should have been added. All of those teachers found the teacher evaluation results to be acceptable and did not find any improvements needed. For those teachers and administrators who did see areas that should have been included, four areas were described: having a system and tools set, clarity on targets, allowing more time and training to allow teachers to understand, and facilitating a time of trial and error for everyone to understand the system better. 55 Several teachers also indicated that more time was needed for the implementation. In some districts, there was not enough time for administrators to talk to teachers and for teachers to talk to other teachers. It appears that the compressed timeframe established by the state for evaluation implementation undermined districts’ ability to effectively communicate information about the new systems. Clarity on targets, especially in the areas of “highly effective” and student growth data, also was described by teachers as an area that should have been included in the implementation. As the systems continued in subsequent years, those student growth areas of the evaluations continued to lack definition to teachers in the study. Both administrators and teachers described a lack of guidance from the state for noncore subject area teachers, special education teachers, and social workers/counselors. Finally, one administrator said time for trial and error would have helped the implementation for both teachers and administrators. S/he described it this way: “I think at the high school level, what I would have liked to have done is to give the staff more opportunities to have a trial and error just to try it out first. Maybe take some teachers and do a pilot program with it before we jumped into it.” In terms of implementation, the selection and description of a teacher evaluation system by the district seemed limited when teachers and administrators described their knowledge about how the new system was developed by the “teacher evaluation team.” Although a framework for evaluation was adapted from each district’s old system, specifics such as how to achieve “highly effective” and how to show student growth in all subject areas were and remained unclear to many teachers. This lack of definition also could relate to the lack of a formal needs assessment. If a district would have asked teachers across the district or the building what their questions were, other than informally through a "teacher evaluation team," a more effective system could have been developed. The ideas of “highly effective” and student growth data were unclear at the outset of the teacher evaluation law’s passage. Three years later, those areas still seemed to be unsettled. The evaluation of the system and the communicated adaptations, based on teacher and administrator feedback, to that system would have allowed these questions about “highly effective” and student growth data to be more clearly defined and understood. Districts may have understood these to be unclear upon implementation, but they seemed to not address them because either they thought they were clear or they did not know how to do so. How Implementation Influenced Teachers’ Perceptions 56 When asked about how implementation influenced a teacher’s perception, several teachers said implementation did not matter because the changes to the evaluation system were mandated by the state. To paraphrase, it was what it was. However, as those same teachers continued to describe their experience, it did seem that implementation mattered. Though the state mandated some changes, districts were allowed many choices in their responses. As teachers described and thought about their experiences, it seemed to dawn on them that the implementation did matter. In three cases, teachers who said implementation did not matter to begin the interview retracted those statements and said their districts’ responses did matter and could either intensify feelings like fear, confusion, anxiety, and a lack of confidence or could reduce those feelings. In their descriptions of how they felt about the implementation, teachers and administrators explained that teachers were worried about their jobs, about what the state was trying to do with the changes in the law, and about their uncertainty in the future. Some teachers and administrators described feelings of fear and anxiety as normal because change is difficult for everyone. One building union representative, core subject area teacher described the implementation this way: “For teachers in general it was frustrating because again you didn't really know what the target was. We're so busy. We just want to know, so we can just take care of it. The perception was that it was unorganized, that nobody really knew really exactly what to do.” In terms of implementation, all of the steps, fully done, would help teachers and administrators come to a better comfort level of the system. Teacher Interview Responses about Noe’s Attributes of Effective Evaluation Systems As noted in was stated in the literature review and the description above, performance appraisal systems are described as effective if they last and they are acceptable to users, according to Noe (2006). In Table 7 below, teachers’ responses to questions about Noe’s attributes (reliability, relevance, specificity, strategic congruence to district goals, and acceptability) display their perceptions about both the system and the implementation. Across districts, there was great variation in their answers. These variations seem to broadly relate to Grote’s implementation steps. A person’s position and perspective on the implementation and system seemed to have an influence on their perceptions. In Table 7, administrator perceptions (about acceptability) were removed, so readers could see how teachers alone assessed the districts’ teacher evaluation system. It is important to understand Noe’s research (2006) in these perceptions. There is no perfect performance appraisal system. Noe’s attributes (2006) of an effective performance system can cause tension with one another. The attributes with brief descriptions are listed below. Reliability- the consistency of the performance measure. 57 Relevance- the extent to which a performance measure assesses all of the relevant aspects of performance. Specificity - the extent to which a performance measure tells employees what is expected of them and how they can meet those expectations. Strategic congruence- the extent to which a performance management system elicits job performance that is congruent with the organization’s strategy, goals, and culture. Acceptability- whether the people who use the performance measure accept it. Noe’s attributes were used to give participants criteria by which to judge each district’s teacher evaluation system. Also included were concrete questions about the most important part of the implementation; what should have been included, but was not; and how implementation influence perceptions. By first answering concrete questions about implementation, as well as questions about Noe’s attributes, participants were able to think about the implementation and the system. The first answer is a union representative teacher; the second answer is from a teacher without official union position; and the third is from the building administrator. Table 8 allows readers to see how teachers across districts answered these questions. A discussion of the responses and attributes follows Table 8. 58 Table 8: Teacher Views of Evaluation System Attributes, by District 1=Building Union Representative 2=General Teacher Noe’s Evaluation Attribute Reliable District 1 District 2 District 3 District 4 District 5 District 6 1. Yes, process different, results same 1. No, inconsistent application 1. No, inconsistency among administrators 1. No, because it is highly subjective 1. No, leadership changes hurt 2. No 2. Yes 1. No, expectatio ns were not clear across teachers 2. No, background in teaching lacking in evaluators Relevant 1. No, one short observation per year is not enough to judge all of a teacher’s performance 2. No, there is not agreement on what is the full extent of teachers’ jobs 1. No. It does not count variation among students and teachers in any class 2. Yes, they do as well as they can without being experts in my field. I get more from when other teachers observe me and give me feedback. 1. Yes, but it doesn’t count the affective side of teaching 2. No, it is too brief of an observation to try to see all the things on the rubric and everything is not included 59 2. No, highly effective ratings would not be given out or it would cost the teacher their job 1. No, because a teacher’s job is during contracted time. 2. Yes, because the administra tor stops by and watches the class 2. No, not uniform 2. No, because techniques vary among subject areas 1. No, positives are underaccounted and negatives are overaccounted 2. Yes and no, because what students think is not included 1. No, because it depends on the kids in the class 2. Yes, lots of good questions about good teaching Table 8 (cont’d) 1=Building Union Representative 2=General Teacher Noe’s Evaluation Attribute Specific Strategic Congruence to District goals Acceptability District 1 District 2 District 3 District 4 District 5 District 6 1. No, but mentors do provide this to beginning teachers 1. No, the expectations were not clear in the beginning and are not clear now. 1. Yes, better this year than last 1. No, evaluations were not specific 1. No, it is not clear on how to become highly effective 1. No, not enough information on how to do their job well 2. Yes, peer teachers and administrators helped us to understand 2. No, it varies from teacher to teacher 1. No, not more than before. 1. No, because teachers know how to align themselves with district goals 2. Yes, I know what I need to do to get a positive evaluation by talking to my administrator and union reps 1. No. District goals do not motivate teachers 2. Yes, as much as a non-subject specific tool can. I get more from my colleagues. 1. No. It feels like hoopjumping. That is not the district’s goal 2. No, they do not pay attention to district goals 2. Yes, this new system pushes teachers to be excellent 1. No, but it doesn’t matter 1. Mixed, it varies a lot from teacher to teacher. 2. Yes, because I trust my administrator 2. Yes, getting better 2. Yes, it was fair and straightforward 1. Yes, the district had this in line. 2. Yes, but there is a lack of support. 1. No, because teachers wanted to be “highly effective” 2. No, because of the variation among administrators 2. Yes, expectations are clear, top down 1. No, because teachers had to say they were going for highly effective if that is what they wanted 2. It depended on the experience level of the teacher 2. Yes, it is up to the teachers to figure this out 1. No, because it is not clear how to get “highly effective” 2. Yes, but some areas are debatable 2. Yes, this was through preobservations 1. No, we do well with kids. Evaluations do not show this 2. Yes, suggestions are a way to collaborate on district goals 1. No, basic needs are not being met 2. Yes, there are good discussions As noted, there may be trade-offs among these various attributes. For example, a system that is specific and is used with faculty members from the various disciplines in a high school setting will struggle to be relevant. A system that aligns with the district’s goals, which may change, could struggle in the area of specificity. A strength in one attribute can be a liability in 60 another. This conception of tension is evident in the discussion below. The discussion also relates the perceptions back to the synthesis of implementation coherence. Reliability Reliability is the consistency of the performance measure. This includes inter-rater reliability, or the consistency among individuals who evaluate the employee’s performance. In five of the six districts in the study, this was a chief area of concern for teachers. In districts with multiple evaluators, teachers perceived evaluators to have different approaches to evaluating teachers. The main reasons for this were the lack of training (in one district, evaluators, according to one teacher, were left to “sink or swim”), the lack of time, and the lack of experience in the teacher’s subject area. Much of the districts’ reliability depended on the individual teachers’ actual results. If the results were positive and in line with what the teachers thought their performance was, there were no concerns about reliability. If the results were lower than what the teachers perceived of themselves, reliability was brought into question. In addition, if teachers had a broader context, for example, a union position, and were exposed to multiple evaluations on a number of teachers, questions about reliability were raised by almost all of them. Differences among buildings or administrators within a building were voiced by teachers with official union roles and without official union roles. In terms of implementation steps, training for administrators, accountability for evaluators, and communication plan were related to the teachers’ perceptions. Some districts who trained administrators did not communicate this fact to teachers in a memorable way. The fact that administrators, either formally or informally, were evaluated on their evaluations of teachers was not known by teachers. This created a perception that teachers were accountable for their evaluations, and administrators were not. Although documentation and perception (among administrators) show that this perception was not true, the perception of only teachers being accountable existed among teachers. Relevance Relevance is the extent to which a performance measure assesses all of the relevant aspects of performance. The teachers’ perception of the relevance of the teacher evaluation system was mixed, almost half-and-half. The teachers’ perception about relevance was not so much about the framework districts used but about the ability of district administrators to evaluate the relevant aspects of teacher performance based on the measures or time allotted for the evaluations. 61 In most districts, the frameworks for teacher practice were measured by a single observation (three observations at most) by the administrator and the data the teacher presented at the teacher evaluation meeting at year’s end. The observations from teachers’ perceptions were seen to be too brief to evaluate if the teacher was reaching the performance goals identified by the evaluation framework. Although the feedback was given, it was limited to one or a few brief observations by administrators. From a teacher’s perspective, a broad framework was relevant, but all of the indicators would be difficult for teachers to prove or to show in a one-class observation. One core subject area teacher who did not hold a union position described his/her frustration: “…there is dialogue that I feel I can go to my administrators, any of them, and discuss my concerns. But once again, it's just a snapshot of what's happened over an entire year.” Student growth data was another area of concern for teachers. First, standardized tests offer a snapshot of a teacher’s effectiveness. Students may or may not try. If students do try, they may or may not have gained knowledge of skills in the class that will be shown on the test. The total amount of skills may or may not be measured by standardized tests. A non-core subject area teacher who did not hold an official union position described the student growth issue from his/her perspective: “It also makes me think that when we assess our kids in math and non-core classes, or those classes that are marginal, whether it's health or whatever, there should be different standards that the State has. That somehow can outgrow, without boxing teachers in, but are really clearly accurate of the uniqueness of what's going on in individual classrooms.” The second area of teacher concern for non-core subject area teachers was how to show student growth. In all districts, this area of student growth was unclear. Teachers had questions about how much student growth was enough for one school year, what student growth could be shown now or later in life, and how to design measures. If the state’s standardized test did not cover their area, non-core subject area teachers felt it was difficult to show that students were learning and retaining that learning. Teachers were left to design their own measures. Some non-core teachers questioned whether these self-developed measures were accurate in evaluating their teaching performance. The third area of concern from teachers in the study was how much time must be spent to show student learning at the expense of teaching. Some teachers felt it was obvious that students were learning in their class and trying to prove it was causing them more harm than help for students. The only reason for testing was because of the teachers’ own evaluations. It cut down on teaching time, which some teachers felt reduced students’ learning. According to one non-core subject area teacher without an official union position, this hurt students: “Part of teaching students is emotional. I'm sending all these kids out to be humiliated or be embarrassed because they can't do something when they shouldn't be able to do something. They'll have to show that they can't do it. It stinks, frankly.” 62 The fourth area of concern from teachers in the study was that all teachers, according to “word on the street” (as two teachers put it), did not actually have an end-of-the-year evaluation meeting with their administrator. With the evaluation emailed or mailed to the teacher, there was no time to give additional information or data from the teacher being evaluated. If there was additional relevant information, there was no way to convey it, from the teacher’s perspective. This led to dissatisfaction and a feeling that the evaluation did not contain the relevant information. Similar to the attribute of reliability, much of the perception of reliability seemed related to the teachers’ acceptance of their overall ranking/score and/or union position. If a person was not happy or was involved in more evaluation conversations than their own (many times, representing an employee who believed s/he received a score lower than what s/he thought was deserved), relevance was questioned more vigorously in interviews. If a teacher accepted his or her ranking or score and was not connected to anyone who did not accept his or her score, then that teacher was likely to find the tool relevant. In terms of implementation, the selection of a system was an area of concern. Who decided what would be included in showing growth of students was unclear to teachers. The relevance of the measures of student growth, in most districts, was not clear. It was seen as “jumping through hoops” and “not meaningful” by some teachers. The lack of a needs assessment at the beginning of the process and continuing throughout the implementation did not allow much time, if any, for teachers to advocate relevant parts of their jobs to be included in the teacher evaluation system. Finally, an assessment of the system, following its introduction, was missed by five of the six districts. Therefore, there was no time or space for teachers to work with administration to develop a more relevant system. Specificity Specificity is the extent to which a performance measure tells employees what is expected of them and how they can meet those expectations. Employees understand expectations and also know how to improve their own performance. In five of the six districts in the study, the expectations for teachers were perceived by most teachers as unclear. The “highly effective” status mandated by the state to be a part of the teacher evaluation system was an area brought up in most districts. Some teachers felt that more work was expected, without a clear definition of what needed evaluations or self-determination on the evaluations was described in two districts. to be done. Self- If a teacher defined their performance as “effective,” that was the highest score s/he could get. In the other district, teachers received the rating they gave themselves. Teachers who rated themselves more critically received lower ratings. This usage of self-evaluation was not made clear, so several teachers in those districts described feeling surprised by the usage of self-evaluation. One non-core subject area 63 teacher without an official union position described the self-evaluation issue this way: “I was being a little more critical of myself, and when we went back through it, after knowing that, it's like ‘You know what, I could say that I was the highest in all these things.’” Some teachers also described how different effective teaching was in several subject areas offered by high schools. Would effective teaching look the same, or even similar, among music and language arts and business classes? Teachers wanted to know how one rubric could define such various disciplines. If it did cover all subject areas, could it be clear on what each teacher should do? Some of the teachers concerned with this area felt that the current system was not specific to them. Outside of the framework, the specificity of measuring student growth was mentioned in all six districts, especially by non-core subject area teachers. Their sense was that because there was no standardized test to measure their subject-matter discipline, any display of student growth was seen as less specific than what core-subject areas could measure. Furthermore, for non-core subject area teachers, it was they who defined and developed the standards of student growth. This created work without clear guidance from the state or the district, so those teachers collaborated on establishing the metrics for student growth. This contributed to a feeling that the specificity of student growth was problematic for non-core subject area teachers. In terms of implementation, the selection and definition of the teacher evaluation system could be related to the perceptions of teachers. In several districts, teachers had questions about how teacher feedback was used in developing a system that did not include what they taught. The “highly effective” status and the question of data were directly related to the specificity of teacher evaluation system in districts. Another area of implementation related to specificity is the needs assessment. If districts asked all teachers, not just those on the “teacher evaluation teams,” how to assess performance and give specific feedback to all teachers, the framework and the system could have been adapted to better provide feedback and clear targets for teacher performance. The third step of implementation related to perception was about specificity in communication. Without a plan to explain to all teachers in the building what each one should do to achieve “effective” or “highly effective” ratings, the communication varied, depending on the administrator and teacher involved in the teacher evaluation. This affected perceptions of specificity in the system. The final step of implementation that seemed to be related to specificity is pilot. If districts had piloted their teacher evaluation system and gained feedback from this pilot, specific feedback for all teachers could have been addressed. As it was, the system was put into place without trying it first. This resulted in a paucity of feedback from all teachers, including non-core 64 special education and social workers/counselors. Once the system was put into place, only slight changes were made, and it became what was done. A pilot could have improved the system before a full implementation. Strategic Congruence Strategic congruence is the extent to which a performance management system elicits job performance that is congruent with the organization’s strategy, goals, and culture. There were mixed perceptions on this attribute. Some teachers saw it as contributing to the general, overall goal of districts as they prepare students for the future. Other teachers considered this attribute when considering school improvement goals for the district. Most teachers described the teacher evaluation system as endeavoring to motivate teachers to perform as the district would want. Several teachers mentioned that it is up to teachers to think of the goals of the district in what they should do. The goals they identified were less from a school improvement plan but more from a shared vision of how to develop students in the district. Other teachers thought about the goals of the district from a data perspective. If the district wants to improve student achievement, then this system did encourage teachers to do that. Several teachers also described how the implementation did not help them see the alignment. They said they did this on their own, and that was their job. One non-core subject area teacher without an official union position described it this way: “Obviously, it's up to us to also figure out how to implement it. Sometimes that's hard. Sometimes it's very difficult. With everything that's being asked of us, sometimes we do a really good job of it, and other times we kind of struggle with it, too.” One area of implementation related to strategic congruence is communication. According to many teachers in the study, without a plan, administrators at both the district and the building level were not able to communicate clearly. How would the teacher evaluation system help teachers better help students? When communication did occur on why teachers should engage with the teacher evaluation system, it was more about compliance. To paraphrase, the state mandated the district to do this. The district’s teachers need to comply with what the district says. In interviews and surveys, there were few descriptions of how the system would help teachers better meet the goals of the district, which ultimately would help students. Another area of implementation related to strategic congruence is the evaluation of the system. Although informal conversations are one way for administrators and teachers to identify areas in need of improvement, they are hard to quantify and move into action steps to improve the system. Surveys or focus groups would have allowed district administrators and teachers with whom they were collaborating to see the lack of clarity and motivation aligned with district goals. This 65 implementation step, combined with an adaptation in the communication and further implementation of the system, would allow teachers to connect with the larger picture of improving student achievement. Building administrators, in their descriptions of the implementation, could have communicated their higher aims for the system. This would have been an opportunity to convey to teachers broadly how this alignment works. Without such vision-casting that shows the ultimate goal of helping students, however, most teachers’ perceptions were limited on the strategic congruence. Acceptability Acceptability in performance appraisal systems is defined as whether the people who use the performance measure accept it. The perceptions of acceptability varied in both surveys and interviews. It was approximately half-and-half. Half of the teachers surveyed and interviewed said they and their colleagues found their evaluations acceptable. Five of the six building administrators reported that their teachers found their evaluations acceptable. Five of the six district administrators reported that teachers in their district found the teacher evaluation system acceptable. Many of the teachers commented that almost all of the teachers generally viewed their evaluations as acceptable in their buildings. Most teachers reported that a few people were upset with the appraisal of their performance. At the district level, the perceptions of acceptability were mixed, too. Several of the teachers stated that they did not know about perceptions outside their building, whereas others said that teachers compared their appraisals and found differences among buildings and administrators. If comparisons happened, the result tended to be a perception that the system was unacceptable. One reason given for such unacceptability was the lack of clarity on “highly effective,” as many teachers described all of their colleagues wanting to reach the highest level of performance. If reaching that level was unattainable or unclear, this caused a negative perception of acceptability. Furthermore, teachers who received ratings with which they agreed found those ratings acceptable, even if the way the teachers received those ratings was not clear. The issue of “self-evaluation” was present in two districts. If a teacher did not grade him or herself high enough in those districts, then they would not receive a “highly effective” rating. In both districts, stating or rating oneself was not clearly explained as important to the overall rating a teacher would receive. Several teachers in those districts were harder on themselves than they might have been if they would have known that their self-evaluation would determine their final rating. 66 Acceptability also seemed to be aligned to the outcomes of the teacher evaluation system. In most districts, few, if any, layoffs had been made using the teacher evaluation system. Retirements, leaves, and other teacher life events had provided enough opportunities to reduce staff. In the current state of economic distress in the state, most teachers felt the evaluations would have little, if any, effect on their job. If a teacher got an “effective” rating and felt they should have gotten a “highly effective” rating, it may have hurt their pride some, but there was no impact on their family or income. Despite this fact, the perception of acceptability was more negative than positive among teacher participants. In two districts, “performance pay” was given to teachers who earned a “highly effective” rating. The amount of money given to “highly effective” teachers in those districts was less than $100 by the second year. Therefore, there was very little economic impact to the evaluation outcomes, and most teachers were willing to accept the appraisals because it really did not matter. For those teachers who found outcomes to be very acceptable, in interviews, it was related to the system giving them a designation that validated their own perception of their performance. It showed, in their minds, that they were effective at what they did. Despite flaws in the system, teachers in Districts 1 and 2 were willing to give the administrator the benefit of the doubt. Why did the teachers in those districts do that? One core-subject area teacher who was a building union representative described the trust this way: “I suppose the process isn't the same, but I'm guessing that the results would be similar. I think that they're pretty like-minded in terms of what they're looking for in a teacher.” Other factors that seemed to be related to this trust and willingness to give their administrators the benefit of the doubt are described in Section 5 below. Those factors include administrators’ communicated confidence in themselves and in the district’s teacher evaluation system, administrators’ communicated positionality in relationship to the state’s mandated changes, and the implementation and usage of “highly qualified” designations, performance pay, and teacher lay-offs based on the district’s teacher evaluation system. In terms of implementation, the lack of a pilot in any district seemed related to the perception of teachers. If time could have been taken to pilot the system (especially without major consequences), acceptance could have been developed, especially if teacher feedback could have been used to improve and to enhance the system. As is noted above, a pilot is hard to utilize when potential job action and compensation hang in the balance. 67 Evaluation and adaption of the system also could be related to acceptability. If teachers felt their input was used to improve the system, even if they were not on the “teacher evaluation team,” it could have built a sense of acceptability, which was important to the district and something the district would endeavor to achieve. Finally, communication about the process for developing the system could be related to acceptability. Without a plan or regular updates for teachers and administrators, some teachers knew more than others. With the time span of implementation being short, the need for communication was even higher than it might have been if there was more time allowed to communicate how to achieve high rankings and how to show student growth. Both factors were mentioned when teachers were asked about the acceptability of the system. ACCEPTABILITY AND MORALE BY POSITION TYPE The perceptions of survey responses by position type present an interesting picture of the analysis across six districts. Volunteer district administrators, volunteer building administrators, and randomly selected volunteer union representatives responded to the surveys at the beginning of the study. These perception questions were mixed into the questions about what the district did to implement. These responses are important as one analyzes the responses through each district. It is clear that building and district administrators perceived the implementation more positively than union representative teachers did. Two of the three survey respondents were administrators based on the survey design, so one must view their responses as positionally positive. This could be for several reasons. One of those reasons is that none of the districts used a district-wide survey about the teacher evaluation system. Another possible reason is that teachers, when they talk to administrators, may be overly positive themselves because they might feel the administrators may rank them lower if they are negative. Teachers may have too much to lose to be honest with their administrators. Therefore, as far as the administrators know, the implementation was positive. From what administrators know, the implementation did not influence or positively influenced results. From a union perspective, the Michigan Education Association may influence union representatives’ negative perceptions. Despite protestations from union leaders, the teacher evaluation laws were passed. In local contracts, many of the protections negotiated into contracts represent a trade-off of job security for money, when districts did not have money to give. To give those protections up without any financial compensation might have seemed unfair to union leaders. In addition, union leaders see the teacher evaluation system across different teachers. Although union representatives themselves may have gotten strong evaluations, they might see what other teachers perceive as a lack of 68 acceptability. Typically, union representatives are only involved in cases where the teachers they represent feel negative about their evaluations. This may skew the union representatives’ perceptions of the teacher evaluation system. Furthermore, union representatives may better understand that speaking honestly in a study should not result in any job action, whereas at-large teachers may fear for their jobs if they answer honestly. All of these factors should be considered when viewing Table 9 below, as well as the district-by-district tables that follow. 69 Table 9: Summary of Perception Responses by Position Table Question Summary of District Administrator Responses Summary of Building Administrator Responses Summary of Union Representative Responses Summary of Teachers Who Did Not Hold Official Union Positions How well was communication done? Very well- 1 Very well- 1 Well- 2 ---- Well-4 Well-3 Not well- 4 Not well-1 Not well-2 How well did teachers understand how to perform? Well-6 Well-4 Well-2 Not well-2 Not well- 4 How did implementation influence building morale? Improved morale-1 Neither improved or diminished morale -3 Diminished morale -5 Diminished morale- 3 Greatly diminished morale- 1 Neither improved or diminished morale -4 Diminished morale-1 How did implementation influence district morale? Improved morale-1 Neither improved or diminished morale -5 Neither improved or diminished morale -4 Diminished morale -5 Diminished morale- 2 Greatly diminished morale- 1 ----- Neither improved or diminished morale-5/18 Negatively influenced-13/18 Neither improved or diminished morale-6/18 Don’t know- 5/18 Negatively influenced-7/18 How acceptable is the teacher evaluation system? Acceptable- 4 Highly acceptable-1 Unacceptable-6 Acceptable- 3/18 Neutral-2 Acceptable- 2 Neutral-6/18 Neutral-1 Unacceptable9/18 Unacceptable- 2 District administrators felt the implementation of the steps went well. A majority of them felt that the communication was done well and teachers understood the expectations well or very well. Building administrators shared this positive view of the communication and clarity of expectations. Union representatives, however, had a different view. A majority of them described the communication as not done well and teachers not understanding expectations well. 70 Morale, which might be difficult for anyone to judge clearly, varied extensively among administrators. A small majority of administrators thought that there was no negative influence on morale in the implementation. All of the union representatives felt that the implementation diminished or greatly diminished the morale at both the building and district level. From the results listed above, the best outcome a district could hope to achieve with its teacher perceptions is no influence on morale and neutral in terms of acceptability. This could be due to several factors. The first is that any change to an evaluation system is traumatic for the employees being evaluated. Change is difficult. Though all of the districts tweaked their previous evaluation system, the state dictated the stakes were higher. Possible performance pay hung in the balance, if not in actuality, then in the wording of the law from the state. Additionally, teachers could feel as if their performance is “highly effective” since that terminology was ambiguously defined. Without previous experience with what the differences between “minimally effective”, “effective”, and “highly effective”, the expectation easily may be that they are “highly effective” because, as stated during the literature review, 85% of Fortune 500 employees felt their performance was above average. A second factor explaining why “neutrality” toward the system is the way the state enacted the laws regarding teacher evaluations. State law mandated that teacher evaluations could not be collectively bargained. Though all of the districts in the study collaborated with teachers on the adaption of their teacher evaluation system, the tone of the law made teachers feel as if system was implemented without formal negotiations. The job protections formally in teacher contracts were null and void, despite the financial concessions teachers may have given to achieve those previous contractual procedures and practices. This enactment combined with other state laws forcing teachers to pay more for their retirement benefits, which amounted to a pay decrease for teachers. Further, the state mandated caps to teacher health insurance, so teachers were forced to pay more for these benefits than they had before the advent of cost reduction laws. All of these laws were passed at the same time. For teachers, it might feel as if they were under attack. Because of this convergence of factors, a neutral response may have been the best a district could achieve. A third factor is the limited amount of broad teacher input into the teacher evaluation system. Whether districts used union positions or teacher volunteers on their “teacher evaluation teams”, teachers at large in districts may have felt disconnected to the process of adapting and implementing each districts’ teacher evaluation system. Without enlisting broad teacher input and communicating the rationale and decisions embedded in the teacher evaluation system, teachers without input generally could be expected to feel neutrally about the system. 71 IMPLEMENTATION COHERENCE AND PERCEPTIONS BY DISTRICT With the broad context of all districts in mind, a detailed description of each district response follows. The responses to the questions about implementation varied between districts and within the districts themselves. The district by district analysis starts with implementation coherence, followed by a narrative description of each district’s implementation, concrete questions about implementation, with attributes of effective performance appraisal systems, and end with an analysis the relationship between the implementation process and educator perceptions in the district. District 1: Large Suburban District District 1 is a large suburban district in Michigan. In this district, there have been no layoffs based on the teacher evaluation system, as attrition and retirements have allowed the district to cut faculty based on lower funding levels and decreased student count over the last three years. No performance pay has been issued to highly-ranked teachers. The implementation and perceptions are influenced by the fact that the impact of teacher rankings has been mostly a matter of pride for teachers and did not influence job action or compensation. District 1: Implementation District 1 implemented the teacher evaluation system using many of the Grote’s steps. Unique to District 1 was an impactful teacher mentoring system for beginning teachers. Also, District 1 used informal feedback from teachers to adjust their usage of self-evaluations in the evaluation system. Table 10 below describes responses by participant. DA indicates District Administrator; BA indicates Building Administrator; UCSAT indicates Union rep Core Subject Area Teacher; CSAT indicates Core Subject Area Teacher; NCSAT indicates Non-Core Subject Area Teacher; and All indicates all participants agreed on this response. 72 Table 10: District 1 Implementation Survey Reponses Grote’s Implementation Steps D1 Implementation Surveys Implementation Documents Interviews Surveys-Do Teachers Remember the District Implementation Steps? (One Building Union Representative, One Central Office Administrator, and One Building Principal) (Provided by Central Office Administrator) (Surveys of two core subject teachers and one non-core subject teacher) Formation of Implementation Team 3/3 –Team- Building and district administrators and teacher and union teacher representatives (All) 3/3 Cooperatively, with stakeholders on team (All) N/A (Interviews with One Building Union Representative, One Building Administrator, and One HS Teacher) 3/3 Remembered team being formed (All) ------ 1/3- They led the process (DA) 1/3- They were informed of the process regularly in a private setting (BA) 1/3- They were informed of the process regularly in a public setting (UCSAT) 2/3- Yes (BA, UCSAT) 1/3- No (DA) Documents used by central office personnel to roll out program were submitted. No one described how team made decision (All) 1/3 Said superintendent led the process with administrators and teachers (BA) No one described a formal needs assessment (All) 2/3 Do not remember very well (2 CSAT) 1/3- Do not remember at all (NCSAT) How was the decision on the model made? Central office involvement Focus group/Survey for needs assessment N/A N/A Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 73 2/3- Do not remember very well (CSAT and NCSAT) 1/3-do not remember at all (CSAT) 1/3- Remembers (CSAT) 1/3 Do not remember very well (CSAT) 1/3 Do not remember at all (NCSAT) Table 10 (cont’d) D1 Grote’s Implementation Steps Pilot Implementation Surveys Communication Modes Communication: How well was new system communicated to teachers? Modes of training for administrators Implementa tion Documents ---- Interviews 3/3- Administrators explained in faculty meetings (All) 3/3-Emails to all certified faculty and staff (All) 3/3 Building and/or district leaders explained in union meetings (All) 1/3- Central office administrators explained in district wide meetings (DA) 1/3- Other “It is basically the same format that we've been using--the Charlotte Danielson Model” (BA) 1/3 Very well (BA) 2/3 Well (DA, UCSAT) Documents used to describe teacher evaluation system to teachers were submitted 3/3 Mentioned it being communicated in meetings (All) 1/3 Mentioned system being clarified in one on one meetings with administrators (UCSAT) 3/3- Described administrators describing teacher evaluation system in individual meetings with faculty members (All) N/A 1/3- Through training by outside vendors (DA) 2/3 Through training by district employees- (BA/DA) 2/3- Through providing administrators with written documents that described the new procedures (BA/DA) 3/3- Through answering questions administrators had about the procedures (All) Documents used to describe system to administrato rs were submitted 3/3 Described understanding the basic parts of the evaluation system well. Data and “highly effective” ratings were not explained well. (All) 1/3 Mentioned administrators being trained by superintendent, who was up on all the laws (BA) 3/3- No (All) ----- Surveys-Do Teachers Remember the District Implementation Steps? 2/3-Do not remember very well (CSAT and NCSAT) 1/3- Do not remember at all (CSAT) 1/3 Remembers well (CSAT) 1/3 Remembers (NCSAT) 1/3 Does not remember at all (CSAT) --- 2/3 Remember (CSAT, NCSAT) 1/3 Does not remember well (CSAT) (Trainings were grouped together) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 74 Table 10 (cont’d) D1 Grote’s Implementation Steps Modes of training for teachers Implementation Surveys Implementation Documents Interviews 1/3- Through emails describing the way teachers were expected to perform (BA) 3/3- Through emails with evaluation procedures and tools attached (All) 3/3- Through mandatory trainings before the year started (All) 2/3- OtherQuote 1: “All of our teachers have mentors in the first 4 years of their employment and we use the Charlotte Danielson Model with our protégés” (BA) Quote 2: “Staff meetings” (DA) 3/3- Yes (All) Documents used to describe system to teachers were submitted 3/3 Mentioned communication about system being done during meetings at the beginning of the year (All) 3/3- Described administrators describing teacher evaluation system in individual meetings with faculty members (All) N/A Teachers Understand How to Perform 3/3 Well (All) N/A District solicited feedback on system 1/3- No (BA) 2/3- Not sure (DA, UCSAT) N/A Adjustment to system based on feedback 3/3- No (All) N/A 0/3 Mentioned administrators being evaluated on how they evaluated teachers 3/3- Said teachers understood the observations, but struggled with the student growth data (All) 0/3 Described district feedback through surveys or focus groups 3/3 Described how the system changed each year. The first year, there was a self-evaluation. The second year was a mix of selfevaluation and administrator evaluation (All) Evaluators evaluated on their evaluations 75 Surveys-Do Teachers Remember the District Implementation Steps? 2/3 Remember (CSAT, NCSAT) 1/3 Does not remember well (CSAT) (Trainings were grouped together) 1/3 Remember (CSAT) 2/3 Do not remember well (CSAT, NCSAT) ---- 2/3 Do not remember very well (CSAT, NCSAT) 1/3 Do not remember at all (CSAT) 2/3 Remember (CSAT, NCSAT) 1/3 Do not remember very well (CSAT) District 1 used a team that collaboratively made decisions. The team included district administrators, building administrators, and teachers. That team, however, was disbanded after initialization, so further feedback was not solicited as usage of the system continued. Teachers surveyed did not remember the team’s formation very well or at all. Central office administrators led the team, though most teachers did not remember the role of central office. This could be because there were several changes in district leadership during the implementation of the teacher evaluation system. No pilot, no formal needs assessment (the teacher evaluation committee provided informal feedback), and no formal communication plan was used. District 1 used district and building administrators to communicate and describe the teacher evaluation system to teachers during meetings. These explanations were extended in one-on-one meetings with teachers and union representatives. Written documents of the system were shared with teachers through email. Teachers remembered the communication and training of teachers. This narrative is supported by the union/administrator surveys, interviews, documents, and teacher surveys on Table 1. According to surveys and interviews, teachers generally understood expectations. This understanding of expectations was supported by the district’s strong and long-established teacher mentor program, which was described by both teachers interviewed and surveyed. A teacher’s first year being mentored was described by a mentor teacher this way: “In the first year we've got all these forms. One each term. You're really getting feedback. You're really getting constant hand holding, and feedback, and help. So I think that this works to help make teachers better. Whereas, I don't really think that the evaluation process has very much to do with anything.” Administrators were trained by district-level personnel on the observation framework and tools. Though several teachers understood that administrators were evaluated on their evaluations of teachers, general teachers did not remember this fact in the surveys. During the first year of implementation, teachers who rated themselves as “highly effective” typically received those ratings. Those teachers who were more critical and gave themselves “effective” ratings received them. Teachers were not informed of the importance of self-evaluation until after they completed the self-evaluations. The district did not formally survey teachers or use a focus group to assess teacher perceptions of the teacher evaluation system. There was informal feedback given during the individual end-of-the-year meetings with teachers. Very few adjustments were made to the teacher evaluation system after initialization. The main change was the weight of the self-evaluations. No union representatives/ administrators remembered administrators being evaluated or the district soliciting feedback on the system. Several teachers remembered the changes to the system, which were identified through meetings with administrators at the end of the year, not through a formal survey or focus group on the evaluation system. 76 District 1: Teacher and Administrator Perceptions Teacher and administrator perceptions varied in some areas across District 10 and were similar in other areas. These similarities are visible below on Table 11, Table 12, and Table 13. Table 11 is organized by responses about the implementation’s acceptability and influence on morale. Table 12 describes specific and concrete questions about the implementation, including its most important part: what influence it had on perception, and what should have been included in it. Table 13 follows Noe’s attributes (reliability, relevance, specificity, strategic congruence, and acceptability). Table 12 is initial-coded like Table 11. Table 13 and 14’s responses are sequenced by respondent. First respondent is teacher union-representative; second respondent is teacher without an official union position; third response (only on questions of most important step, influence of implementation on perception, and what should have been included). Table 11: D1 Teacher/Administrator Survey Perception of Implementation’s Influence District 1 Teacher Morale and Acceptability from Surveys In building, how did the changes influence morale? Implementation Surveys Implementation Documents Surveys-Do Teachers Remember the District Implementation Steps? Neither improved or diminished morale 2/3 (BA, DA) Diminished morale 1/3 (UCSAT) N/A In district, how did the changes influence morale? Neither improved or diminished morale 2/3 (BA, DA) Diminished morale 1/3 (UCSAT) 1/3 Highly acceptable (BA) 1/3 Neutral (DA) 1/3 Unacceptable (UCSAT) N/A Quote 1: “none as far as I have seen” (CSAT) Quote 2: “It seems to be accepted as just another evaluation tool. There has been some discrepancy/discussion about what constitutes effective/highly effective self selection.” (CSAT) Quote 3: “Neutral” (NCSAT) Quote 1: “none as far as I have seen” (CSAT) Quote 2: “Hard to speak outside of my building, haven't heard much one way or another.” (CSAT) Quote 3: “Neutral to my knowledge” (NCSAT) 3/3 Neutral (All) Based on your experience overall, how acceptable have you found the district’s teacher evaluation system? N/A Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 77 Table 12: District 1 Interview Perception Response Summary 1: Building Union Representative Question Most Important Part Implementation influence perception Should have been included, but was not Summarized Responses 1. Being prepared and ready to roll out to teachers 2. Explaining self-evals were going to determine rankings 3. Have enough time to work with teachers 1. It caused fear. 2. Do not know if it mattered 3. The teachers lacked confidence in administrators 1. Have things prepared 2. Communicate importance of different parts of the system 3. Some sort of template for teachers to know how to meet expectations 2: Teacher 3: Administrator Quotes 1. “..they were inventing it as they were going along with it.” (UCSAT) 2. “Had I known that, I would have answered the questions a little differently.” (NCSAT) 3. “I think the problem is, we just don't have enough time to do those things.” (BA) 1. “You have a pretty large group of teachers who are "Am I going to lose money? Am I going to lose my job," who are really stressed about it.” (UCSAT) 2. “I don't know that it had anything to do with it frankly, one way or the other.” (NCSAT) 3. “I don't think we….me..helped that when they had questions that I couldn't answer.” (BA) 1. “..they were inventing it as they were going along with it.” (UCSAT) 2. “I was being a little more critical of myself and when we went back through it, after knowing that, it's like "You know what, I could say that I was the highest in all these things." (NCSAT) 3. “As administrators, we couldn't say to teachers we're not really sure what a good evaluation looks like.” (BA) 78 Table 13: District 1 Interview Perception Response Summary based on Noe’s Attributes 1: Building Union Representative Question Reliable Summarized Responses 1. Yes, process different, results same 2. No Relevant 1. No, one short observation per year is not enough to judge all of a teacher’s performance 2. No, there is not agreement on what is the full extent of teachers’ jobs Specific 1. No, but mentors do provide this to beginning teachers 2. Yes, I know what I need to do to get a positive evaluation by talking to my administrator and union reps 1. No, district goals do not motivate teachers 2. No, they do not pay attention to district goals Strategic Congruence to District goals Acceptability 1. No, but it doesn’t matter 2. Yes, because I trust my administrator 3. Yes, only a handful didn’t like that their self-evaluation was their actual evaluation 2: Teacher 3: Administrator Quotes 1. “I think that they're pretty like-minded in terms of what they're looking for in a teacher.” (UCSAT) 2. “With principals and with colleagues or whoever's coming in. If they're your friends, they're going to be more lenient.” (NCSAT) 1. “If you walk through in five minutes, you could see me saying, "Stop it, Andrew. You've been doing this for 20 minutes, and it's really driving me crazy," because I've had it with him, or you could see me saying, "OK, that's a good idea. Let's pull this together. How can we turn that into law?" Five minutes isn't going to give you very much information.” (UCSAT) 2. “I don't know that if you put the best and brightest teachers in a room and gave them a year that they could come up with something that accomplishes that task.” (NCSAT) 1. “I think that the mentor program does a pretty good job of that.” (UCSAT) 2. By talking to administrators one-on-one. By talking to union representation one-on-one.” (NCSAT) 1. “ I think something that's as artificial as an evaluation by a principal who comes in your room only knows because he's got it on a list, isn't making a difference.” 2. In the day-to-day goings of teaching, you don't pay a whole lot of attention to the district goals and objectives, no matter how hard the superintendent and other people try. We have the day-to-day business of taking care of kids. 1. “So effective, how effective, is really no difference in the long run.” 2. “I think if it was somebody else doing my eval, I would be much more critical. There has to be some level of trust, or at least respect of their credibility to come in and judge me.” 3. “Some of them self-evaluated effectively, so we gave them the evaluation, and they didn't like that.” Interview participants felt that the most important parts of implementation were having things set up before rolling them out, explaining self-evaluation’s importance, and having enough time to work with teachers. Teacher and administrator participants felt that issues revolving around the usage of self-evaluations and the whole system were not thought-out well beforehand, so the implementation held flaws with logistics and follow-through. In the years after implementation, no feedback was formally elicited from teachers through surveys or focus groups, leaving teacher questions about data and the “highly effective” status unanswered. The observation framework and the mentor teacher program, however, did provide feedback to teachers outside of the context of formal evaluations. 79 Two of the interview participants expressed a negative view of the implementation because the system was not clear to administrators or teachers, which caused fear and confusion. A third participant said that the implementation did not matter because the system was not clear; therefore, the way it was explained did not matter. As to the attributes of a performance appraisal system, both teachers interviewed indicated that the system was not relevant or aligned to district goals. Areas of concern among participants included the lack of a clear template for what teachers needed to do, having the system prepared, and a missing specificity on areas of importance in the teacher evaluation system. The two teachers interviewed gave mixed opinions on the specificity, reliability, and acceptability of the system. All five of the teachers who finished the survey said that teachers were stressed out at the change and the lack of understanding of the new system. In terms of building morale, only one teacher (of the five surveyed) described the system as negatively influencing teacher morale in the building. Interviews revealed that the district’s well-developed new teacher mentoring system, along with the experience of the building’s administrators and trust in them, forged stability if parts of the teacher evaluation system were unclear. Embedded in quite negative perceptions of the system were two positive perceptions. One was trust in the evaluators. Both interview participants said that their relationship with administrators and a shared vision of the district’s way of doing things made an unclear and roughly implemented system acceptable. A second positive perception was the role the other teachers took in helping inexperienced or struggling teachers understand expectations and provide observational feedback to those teachers. Both of those implementation factors seemed to influence the teachers’ positive perception of morale and acceptability in the district’s high school. District 1: Relationship between Implementation and Perception District 1 followed several of Grote’s implementation steps: formation of a “teacher evaluation team”, collaborative decision-making in the team, central office leadership/buy-in, informal feedback from teachers on the team, communication, training for teachers and administrators, and adjustments to the teacher evaluation system (chiefly around the weight of selfevaluations by teachers). Of those steps, and the steps that were not used, communication was not organized and executed at the district or building level. There was no communication plan developed or implemented. This created a lack of understanding of the strategic congruence and acceptability of the system. Teachers talked amongst themselves and with administrators to understand better how the system aligned with district goals and was fair and acceptable to employees. 80 Trust of administrators became more important in the teachers’ perceptions than widely distributed and understood communications from the district. Issues like the weight of self-evaluation, length and importance of teacher observations, and the display of student growth were described in teacher and administrator perception interviews as concerns. The lack of a clear communication plan seemed to be linked to some of the missed steps in the implementation. Those steps will be detailed below. The teacher evaluation team seemed to be more focused on completing a system than on communicating the system. According to Grote, communication is key to any implementation of a performance appraisal system. A lack of communication can impact all components of the system. Teachers had to depend on information from union representatives and administrators, which could vary based on the amount of time to talk about the system with administrators. The first step missed was a formal needs assessment. A formal needs assessment would have allowed the district to identify the challenges of the current system and to set goals and obstacles to implementing a new system. A chief area of concern was measuring relevant parts of a teacher’s job. Without the input of teachers throughout the district, the system was unclear to those not on the “teacher evaluation team.” Teachers did not feel they had input. The district team did not have an understanding of all stakeholders’ concerns about the current system. With more input, a more inclusive and relevant teacher evaluation system could have been developed and used. District 1 did not use a pilot. When teachers and administrators said that more time was necessary to understand the system better, they might also have identified the need to try the system prior to a high-stakes implementation. According to the building administrator, there was no time available for the pilot because of state-mandated implementation and rankings. The district could have benefited from piloting a system that used the rankings. It could have communicated this to teachers and to administrators. As a result, the district might have had all components of the system, including the complete toolbox of resources, without the pressure of the teacher evaluation system being linked to positive or negative job action and/or compensation. The lack of a pilot manifested itself in a lack of reliability among evaluators. There was not time to focus on the specifics of the system and learning it. Administrators had to be competent and knowledgeable, even if they were not. Teachers then perceived the system as lacking in reliability. An important step District 1 did not follow was a formal assessment of the system. There was no reconvening of the “teacher evaluation team.” There were no surveys. There were no focus groups. District 1 did reduce the weight of selfevaluations, which were perceived to influence directly each teacher’s ranking. Teachers were not informed of the weight of the self-evaluations before they completed them. The negative perception of this practice, of giving teachers what they gave themselves on the self-evaluations, was communicated by some to administrators through informal feedback to administrators 81 by teachers. Without a wider-ranging attempt to evaluate the teacher evaluation system, the concerns in the teacher evaluation system were handled here and there, or not at all. This caused the sense that the system, which was perceived as very flawed, would be the system the district used until the state mandated a new system. Three years later, the state has not mandated a new system, and District 1 made little attempt to gather feedback on the system. Therefore, only the minor change about teacher self-evaluation was made, but many questions and concerns remain. The final step of Grote’s that was not clearly used was evaluation of principals and directors. It may have been that central office administrators were providing feedback, even informally, to building administrators on their evaluations of teachers. If this was the case, it was not communicated to teachers. Though no teachers directly stated this was a problem, the perceptions of problems with reliability among evaluators could be attributed to either a lack of “someone else” evaluating the performance appraisals or a lack of communication of “someone else” evaluating the performance appraisals. Either way, teachers felt the evaluations were inconsistent and were more dependent on relationships with administrators than job performance. On the positive side of perceptions, teachers felt they understood the observation rubric. Even though all parts of the evaluation process were not clear, the teachers trusted the stable leadership in the building to use the system as best they could. Building administrators built trust by admitting they were learning the system and the system had flaws. Though usage of self-evaluations so heavily was not communicated well, there did not seem to be fear about using those self-evaluations. Including a self-evaluation in the evaluation system does allow the teacher to have a say in their evaluations. This also built trust in the administrators and their ability to navigate the system with the teachers. Overall, the teachers in the system found the system neither bad nor good, in terms of acceptability and morale. Their perceptions were that the administrators were experienced and that nothing too bad could happen because the teachers trust their administrators. Though one could view this as a positive, it also is indicative of a lack of information or ongoing improvement in the teacher evaluation system in District 1. Below you will find Table 14 summarizing the relationship between implementation and perceptions for District 1. 82 Table 14: District 1 Implementation and Influence on Perception of the System Implementation Steps/Lack of Implementation Steps Lack of communication about evaluation team’s process Lack of pilot and needs assessment Lack of evaluations of administrators Trust and honesty about the system by administrators Perceptions Influenced Some teachers felt that they had to figure out the system through individual meetings with other teachers and administrators to “figure out” the system, rather than all teachers being fully informed of the system Lots of logistical issues and unanswered questions (that continue as the system continues to be used) that could have been avoided with a pilot or a needs assessment Some teachers felt as if the evaluations had more to do with the relationship and connection to their administrator than their performance as a teacher Because teachers trusted their administrators and because administrators admitted they were figuring out the system, anxiety was reduced. Even when teachers were disappointed by their results (maybe too weighted by self-evaluations in the first year), they trusted the administrators were doing their best to give the teachers a fair evaluation. 83 District 2: Medium Town Fringe District 2 is a medium-sized town in Michigan. In this district, there have been no layoffs based on the teacher evaluation system, as attrition and retirements have allowed the district to cut faculty based on lower funding levels and decreased student count over the last three years. No performance pay has been issued to highly-ranked teachers. The implementation and perceptions are influenced by the fact that the impact of teacher rankings have been mostly a matter of pride for teachers and did not influence job action or compensation. District 2: Implementation District 2 used many of the same implementation steps as District 1, with a few unique steps. Unique to District 2 was their usage of an assessment of the teacher evaluation system after the first year, using union representatives, district representatives, and building representatives to identify what worked and what did not work. District 2 also utilized faculty meetings throughout the year to allow teacher and administrators to discuss and to understand the teacher evaluation system. Table 14 below describes responses by participant. DA indicates District Administrator; BA indicates Building Administrator; UCSAT indicates Union rep Core Subject Area Teacher; CSAT indicates Core Subject Area Teacher; and NCSAT indicates NonCore Subject Area Teacher. 84 Table 15: District 2 Implementation Survey Reponses D2 Grote’s Implementation Steps D2 Implementation Surveys Implementation Documents Interviews (One Building Union Representative, One Central Office Administrator, and One Building Principal) (Provided by Central Office Administrator) Formation of Implementation Team 3/3 –Team- Building and district administrators and teacher and union teacher representatives (All) 1/3 Cooperatively, with stakeholders on team (DA) 1/3 Team discussed, but final decision was made by administration (UCSAT) 1/3 Selected a tool from external consultant, with input from team (BA) 2/3- They led the process (BA, DA) 1/3- Not sure how they were involved(UCSAT) Agenda from Team Meeting in Summer 2012 (no members listed) (Interviews with One Building Union Representative, One Building Administrator, and One HS Teacher) 1/3 Mentioned the formation of the team (BA) 1/3- Yes (DA) 1/3- No (UCSAT) 1/3- Not sure (BA) N/A How was the decision on the model made? Central office involvement Focus group/Survey for needs assessment Surveys-Do Teachers Remember the District Implementation Steps? (Surveys of two core subject teachers and one non-core subject teachers) 1/3 Do not remember very well (NCSAT) 2/3- Do not remember at all (CSAT 2) N/A 1/3 Mentioned the teacher-admin team working together. (BA) ----- Implementation documents were provided by central office 1/3 Mentioned central office administrators being involved in the implementation (CSAT) 1/3 Mentioned teachers and union representatives giving feedback to process and system before implementation. (BA) 1/3- Remembers well (NCSAT) 1/3 Remembers (CSAT) 1/3 Do not remember at all (CSAT) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 85 1/3 Do not remember very well (CSAT) 2/3- Do not remember at all (CSAT, NCSAT) Table 15 (cont’d) D2 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews Pilot 3/3 No (All) N/A 0/3 Mentioned a pilot Communication Modes 3/3Administrators explained in faculty meetings (All) 1/3- Emails to all certified faculty and staff (BA) 1/3- Building and/or district leaders explained in union meetings (BA) 1/3- Central office administrators explained in district wide meetings (BA) 1/3- Other (please explain)- Teacher stakeholder input meeting (BA) Documents detailing the process/schedule teacher goals were included. These were communicated to certified faculty leading up to beginning usage. 3/3 Talked about building meetings where administrators explained system, process, and schedule. (All) 2/3 Described district meetings where district wide system, process, and schedule were discussed. (BA, UCSAT) 3/3- Described administrators describing teacher evaluation system in individual meetings with faculty members (All) Communication: How well was new system communicated to teachers? 1/3 Very well (DA) 1/3 Well (BA) 1/3 Not well (UCSAT) N/A 2/3- Described that communication about the basic system was handled by administrators. Questions about data emerged while system was being implemented. (UCSAT, NCSAT) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 86 Surveys-Do Teachers Remember the District Implementation Steps? Not included because no evidence was given this was a part of implementation 1/3 Remembers well (NCSAT) 1/3 Remembers (CAST) 1/3 Does not remember very well (CAST) ---- Table 15 (cont’d) D2 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews Modes of training for administrators 2/3-Through training by outside vendors (BA, DA) 1/3- Through training by district employees (BA) 2/3- Through providing administrators with written documents that described the new procedures (BA, DA) 1/3-Through answering questions administrators had about the procedures (BA) 1/3- Other (please describe) Quote: “ I'm not sure how administrators were trained.” (UCSAT) 3/3- Other (please describe) Quote 1: “The evaluation tool/system was discussed at meetings.” (UCSAT) Quote 2: “We really didn't train teachers on how to perform to be highly rated.” (DA) Quote 3: “Reviewed domains and performance rubrics with teaching staff at faculty meetings” (BA) N/A 0/3- Mentioned training for administrators Documents detailing the process/schedule teacher goals were included. These were communicated to certified faculty leading up to beginning usage. 3/3 - Described administrators (district and building) explaining rubrics, goals, and data at building and district meetings 3/3- Described individual meetings with teachers about teacher evaluation system (All) Modes of training for teachers Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 87 Surveys-Do Teachers Remember the District Implementation Steps? 2/3- Remember well (CSAT, NCST) 1/3- Remember (CSAT) (Trainings were grouped together on teacher survey) 2/3- Remember well (CSAT, NCSAT) 1/3- Remember well (CSAT) (Trainings were grouped together on teacher survey) Table 15 (cont’d) D2 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews Evaluators evaluated on their evaluations 2/3- Yes (BA, DA) 1/3- Not sure (UCSAT) N/A Teachers Understand How to Perform 3/3 Well (All) --- District solicited feedback on system 2/3- Yes, with a focus group (BA, DA) 1/3- No (UCSAT) Adjustment to system based on feedback 1/2- Changed language/descriptions in tool (DA) 1/2- Other (please explain)-Quote: “Used the informal feedback to fine tune the logistical process (not the evaluation as a whole)” (BA) Agenda from July 2013 to review evaluation procedures, practices, and system N/A 0/3- Mentioned administrators being evaluated on their ability to evaluate teachers (All) 0/3 Described how well teachers understood how to perform 1/3- Mentioned meeting with teachers to assess effectiveness of teacher evaluation system (BA) 3/3- Described logistical changes to system from conversations between teachers and administrators (All) Surveys-Do Teachers Remember the District Implementation Steps? 3/3 Do not remember at all ---- 3/3 Do not remember at all (All) 1/3 Remember (CSAT) 2/3 Do not remember very well (CSAT, NCSAT) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher District 2 used a team. It was made up of teachers and administrators. According to teachers, the final decision was administration’s; according to administrators, the decision was collaborative. Central office administrators led the “teacher evaluation team.” The district did solicit needs from its implementation from the “teacher evaluation team”, but did not seek opinions from district teachers as a whole. 88 Communication and training were given to teachers during faculty meetings and district meetings by administrators. Later faculty meetings allowed teachers to discuss the specifics of the teacher evaluation system. As teachers met with administrators to start the first year of implementation, administrators provided further detail on the system. Training was provided to administrators from outside vendors (on the tools to be used) and district personnel (on the rubric). There was no pilot. After the first year of implementation, district union representatives and building/district administrators met to go over what worked and what did not work in the teacher evaluation system. There was no formal survey/focus group, other than the team who attended this meeting. A few logistical changes were made to the system. District 2: Teacher and Administrator Perceptions Teacher perceptions varied greatly among participants. Each interviewed teacher participant had a different view; administrators had a positive view. These similarities are visible below on Table 16, Table 17, and Table 18. Table 16 is organized by responses about the implementation’s acceptability and influence on morale. Table 17 describes specific and concrete questions about the implementation, including its most important part: what influence it had on perception, and what should have been included in it. Table 18 follows Noe’s attributes (reliability, relevance, specificity, strategic congruence, and acceptability). Table 16 is initial-coded like Table 15. Table 17 and 18’s responses are sequenced by respondent. First respondent is teacher union-representative; second respondent is teacher without an official union position; third response (only on questions of most important step, influence of implementation on perception, and what should have been included). 89 Table 16: D2 Teacher/ Administrator Survey Perceptions of Implementation’s Influence D2 Teacher Morale and Acceptability from Surveys In building, how did the changes influence morale? Implementation Surveys Implementation Documents Interviews Surveys-Do Teachers Remember the District Implementation Steps? 2/3- Neither improved or diminished morale (BA, DA) 1/3 - Diminished morale (UCSAT) ---- 3/3Mentioned stress about changes. (All) In district, how did the changes influence morale? Neither improved or diminished morale 2/3 (BA, DA) Diminished morale 1/3 (UCSAT) ---- 0/3Mentioned district morale Quote 1: “there was some stress with the new requirements and system, but that might be expected.” (NCSAT) Quote 2: “Responses vary. Some are stressed out, some are note affected much.” (CSAT) Quote 3: “Negatively, in that teachers are expected to put in many hours on their own evaluations, but are not given any addtional time” (CSAT) Quote 1: “I think teachers as a whole are split.... some see it as a way to develop their skills as a well-rounded educator and some see it as a means to rate teachers and punish” (NCSAT) Quote 2: “See #7 answer” (previous question) (CSAT) Quote 3: “I can't answer this I have no contact with teachers other than those in my building” (CSAT) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 90 Table 16 (cont’d) D2 Teacher Morale and Acceptability from Surveys Based on your experience overall, how acceptable have you found the district’s teacher evaluation system? Implementation Surveys Implementation Documents Interviews Surveys-Do Teachers Remember the District Implementation Steps? 2/3- Acceptable (BA, DA) 1/3- Unacceptable (UCSAT) N/A 2/3 Described the system as new and stressful, but acceptable. (BA, NCSAT) 1/3 Described the system as one in which no one buys in and can be seen as a way for the state to force districts to “get” teachers (UCSAT) 2/3- Acceptable (CSAT, NCSAT) 1/3 Unacceptable (CSAT) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 91 Table 17: District 2 Interview Perception Response Summary 1: Building Union Representative Question Most Important Part Implementation influence perception Should have been included, but was not Summarized Responses 1. Communication from administrators on how it impacted us 2. On-going training and communication from administration 3. Have enough time to describe system to teachers. Teachers had time to work with one another 1. More about relationship than implementation because the system is so flawed. 2. Lot of anxiety at first. It’s gotten better as people have learned the system. 3. Difficult at first. Communication has brought understanding and comfort with system. 1. A follow-up at the end of the year. 2. Nothing 3. More opportunities to run trial and error on the system. 2: Teacher 3: Administrator Quotes 1. “..The communication from our administrators about their intent for the evaluation system, and to feel like it is truly a tool to help us grow, rather than a punitive device to be used against us.” 2. “it was a way for us to communicate about evaluation and improve teacher performance as opposed to the way they grade teachers.” 3. “But using the terminology that you live in effective, you visit highly effective, you are not going to be highly effective in every single area.” 1. “The system is flawed, and everybody knows that.” 2. “It's easier this year, that's for sure.” 3. “I don't care what model you choose, there's always going to be a defensive side to the process because you are being evaluated.” 1. “I know some teachers didn't have their follow-up meetings last year for their evaluations. Everybody's just spread so thin that things just aren't being done consistently.” 2. “I haven't anything that comes to mind.” 3. “Maybe take some teachers and do a pilot program with it before we jumped into it.” 92 Table 18: District 2 Interview Perception Response Summary based on Noe’s Attributes 1: Building Union Representative Question Reliable Summarized Responses 1. No, inconsistent application 2. Yes 2: Teacher 3: Administrator Quotes 1. “word on the street is that it's not. I guess the fact that some people didn't meet with their evaluators last year. There's different numbers. We get observed in different ways. We get a different amount of warning. That's not quite consistent.” 2. “I would say it's the same.” Relevant 1. No. It does not count variation between students and teachers in any class 2. Yes, they do as well as they can without being experts in my field. I get more from when other teachers observe me and give me feedback. Specific 1. No, the expectations were not clear in the beginning and are not clear now. 1. “It doesn't take into account the variation between classes and students. You come to my fourth hour, it looks very different from my first hour. My classes look very different from her classes. It doesn't take into account those students that you're working with, and how maybe everybody's best is different.” 2. “I look for ways to develop my own skills as a teacher, my own performance evaluation by bringing in colleagues who teach what I do. They come in and work with me.” 1. “That gets back to the communication and clarity and having answers before we start things.” 2. “I do. I really do, I don't have much to add there but again it does what I expect that tool to do for me to help.” 2. Yes, as much as a nonsubject specific tool can. I get more from my colleagues. District goals 1. No. It feels like hoopjumping. That is not the district’s goal. 2. Yes, this new system pushes teachers to be excellent Acceptability 1. Mixed, it varies a lot from teacher to teacher. 2. Yes, it was fair and straightforward 3. Yes, they felt invested and comfortable in the process 1. “It was clear they (sic: administrators) didn't buy into what the State was saying.” 2. “We do what we needed to do, as opposed to, here's some areas you can challenge yourself to be truly excellent, and really contribute to the whole of what is our school, as opposed to just your target area.” 1. “So effective, how effective, is really no That varies a lot depending on the teacher. Some teachers feel targeted and so they feel it's not fair, they feel like certain pieces of it were, maybe they were observed on a day when students are naturally more rowdy. No, I would say, a lot of people don't feel it is fair.” 2. “I thought it was very fair. I liked the challenges that were submitted to me by the administrators through the process that they did. For me, it reflected what I do every day. Outside classroom, in the classroom, I thought it was valid.” 3. I think in the long run they were comfortable with that decision. They felt invested in the process.” 93 Teachers and administrators thought faculty meeting time to discuss evaluations with administrators and other teachers were most important in the implementation, as well as one-on-one meetings with administrators. The implementation itself was described as stressful, as both administrators and teachers tried to figure out a system that was new and a mandate from the state. In terms of what should have been included, there were mixed results. One participant thought nothing needed to be added. Another participant thought that a pilot would have been helpful. Another participant thought that some teachers missing their “end of the year” evaluation meetings was detrimental to the implementation (and should have been included for all teachers). Interview participant responses on reliability, relevance, specificity, alignment to district goals, and acceptability were mixed. The core subject area teacher thought the system was not fully developed, and the building administrators did not believe in what the state mandated the district to do. In addition, the area of student growth and the rankings of teachers were unclear and problematic. On the other hand, the non-core subject area teacher thought the implementation and the system were clear and an improvement from the previous system. S/he acknowledged that there was only a certain extent that an administrator, who did not teacher his/her subject area, could give specific and relevant feedback to him/her. This response pattern held true in responses to electronic surveys; however, the majority of the participants found the system acceptable. In terms of morale, responses were mixed. All participants described stress at the change in teacher evaluations because it was new. Several participants said the influence on morale and perception of acceptability varied from teacher to teacher. Depending on what score each teacher got, in comparison to what they thought their performance was, influenced their morale and perception of acceptability. Administrators rated the system as acceptable and not influencing morale. One participant opined that the overall tone of evaluations felt forced by the state, and the district administrators did their best to work with the system, despite not believing in its merits. District 2: Relationship between Implementation and Perception Like District 1, District 2 followed several of Grote’s implementation steps: formation of a “teacher evaluation team”, collaborative decision-making in the team, central office leadership/buy-in, informal feedback from teachers on the team, communication, training for teachers and administrators, a meeting to assess the teacher evaluation, and logistical adjustments to the teacher evaluation system. After the first year, there was no evidence of further meetings, focus groups, or surveys to identify unclear areas like student growth or achieving rankings. For one teacher interviewed, this continued lack of clarity caused the whole system to seem like it was focused on state compliance, rather than motivating teachers. Linking back to implementation, the agendas submitted seemed to focus on designing a system similar to the previous system and making it 94 compliant with the state mandate. A formal needs assessment was not on agendas or described in surveys or interviews. Teachers and administrators were not asked what they needed or wanted in the new system. If those questions were asked before the system was introduced, a more understandable system could have been implemented. The descriptions of stress and uncertainty could be attributed to a lack of pilot. Several teachers mentioned that this kind of stress was normal. The system was a change. Though change can be stressful, allowing teachers feedback through a needs assessment and testing out the system with a pilot could have reduced the anxiety. If the district had piloted a new system without the threat of negative rankings, it could have elicited feedback and adjusted the system to meet all the needs of the stakeholders. The district actually did collaborate with its union representatives throughout the process. It is unclear from the documents, teachers’ memories, and interview responses if those efforts were communicated to teachers throughout the district. Had the district described what was discussed as the system was being developed or explained that a team was meeting at the end of the first year to assess the new system, such communication could have provided solace to teachers and could have improved their overall perception of the system and its implementation. As it turned out, those efforts never were communicated, and therefore those outside the teacher evaluation team or unacquainted with anyone on it perceived that they didn’t occur. Several positive perceptions about the implementation also emerged from the interviews, surveys, and documents. The first was that the building faculty meetings, during the first year, focused significant time to discussing and understanding the system. Teachers and administrators appreciated the time and the discussions in those meetings. These built a sense of calm, if not overall acceptance, with the teachers interviewed and surveyed. A second perception, similar to that in District 1, was an overall trust in district and building administrators to do their best, even if the system was flawed. This was supported by the fact that the district did not have to make any major teacher lay-offs during the implementation. The implementation steps and perceptions are detailed in Table 10. 95 Table 19: District 2 Implementation and Influence on Perception of the System Implementation Steps/Lack of Implementation Steps Lack of formal needs assessment Lack of pilot Lack of communication about implementation Building faculty meetings dedicated to understanding the new teacher evaluation system Trust in district and building administrators Perceptions Influenced Teachers not on the teacher evaluation team did not feel connected to the process of developing the new teacher evaluation system, which caused stress. This also caused obstacles in the system that could have been identified and improved through a needs assessment. Lots of logistical issues and unanswered questions (that persist as the system continues to be used) could have been avoided, had the system been piloted Some teachers felt as if the evaluation criteria (in the areas of student growth and rankings) were a product of a state mandate, even though the district did collaborate with its union and make an effort to improve the system after the first year. The time spent at district and building meetings allowed teachers and administrators to better understand the system and how it would be used to evaluate teachers. This came through shared ideas. The trust developed through the meetings, as well as one-on-one conversations reduced the stress about the new evaluation system as it was being used in the first year of implementation. 96 District 3: Suburb Large District 3 is a large suburban district in Michigan. In this district, like Districts 1 and 2, there have been no layoffs based on the teacher evaluation system, as attrition and retirements have allowed the district to cut faculty based on lower funding levels and decreased student count over the last three years. No performance pay has been issued to highly-ranked teachers. District 3 is one of the biggest districts in the study. The implementation and perceptions are influenced by the fact that the impact of teacher rankings have been mostly a matter of pride for teachers and did not influence job action or compensation. District 3: Implementation District 3 used many of the same implementation steps as Districts 1 and 2. A major difference for District 3 is that the district hired a new lead administrator at the high school level. A new principal at the building level, without experience in the district, seemed to influence teachers’ perceptions of the implementation. There was also a major issue of “highly effective” status. The district initially said that it wanted participants to “apply” for this status. Based on feedback and a survey, the district, mid-year, changed this decision and allowed all teachers to be eligible without having to apply. The district also used an online system for teachers to place their work. Besides teachers who were not technologically proficient, the system itself was troublesome and slow. These two issues made the implementation different, though similar in steps. Table 19 below describes responses by participant about implementation. DA indicates District Administrator; BA indicates Building Administrator; UNCSAT indicates Union rep Non-Core Subject Area Teacher; CSAT indicates Core Subject Area Teacher; and NCSAT indicates Non-Core Subject Area Teacher. 97 Table 20: District 3 Implementation Survey Reponses D3 Grote’s Implementation Steps D3 Formation of Implementation Team How was the decision on the model made? Implementation Surveys Implementation Documents Interviews (One Building Union Representative, One Central Office Administrator, and One Building Principal) 3/3 Team-Union teacher representatives and building and district administrators (All) (Provided by Central Office Administrator) 3/3 Team discussed, but final decision was made by administration (All) N/A (Interviews with One Building Union Representative, One Building Administrator, and One HS Teacher) 2/3- Described team of teachers and administrators working on teacher evaluation system (BA, UNCSAT) 1/3- Described team working, but final decision was made by administration (BA) 1/3- Described teachers making recommendations that were not followed in the implementation (UNCSAT) Agendas from team meetings on teacher evaluation were submitted Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 98 Surveys-Do Teachers Remember the District Implementation Steps? (Surveys of two core subject teachers and one non-core subject teachers) 2/3 -Do not remember very well (CSAT, NCSAT) 1/3- Do not remember at all (CSAT) ---- Table 20 (cont’d) D3 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews Central office involvement 2/3- They led the process (DA, UNCSAT) 1/3- They were informed of the process (BA) Agendas from team meetings on teacher evaluation were submitted by central office administrator who led the process 2/3- Described central office administrator leading the team of teachers and administrators. (BA, UNCSAT) Focus group/Survey for needs assessment 1/3- Yes (DA) 2/3- No (BA, UNCSAT) N/A 2/3- Described team working with teachers. (BA, UNCSAT) Pilot 2/3- No (BA, DA) 1/3- Not sure (UNCSAT) N/A 0/3- Mentioned a pilot Communication Modes 3/3- Administrators explained in faculty meetings and district meetings (All) 1/3- Emails to all certified faculty and staff (DA) 1/3- Building and/or district leaders explained in union meetings (BA) 2/3- Central office administrators explained in district wide meetings (BA, UNCSAT) Descriptions of introduction, evaluation rubrics and descriptions, and updates on teacher evaluation system were submitted 3/3- Described district and building administrators explaining the new system at district and building meetings (All) 3/3- Described administrators describing teacher evaluation system in individual meetings with faculty members (All) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 99 Surveys-Do Teachers Remember the District Implementation Steps? 1/3- Remembers (CSAT) 1/3 Does not remember very well (NCSAT) 1/3 Does not remember at all (CSAT) 1/3 -Does not remember well (CSAT) 2/3- Do not remember at all (CSAT, NCSAT) 1/3 -Do not remember very well (CSAT) 2/3- Do not remember at all (CSAT, NCSAT) 3/3 Remember (All) Table 20 (cont’d) D3 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews Communication: How well was new system communicated to teachers? 2/3 Well (BA, DA) 1/3 Not well (UNCSAT) N/A Training for administrators 2/3-Through training by district employees and outside vendors (BA, DA) 3/3- Through written procedures (All) 2/3- By answering questions administrators had (DA, UNCSAT) 1/3- Other (please describe) “left to sink or swim” (UNCSAT) Documents about procedures and policies of teacher evaluation system were submitted. 2/3- Described a full initial implementation. The district initially said teachers had to apply for “highly effective” status. This was a point of contention. The resolution was that the district did not make people apply for “highly effective” status. (BA, UNCSAT) 2/3 Described difficulties with the online E-portfolio section of the evaluation. (BA, UNCSAT) 1/3- Described weekly meetings with all building level administrators to try to increase reliability of the teacher evaluation system. (UNCSAT) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 100 Surveys-Do Teachers Remember the District Implementation Steps? ----- 2/3 Remember (CSAT, NCSAT) 1/3 Does not remember very well (CSAT) (Trainings were grouped together on teacher survey) Table 20 (cont’d) D3 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews Modes of training for teachers 2/3- Through mandatory trainings/meetings before the year started (BA, UNCSAT) 1/3- Through emails describing the way teachers were expected to perform (BA) 2/3- Through emails with evaluation procedures and tools attached (DA, BA) 1/3- Yes (BA) 1/3- No (DA) 1/3- Not sure (UNCSAT) Attachments from emails with process and rubrics were submitted. Documents from training on EPortfolio was submitted 3/3-Described training for teachers at district and building meetings. (All) 3/3- Described emails with teacher evaluation documents (All) 3/3- Described individual meetings with teachers about teacher evaluation system (All) 0/3 Mentioned administrators being evaluated on how they evaluated teachers 2/3 Well (BA, DA) 1/3 Not well (UNCSAT) Documents with rubrics and training on EPortfolio were submitted Evaluators evaluated on their evaluations Teachers Understand How to Perform N/A Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 101 2/3- Described changes during the first year of implementation in two areas: domains assessed and “highly effective” status. (UNCSAT, CSAT) Surveys-Do Teachers Remember the District Implementation Steps? 2/3 Remember (CSAT, NCSAT) 1/3 Does not remember well (CSAT) (Trainings were grouped together on teacher survey) 1/3 Remembers (CSAT) 2/3 Do not remember at all (CSAT, NCSAT) ---- Table 20 (cont’d) D3 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews Surveys-Do Teachers Remember the District Implementation Steps? District solicited feedback on system 1/3- Yes, with a survey (DA) 1/3- Yes, with a focus group (BA) 1/3- No (UNCSAT) N/A-Because of the discrepancy with one participant mentioning a survey, district administration described a survey given about “applying for highly effective status.” That survey was not submitted. 2/3- Described the district’s decision about having teachers apply for “highly effective” status and then changing that decision based on teacher feedback (CSAT, UNCSAT) 2/3 Remember Very well (CSAT 2) 1/3 Remembers (NCSAT) Adjustment to system based on feedback 1/2- Provided more training for administrators/evaluators on performing evaluations (DA) 1/2- Not sure (UNCSAT) N/A 2/3- Described the district’s decision about having teachers apply for “highly effective” status and then changing that decision based on teacher feedback (CSAT, UNCSAT) 2/3 Remember (CSAT, NCSAT) 1/3 Do not remember very well (CSAT) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 102 District 3 formed a team, led by central office administrators. The team included teacher union representatives, teachers, building administrators, and district administrators. The “teacher evaluation team” provided feedback on the needs of the new system, but no formal survey or focus group was used. No pilot was used. The new evaluation system was explained by district and building level administrators in beginning-of-the-year meetings. Building level administrators further communicated and trained teachers in faculty meetings throughout the year. An electronic portfolio, in which teachers were to place their work, was a portion of the evaluation in which specialized training was offered consistently throughout the year and also was confusing and inefficient to users. A core subject area teacher who did not hold an official union position described working on the electronic portfolio this way: “You're trying to do some simple things like upload a resume and it took 20 minutes to figure out how to do it. It's just so simple things like that...it seems petty but it is time consuming when you're trying to get things done, and you can't do them the way you want them to be done. Then, really more direction, they want to have our freedom to do stuff, yet again...tell me what I need to do, and I'll do it.” Building administrators met weekly to improve consistency and reliability. This training of administrators, however, was not communicated to teachers. The communication of the basic system was clear to teachers, but the designation “highly effective” caused confusion, as the district first told teachers they had to apply for it and then reversed this decision. Another area of confusion was the domains on which teachers would be evaluated. They varied depending on who the evaluator was. One teacher thought s/he would be evaluated on one domain of teaching, but all four domains were evaluated at the end of the first year. S/he described finding out about the domains to be evaluated this way: “More like March. Maybe they told us. It didn't come to my realization until I was being evaluated at the end sitting down with my administrator and she was going through and saying, ‘Well, you're effective everywhere. I thought you were highly effective in these areas.’ I said ‘OK, that's good.’ She said ‘But you're going to have to show me proof that you're highly effective in these areas.’" The district did not seek formal feedback after initial usage through a survey or focus group. Small adjustments were made to the system through central office. The major change was the adjustment to the designation “highly qualified.” 103 District 3: Teacher and Administrator Perceptions Teacher perceptions were largely negative. Administrators’ perceptions were neutral. The differences between these perceptions are displayed below in Tables 21, 22, and 23. Table 21 is organized by responses about the implementation’s acceptability and influence on morale. Table 22 describes specific and concrete questions about the implementation, including its most important part: what influence it had on perception, and what should have been included in it. Table 23 follows Noe’s attributes (reliability, relevance, specificity, strategic congruence, and acceptability). Table 21 is initial-coded like Table 20. Table 22 and 23’s responses are sequenced by respondent. First respondent is teacher union-representative; second respondent is teacher without an official union position; third response (only on questions of most important step, influence of implementation on perception, and what should have been included). 104 Table 21: D3 Teacher/ Administrator Survey Perceptions of Implementation’s Influence D3 Teacher Morale and Acceptability from Surveys In building, how did the changes influence morale? Implementation Surveys Implementati on Documents Interviews Surveys-Do Teachers Remember the District Implementation Steps? 1/3- Neither improved or diminished morale (DA) 1/3- Diminished morale (BA) 1/3-Greatly diminished morale (UNCSAT) N/A 2/3- Described stress and disappointment with the overall system because some parts were unclear and some parts were unfair (CSAT, UNCSAT) 2/3- Described the short time of observations (one per teacher per year) as being inadequate to see all of what a teacher does. (CSAT, UNCSAT) 1/3- Described dissatisfaction by teachers about evaluation results (BA) Quote 1: “Negative. It has crushed morale. We are now under a microscope attempting to attain unattainable status of Highly Effective.” (NCSAT) Quote 2: “Very negatively, unfortunately...it seems the state is not requiring an eportfolio to be part of the teacher evaluation process, yet our district has an eportfolio as part of our evals. Consistent evals across evaluators is not happening. My school was #4 in the state of MI last year in a ranking of overall performance YET less than a handful of teachers were evaluated at the highest rating. When the district first implemented the new eval system, they did not even have a rubric for the highly efffective domains, kind of frustrating when as a professional, you want to do your best, yet we had no idea what they were looking for. It is hard to be evaluated on one lesson observation for 69 minutes, it is hard to hit every domain in one class period.” (CSAT) Quote 3: “Negative influence” (CSAT) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 105 Table 21 (cont’d) D3 Teacher Morale and Acceptability from Surveys In district, how did the changes influence morale? Implementation Surveys Implementation Documents Interviews 1/3- Neither improved or diminished morale (DA) 2/3- Diminished morale (BA, UNCSAT) N/A 1/3- Described district morale as being influenced by the lack of clarity on “highly effective” (UNCSAT) 2/3- Described a lack of reliability across buildings and administrators as being a cause of negative morale. (CSAT, UNCSAT) Based on your experience overall, how acceptable have you found the district’s teacher evaluation system? 1/3 Acceptable (DA) 1/3 Neutral (BA) 1/3 Unacceptable (UNCSAT) N/A 2/3 Described the system as unacceptable based on lack of clarity on “highly effective” and observations not being an adequate amount of time. (CSAT, UNCSAT) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 106 Surveys-Do Teachers Remember the District Implementation Steps? Quote 1: “Negative. It's what most of the conversation centers on when you talk to others in the district. Our district also has the issue of buildings not being consistent between each other. One building has MANY highly effective staff members after the evaluation process and others have 1 or 2. It sucks.” (NCSAT) Quote 2: “Same as above in #7, not positively...many teachers talk across buildings about who got a highly effective rating and who didn't, who evaluated who, etc..” (CSAT) Quote 3: “Negative influence” (CSAT) 3/3 Unacceptable (All) Table 22: District 3 Interview Perception Response Summary 1: Building Union Representative Question Most Important Part Implementation influence perception Should have been included, but was not Responses 1. First initial sit-down with administrator where they told you what you were going to need to do. 2. Training so all administrators are on the same page 3. Faculty meetings with descriptions and explanations. 1. Hurt morale. People took it personally. 2. Paranoia because the criteria (including Eportfolios) was not clear 3. Hard to say. There is a lack of trust. 1. Clarity on “highly effective” 2. More time to implement things 3. Clarity on “highly effective” 2: Teacher 3: Administrator Quotes 1. “I think that your first initial sit-down with your administrator was probably key, because there you were talking with, whoever. It didn't matter. When you sat down with that person, they said, "OK. Here's what I want from you. Here's what I really want. I want you to write a lesson plan.” 2. “That was something that I think we have one administrator who is very good at that, and the rest are learning. They realize that, that they needed more training, and so they're working on that, but that doesn't change the past.” 3. “..was trying to help them understand that framework. We were studying the framework.” 1. “For other people, it was a big deal. Even to this day, there are people in this building, morale wise, that are very angry, that are very irritated, they're very hurt, that they weren't deemed highly effective last year. It runs right to the top.” 2. “The overall system, a lot of teachers were, "This is setting up to get rid of teachers." Yes, we should be held accountable, but the criteria. There's a lot of paranoia. I'm not real paranoid, because I'm a pretty good teacher, I've been doing this for a number of years, and I've been told I'm a good teacher, so it's not going to change what I do.” 3. “I don't think they trust this place (central administration building) a whole lot.” 1. “As far as I'm concerned, the district really shot themselves in the foot. I'm not sure if this was only in high school or if this was across the district. I don't know. With the whole highly effective piece, where we'd got to apply for it.” 2. “I think, more time to implement things. We were supposed to do this E portfolio. Here's the website. They did offer trainings once a month where you could sit down, and have an administrator...it wasn't even necessarily developed in the E portfolio itself, but it was actually working with the web program itself was very difficult, the website program.” 3. “I think probably that idea of the how the ranking is working. I don't think everybody understood that if you got a highly effectives into effectives, I don't think that was really clear made.” 107 Table 23: District 3 Interview Perception Response Summary based on Noe’s Attributes 1: Building Union Representative 2: Teacher Question Reliable Responses 1. No, inconsistency between administrators 2. No, background in teaching lacking in evaluators Relevant 1. Yes, but it doesn’t count the affective side of teaching 2. No, it is too brief of an observation to try to see all the things on the rubric and everything is not included Specific 1. Yes, better this year than last 2. Yes, getting better 108 3: Administrator Quotes 1. “No. Absolutely not. It's all over the place.” 2. “Within the building itself the background of the administrators, what training they've had. Have they been in a classroom and how long has it been since they've been actually teaching? You don't have to be a good hockey player to be a good coach, I understand that, you know what to look for, but you've hopefully played hockey. But to base your training and evaluating someone, you have to put yourself in their shoes.” 1. A kid that's been gone for two or three days, he's behind. The first thing I asked was, "Where have you been? What do I have to do to get you caught up?" There's that compassion for that kid. At first I might give that kid a hard time, but say, "If you want to be here, if you want to succeed, you missed this demo, you missed this lab." You try to drag the kid along anyway. Got to get him back up. 2. “Relevant in all of them? No. In many of them, yes. Like I said, there is dialogue that I feel I can go to my administrators, any of them, and discuss my concerns. But once again, it's just a snapshot of what's happened over an entire year.” 1. “More so this year rather than last. People are getting a little better grasp of the Daniels system and the evaluation tool. We're getting better able to know about Daniels.” 2. “It's getting better [laughs] than it was. We're more familiar with the process. I don't think it's extremely clear, but it's getting better, if that makes sense.” Table 23 (cont’d) 1: Building Union Representative 2: Teacher Question District goals Responses 1. Yes, the district had this in line. 2. Yes, but there is a lack of support. Acceptability 1. No, because teachers wanted to be “highly effective” 2. No, because of the variation between administrators 3. No, the way “highly effective” was designated caused problems. 3: Administrator Quotes 1. “The district had their ducks in a row as far as that was concerned.” 2. “I don't know. The district goals are pretty broad and what we tried to do this year as a department is, we look at the district goals and the school goals and as well as our own departmental goals and try to line them up. “One of the goals is to concentrate on the lower thirty per cent of population, our own departmental goals and personal goals are aligned with that as well. The problem is, put your money where your mouth is, when you need resources or things to help this lower third and that's the district goal but the district's not that willing to help fund that.” 1. “I don't care. I didn't get too twisted up, but there are other teachers that I had also explained to you that were very, very upset with their evaluators and evaluations. They're very good teachers.” 2. “Once again the variation between administrator to administrator and building to building, the competence of the administrators in evaluating, I think it still needs to be worked on. I think we can get there, but they have to allow for variation as we do. If there is a problem time to work on it and give training or something, if there is a problem, give time to work on that and correct it. If it's not corrected then I can see there will be repercussions.” 3. “Do I think they think it's fair? Probably not.” Important parts of the implementation described by participants were the faculty meetings dedicated to understanding the system and “first sit-downs” with administrators so teachers and administrators could reach clarity on what will be expected. All three interview participants said the implementation was negative because of a lack of trust and a lack of clarity on E-Portfolios and the “highly effective” designation. Areas that should have been included were a pilot and clarity on how to achieve the “highly effective” ranking. Perceptions of reliability, relevance, specificity, alignment to district goals, and acceptability varied. Participants split evenly on whether or not the evaluation system, as it was used, contained these characteristics. A common theme in all of the comments was variance among evaluators. Those teachers who felt they had knowledgeable evaluators described the system as fair, if a little unclear. Those teachers who did not have knowledgeable evaluators felt as if the system was not reliable, specific, relevant, aligned to district goals, or acceptable. All three teachers surveyed said that the system was not acceptable, while administrators said that it was. 109 In terms of morale, both at the building and district level, the system was described as having a negative influence on morale among teachers. Administrators were split between a negative influence and a neutral influence. Main reasons for this negative influence included lack of consistency among administrators both within and between buildings. The designation “highly effective”, and all of the evaluation being based on a single observation, were cited as reasons for the negative influence also. In terms of morale, all teachers described a negative influence on morale. Administrators were split on the issue of morale, between negative and neutral. District 3: Relationship between Implementation and Perception The relationship between implementation and perception was clear in several areas. The first was the “highly effective” rating. To begin the first year of implementation, the district told teachers they would have to apply for this designation. The application included a lot of additional work to show they should be considered for the ranking. Even if a teacher applied, there was no certainty the teacher would receive the rating desired. This caused significant consternation among the teachers. As a result, the district surveyed teachers on the process for obtaining the “highly effective” ranking. Based on this feedback, the district changed its stance on the “highly effective” designation. Teachers would no longer have to apply, but the work that needed to be deemed “highly effective” remained unchanged. In terms of implementation, the district’s position on “highly effective”, though later adopted, could have been perceived as clearer and more accessible had the district used a pilot. As it turned out, without the pilot, the district had to change its policy mid-year. The overall perception of teachers, even after the change, was that the new teacher evaluation system required more work from teachers for the chance at a “highly effective” ranking. This led teachers to a negative perception of the whole implementation scenario. The second perception of teachers which could be related back to implementation was a plan of communication. Though the central office administrator who led the process did email teachers throughout the district, there was no formal communication plan. The process, discussion, and decisions of the teacher evaluation team were communicated through documents and rubrics. Identification of the needs of teachers and how those needs would be met, ultimately benefitting students, were missing. Because of these deficiencies, questions about student growth, the weight and usage of the electronic portfolio, and even the domains to be assessed were not clear to some teachers. Those teachers who were on the teacher evaluation team may have been clear, but not so the other teachers throughout the building. At the high 110 school level, the administrative team met weekly to tune their responses and improve inter-rater reliability; however, teachers did not know this. The district did not publicize the steps of any plan to make the system work, at least any plan of which the participant teachers in the study were aware. A third perception was that there were major differences among administrators. Major differences included end-of-the-year meetings (some teachers did not have these with the administrators who evaluated them), the length of observations, the length of the evaluations based on the observations, and the length of the conferences after observations. All were perceived to vary greatly among administrators. Equally important, full evaluations, based on all factors, were said to vary among administrators, both in the building and in the district. Going into the system’s second year, the perceptions of variance continued. Two areas of implementation relate to these concerns. The first is evaluations of evaluators. If teachers believed their administrators were being evaluated on the evaluations they gave to teachers, it would give them hope that even if there was some variance, someone at central office was reading and evaluating the performance appraisals given at the building level. The second area of implementation related to these concerns was communication. If the district could have communicated the trainings it provided to administrators or even permitted teachers to participate in these trainings, it could have improved teacher perception of reliability. It also could have been communicated that there are going to be differences in administrators. The district could state that though each administrator has a different background, the final result was going to be aligned and checked by a central office administrator. The concern about reliability also relates to an implementation factor outside of Grote’s list of steps. Administration in the building changed in the middle of implementation. Several of the new administrators were new to the district and to their positions. Because of this fact, a formal plan of communication, evaluations of the performance appraisals, and a pilot were even more important. The final perception related to implementation was feedback on the system. The district did seek teacher feedback on the controversial issue of “highly effective” rankings, but there was no formal end-of-the-year meeting/focus group or survey. The system did change, but it was not based on teacher feedback or needs. This led to teachers in the study feeling like the system did not include their perspectives, which they were ready to express. Related to this mindset was the perception of one teacher in the study who said that the initial meetings of the teacher evaluation team gathered feedback, but that it was not utilized in implementation. 111 On the positive side of perceptions, the district did offer training on a stumbling block for teachers: the electronic portfolio. These trainings were offered weekly throughout the year. The perception of teachers was that this training was helpful and allowed them to understand the portfolio system better. Another positive perception was that teachers appreciated the dedication of faculty meetings throughout the year with activities and discussions to help teachers understand the observation framework. The administrator participant felt that teachers may have grown weary of talking about the observation framework. According to both teacher interview participants, this dedication of faculty meetings led teachers to use their own time, outside of meetings, to work on fulfilling the expectations of the teacher evaluation system. The implementation steps and perceptions are detailed in Table 15 below. 112 Table 24: District 3 Implementation and Influence on Perception of the System Implementation Steps/Lack of Implementation Steps Lack of clarity about “highly effective” designation Lack of formal communication plan Lack of evaluations of administrators Lack of ongoing feedback on the teacher evaluation system Training for teachers on difficult parts of the evaluation system Faculty meetings dedicated to understanding the observation framework Perceptions Influenced Both teachers and administrators described the initial application process and subsequent changes caused confusion. A pilot of the system could have included and then evaluated the application process for “highly effective” without causing the perception among teachers that the district was asking much more from teachers for the chance at a “highly effective” rating. The district did communicate with teachers. However, it did many things like meeting (as administrators) to improve inter-rater reliability, surveying teachers on “highly effective”, and possibly more. Without a plan, input from teachers, if used, was not known by teachers throughout the building. Teacher’s lack of knowledge and understanding of positive district steps seemed to influence teacher perception negatively. Some teachers felt that the system was not reliable among administrators. Without a plan for “checking” what how administrators were evaluating teachers, teachers felt that different administrators in the building took different approaches and gave different rankings. This seemed to influence teacher perception of reliability negatively. Though the district did assess teacher perceptions about “highly effective” designations, it did not return to this practice at the end of the first year of implementation. This negatively influenced teacher perceptions of acceptability. The teachers included in the study felt they had suggestions to improve the system, but they did not have a formal way to give these suggestions to the district. The district offered training on the electronic portfolio. Teachers positively perceived this effort on the part of the district. Teachers saw the district as seeing a need with the difficulty of the new system and working with teachers to help figure out the system. The building administrators focused faculty meetings on the understanding and usage of the observation framework. This allowed teachers to be clear on the expectations in the system. Teachers perceived this as important and positively perceived this effort. This dedicated district time led teachers to give their own time to meet together and work on meeting district expectations. 113 District 4: Medium Rural Fringe District 4 is described as a medium rural-fringe district in Michigan. In this district, there have been layoffs based on the teacher evaluation system. In addition, performance pay, considerable in the initial year of implementation and under $100 in following years, have been issued to highly-ranked teachers. The implementation and perceptions are influenced by the fact that the district both laid off teachers and distributed performance pay based on the teacher evaluation system. District 4: Implementation District 4 implemented the teacher evaluation system using many of the Grote’s steps. It submitted many documents of their process. Unique to District 4 was the building of the teacher evaluation system into their school improvement plan. District 4, like District 3, had a new building administrator from outside the district take over during the implementation. In addition, District 4 laid teachers off during the second year of implementation and also gave teachers performance pay if they achieved “highly effective” rankings, though the amount was under $100. Table 24 below describes responses by participant. DA indicates District Administrator; BA indicates Building Administrator; UCSAT indicates Union rep Core Subject Area Teacher; CSAT indicates Core Subject Area Teacher; and NCSAT indicates Non-Core Subject Area Teacher. 114 Table 25: District 4 Implementation Survey Reponses D4 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews Sources (One Building Union Representative, One Central Office Administrator, and One Building Principal) 3/3 –Team-Building and district administrators, union representatives, and teacher representatives (All) 2/3 Cooperatively, with stakeholders on team (DA, UCSAT) 1/3 Team discussed, but final decision was made by administration (BA) 2/3- Led the process (DA, BA) 1/3- Were informed of the process (UCSAT) (Provided by Central Office Administrator) (Interviews with One Building Union Representative, One Building Administrator, and One HS Teacher) List of participants was submitted 3/3- Described a team of teachers and administrators working on the teacher evaluation system. (All) 1/3 Remember Very well 2/3 Remember well Agendas of all meetings of the teacher evaluation team were submitted 3/3- Described a team of teachers and administrators working together. There were no direct statements on the nature of the decision. (All) ---- Agendas and documents were submitted by central office administrator who led the process 0/3- Mentioned central office involvement in the implementation process. (All) 2/3 Remember Very well (CSAT, NCSAT) 1/3 Remember well (CSAT) Formation of Implementation Team How was the decision on the model made? Central office involvement Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 115 Surveys-Do Teachers Remember the District Implementation Steps? (Surveys of two core subject teachers and one non-core subject teachers) Table 25 (cont’d) D4 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews Focus group/Survey for needs assessment 1/3- Yes (BA) 1/3- No (DA) 1/3- Not sure (UCSAT) Agendas of all meetings of the teacher evaluation team were submitted Pilot 3/3- No (All) N/A 0/3- Mentioned a survey or formal method of assessing the needs of the district. 3/3- Described a team of teachers and administrators working together on the district’s teacher evaluation system. (All) 0/3- Described a pilot. Communication 2/3- Administrators explained in faculty meetings (DA, BA) 1/3- Emails to all certified faculty and staff (BA) 2/3- Building and/or district leaders explained in union meetings (BA, UCSAT) 2/3- Central office administrators explained in district wide meetings (DA, BA) Rubrics for all certified faculty and staff were submitted Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 116 1/3- Described the teacher evaluation process being embedded in the School Improvement Plan. (BA) 3/3- Described building administrators going over rubrics and systems in beginning of the year meetings. (All) Surveys-Do Teachers Remember the District Implementation Steps? 1/3 Remembers well (CSAT) 1/3 Remembers (NCSAT) 1/3 Does not remember very well (CSAT) 1/3 Remember well (NCSAT) 1/3 Remember (CSAT) 1/3 Do Not Remember At All (CSAT) 1/3 Remember well (NCSAT) 2/3 Remember (CSAT 2) Table 25 (cont’d) D4 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews Communication: How well was new system communicated to teachers? 3/3 Well (All) Rubrics for all certified faculty and staff were submitted 1/3- Described being unclear on how the use of data applied to his/her subject area. (NCSAT) 1/3- Described being unclear on what the target was. “Highly effective” was linked to performance pay, but was not clear. (UCSAT) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 117 Surveys-Do Teachers Remember the District Implementation Steps? ---- Table 25 (cont’d) D4 Grote’s Implementation Steps Training for administrators Modes of training for teachers Implementation Surveys Implementation Documents Interviews 3/3- Through providing Agendas of administrators with training for written documents that described administrators the new were submitted procedures (All) 2/3- Through answering questions administrators had about the procedures (DA, BA) 1/3- Through training by district employees (DA) 1/3- Through training by outside vendors (DA) Other (please describe) Quote 1: “Routine meetings aimed at creating a balanced approach amongst administrators” (BA) Quote 2: “Read Danielson Book” (DA) 1/3- Through emails Rubrics and describing the way processes for teachers were expected to evaluating perform (BA) teachers were 1/3- Through emails with submitted evaluation procedures and tools attached (BA) 2/3- Other (please describe): Quote 1: “No training” (UCSAT) Quote 2: “Discussion of Rubric, Individual Goal Meetings” (DA) 0/3Described training for administrat ors (All) 118 3/3Described meetings between teachers and administrat ors (All) 3/3Described beginning of the year meetings where building administrat ors explained rubrics and process. (All) Surveys-Do Teachers Remember the District Implementation Steps? 1/3 Remembers well (CSAT) 1/3 Remembers (NCSAT) 1/3 Does not remember at all (CSAT) (Teacher and Administrator Training were grouped together on the survey) 1/3 Remembers well (CSAT) 1/3 Remembers (NCSAT) 1/3 Does not remember at all (CSAT) (Teacher and Administrator Training were grouped together on the survey) Table 25 (cont’d) D4 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews Evaluators evaluated on their evaluations 1/3- Yes (DA) 2/3- No (BA, UCSAT) 0/3 Mentioned administrators being evaluated on how they evaluated teachers Teachers Understand How to Perform 2/3 Well (BA, DA) 1/3 Not well (UCSAT) Administrative evaluations were submitted. They include a rubric on supervision which gives clear expectations for administrators to evaluate faculty/staff Rubrics and processes for evaluating teachers were submitted District solicited feedback on system 2/3- Yes, with a focus group (DA, BA) 1/3- No (UCSAT) Agendas of meetings with teacher evaluation team were submitted Adjustment to system based on feedback 1/2- Provided more training for administrators/evaluators on performing evaluations (BA) 1/2- Other (please explain): “Changed Process Slightly” (DA) N/A Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 119 2/3- Described clear teacher expectations (BA, NCSAT) 1/3- Described being unclear on how to perform (UCSAT) 0/3 Mentioned the district soliciting feedback from teachers on the teacher evaluation system 0/3- Described adjustments to teacher evaluation system Surveys-Do Teachers Remember the District Implementation Steps? 1/3 Remembers well (CSAT) 1/3 Remembers (NCSAT) 1/3 Does not remember at all (CSAT) ----- 1/3 Remembers well (CSAT) 2/3 Do not remember at all (CSAT, NCSAT) 1/3 Remembers well (CSAT) 1/3 Remembers (CSAT) 1/3 Does not remember very well (NCSAT) The district formed a team of teachers, union representatives, building administrators, and district administrators. A central office administrator led the committee. Needs for the system were determined by the “teacher evaluation team.” There was no survey or formal focus group to provide the needs assessment. There was no pilot; however, teachers were told no teachers would be getting “highly effective” ratings in the first year, but it was reported that some administrators in the district did issue these rankings, and those teachers were awarded performance pay. This was how one core subject area teacher with a building union position described the district leader’s communication about “highly effective”: “I will say this, the first year of implementing it our Superintendent basically said that, ‘No one is going to get highly effective this year.’ That was just left at that.” At the building level, the teacher evaluation implementation was a part of the School Improvement Plan. The initial communication and training for teachers was provided at district and building meetings. Further communication occurred in one-on-one meetings with teachers and at faculty meetings. Administrators at the building level met regularly to tune their evaluations. These meetings/trainings were not communicated to teachers. On the administrator evaluation form submitted, there is a specific area about how administrators evaluate their teachers and staff; however, evaluators being evaluated was not communicated to teachers, even though it was a part of the administrator evaluation system. After the initial usage of the system, no surveys or focus groups were used to gain feedback about the system. Based on informal feedback, the system was changed slightly. District 4: Teacher and Administrator Perceptions Teacher perceptions generally were mixed. Overall, administrators’ perceptions were positive. The differences between these perceptions are displayed below in Tables 26, 27, and 28. Table 26 is organized by responses about the implementation’s acceptability and influence on morale. Table 27 describes specific and concrete questions about the implementation, including its most important part: what influence it had on perception, and what should have been included in it. Table 28 follows Noe’s attributes (reliability, relevance, specificity, strategic congruence, and acceptability). Table 26 is initial-coded like Table 25. Table 27 and 28’s responses are sequenced by respondent. First respondent is teacher union-representative; second respondent is teacher without an official union position; third response (only on questions of most important step, influence of implementation on perception, and what should have been included). 120 Table 26: D4 Teacher/ Administrator Survey Perceptions of Implementation’s Influence D4 Teacher Morale and Acceptability from Surveys In building, how did the changes influence morale? Implementation Surveys Implementation Documents Interviews Surveys-Do Teachers Remember the District Implementation Steps? 1/3 Improved morale (DA) 1/3 Neither improved or diminished morale (BA) 1/3 Diminished morale (UCSAT) N/A 2/3Described teachers being stressed about the new evaluation system (BA, UCSAT) Quote 1: “I am a teacher in the high school. In our first year, no one received a "highly effective" rating as the district lawyers told the superintendent that would make it hard to remove them from their position. That clearly impacted morale. In year two, four teachers in our building received highly effective ratings; almost everyone else was rated effective. Again, that caused a problem with morale as some teachers receiving an effective rating were clearly more dedicated and effective than others receiving the rating. I think that, as a whole, teachers are less dedicated to professional areas like "professional development" and " serving the district on committees" as they can receive an effective rating without demonstrating these areas of professionalism. As they feel it is highly unlikely they'll receive a highly effective rating anyway, they have decreased in their involvement in district matters beyond the school day.” (CSAT) Quote 2: “It has created a great deal of stress since the larger burden of the evaluation is placed upon the teacher.” (NCSAT) Quote 3: “The morale is OK. No question it was lower than before the implementation though. People feel like we are under fire and many of the areas are not welldefined.” (CSAT) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 121 Table 26 (cont’d) D4 Teacher Morale and Acceptability from Surveys Implementation Surveys Implement ation Documents Interviews Surveys-Do Teachers Remember the District Implementation Steps? In district, how did the changes influence morale? 1/3 Improved morale (DA) 1/3 Neither improved or diminished morale (BA) 1/3 Diminished morale (UCSAT) N/A 0/3- Mentioned district morale in their interview Based on your experience overall, how acceptable have you found the district’s teacher evaluation system? 2/3 Acceptable (DA, BA) 1/3 Unacceptable (UCSAT) N/A 2/3- Yes. It is not perfect, but acceptable. (BA, NCSAT) 1/3Unacceptable, because “highly acceptable” only was accessible if someone wanted to go for it (UCSAT) Quote 1: “I truly can't speak for the other buildings. I have heard that the middle school principal awards highly effective ratings to those teachers who support him and his initiatives. This does not surprise me. I suspect many administrators do this.” (CSAT) Quote 2: “Not in a positive way.” (NCSAT) Quote 3: “The morale as a whole in the district is lower than before. I have seen people move on and again, feel like they are under fire for trict doing their jobs.” (CSAT) 1/3 Acceptable (CSAT) 2/3 Unacceptable (CSAT, NCSAT) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 122 Table 27: District 4 Interview Perception Response Summary 1: Building Union Representative 2: Teacher 3: Administrator Question Most Important Part Responses 1. Understanding what the target is 2. Teachers understanding what they would be evaluated on 3. Making it a part of the school improvement plan to communicate it to faculty Quotes 1. “Just clearly defining what the target is. What do the evaluators want of the data portfolio in the end? Then the specific percentages of how it's going to break down. Like what we're going to be evaluated on.” 2. “They talked about it a lot and they made sure that we understood. We had the first year to do it as a trial run. This is practice.” 3. “The School Improvement Plan is trying to get data to drive instruction. It was sort of a new concept. Our teachers, for the most part, they're good teachers. They have looked at data. They didn't really have a formal plan. Nothing was really organized. It was trying to get that system in place.” Implementation influence perception 1. Yes. Teachers did not know what the target was. 2. Yes. You are accountable to show students are growing. 3. It was okay. Teachers say it was not a “gotcha” system. Should have been included, but was not 1. Clarity on what teachers were supposed to do. 2. How to show student growth 3. Consistency amongst evaluators 1. “For teachers in general it was frustrating because again you didn't really know what the target was. We're so busy. We just want to know, so we can just take care of it. The perception was that it was unorganized, that nobody really knew exactly what to do.” 2. ” Basically my district's message to me was, ‘I don't care how you demonstrate it, but I better be able to see, that this is where this kid was in September, this is where he is in January.’” 3. “Approaches are trying have been conversations and make it more about improving your own practice.” 1. “Just wondered, is this going to be consistent from one teacher to the next?” 2. “I just need to be able to see something, even if you don't fit in to a State Model that is workable for you as a marginalized teacher.” 3. “I think certainly one of the weaknesses is the consistency and evaluation of the evaluators.” 123 Table 28: District 4 Interview Perception Response Summary based on Noe’s Attributes 1: Building Union Representative Question Reliable Responses 1. No, expectations were not clear across teachers 2. No, highly effective ratings would not be given out or it would cost the teacher their job Relevant 1. No, because a teacher’s job is during contracted time. 2. Yes, because the administrator stops by and watches the class Specific 1. No, evaluations were not specific 2. Yes, peer teachers and administrators helped us to understand 1. No, not more than before. 2. Yes, expectations are clear, top down Strategic Congruence to District goals Acceptability 1. No, because teachers had to say they were going for highly effective if that is what they wanted in the second year. 2. It depended on the experience level of the teacher 3. Yes, because administration was open and was not trying to catch teachers. 2: Teacher 3: Administrator Quotes 1. “It wasn't very clear cut. The state left it too wide open for districts, for interpretation.” 2. “You don't want to get top of the line in everything. That's a serious, and I'm not sure that's a superintendent issue, because if that's a reality, if that means that a really, really outstanding teacher may get the same rating as someone who's pretty good.” 1. “Because in our minds, what's relevant is what you do during your, let's be straightforward, the contracted time. That's when you should be evaluated, your performance, when you're interacting with the kids, for us, it's 7:30 until 2:10.” 2. “I think a picture is worth 1,000 words. You can capture whether or not a teacher has control of their classroom and is monitoring things like bullying and has direct conversations with kids, that kids feel included even if they're in the margins of the classroom. I think that the assessment directs administrators to be in a classroom.” 1. “If you asked a teacher that, you probably would get a different response in terms of how that's supposed to be met.” 2. “It's easy to grab people and make sure we're all being compliant or make sure we get help.” 1. “I think the motivation is more about not wanting to get ineffective and lose your job.” 2. “I think there's a good deal of consistency across the district in terms of expectations and top-down.” 1. “At the beginning of our evaluations ,we were asked what we were going to go for. Somebody could have said, ‘Well, I just want to go for effective’ but they could have demonstrated a highly-effective. if you just said you were going for effective then it's just what you would have gotten. 2. “As I stood back and looked, I saw younger teachers panicking, mid-ranged teachers grumbling about more work to do and not enough time and not enough compensation, and older teachers saying, ‘Oh my gosh, I don't want to learn this. The slope is too high.’” 3. “I think that probably quite a bit because we were trying to be as open as possible with it.” The most important part of the implementation was making it clear to teachers (through the School Improvement Plan) what was expected of them. All three participants felt the implementation was important. Two of the three interview participants felt the building and district made things clear. One teacher participant stated that the lack of clarity on self- 124 evaluation and “highly effective” status should have been included. Another teacher mentioned clarity for all teachers on how to show student growth should have been included. Interview participants agreed that reliability was difficult, as evaluators varied in their approach to the system. Lack of direction from the state was also cited as a problem with reliability. Perceptions varied in participant responses on specificity, relevance, alignment to district goals, and acceptability. Two of the three teacher survey participants found the system unacceptable. Administrator participants felt that teachers found the system acceptable. In terms of morale, reaction was mixed. A majority of the participants said morale was hurt because of a lack of consistency (within the building and the district), an overall attack on teachers by the state, and the designation “highly effective.” Administrator participants felt there was no influence or a positive influence on teacher morale. Two interview participants, one teacher and one administrator, described the district’s size, smaller than surrounding districts. They detailed teachers working with one another to understand and to learn the new system. The same two interview participants also stated that there is a common understanding of customer service and focusing on students in the district. In addition, two interview participants described the observations as straightforward, if not reliable. Teachers said the district communicated the observation expectations clearly. The district used unplanned teacher observations as a part of their framework. One teacher described those as especially telling, as the students’ engagement during those observations really told administrators what they needed to know about a teacher’s performance. One teacher described a concern with the way expectations were implemented. If teachers wanted to reach a “highly qualified” designation, they had to state this at their beginning-of-the-year meeting with the administrator. The teacher’s perception was that if a teacher did not say s/he wanted a “highly effective” rating, the highest s/he could get was “effective.” One core subject area teacher with an official union position described the decision at the beginning of the year this way: “I know my approach, one of the reasons, like last year, just say. I just said, ‘You know what, I'm just going to go for effective.’ Do I feel like I could go for highly effective? Yes, which I said I thought maybe I would try for this year just to see how it would go. I would say that one of the reasons why I didn't even try for it last year was because unlike based on the seniority, we knew that there were potentially going to be some layoffs. I think if I was a lone man on the totem pole I probably would have done it, because it gives you that security, I guess.” 125 The lack of reliability across the building and the district was also mentioned. One teacher understood there would be no “highly effective” ratings the first year, yet there were. Performance pay was distributed, in this teacher’s eyes, based on these ratings that were not going to be given. Lay-offs based on performance appraisals, along with performance pay, gave the system a different tone than in Districts 1, 2, and 3. Lay-offs and recalls are mandated to be dictated by the outcomes of the teacher evaluation system, and District 4 followed the law. Performance pay is mandated to be a part of the teacher evaluation system, and the district followed the state law in this area, too. Performance pay was mentioned by only one of the three interview participants. The amount of the pay was so minimal that the teacher felt the amount of work to reach “highly qualified” was not worth the money. 126 District 4: Relationship between Implementation and Perception The relationship between implementation and perception was displayed in several areas in District 4. The district followed many of Grote’s implementation steps. They built the teacher evaluation system’s implementation into their school improvement plan. They had a collaborative team to design the system. District 4 had evaluations of teachers as a formal part of their administrators’ evaluations. The first perception of teachers was that “highly effective” was unreachable. According to their perceptions, a teacher had to determine in the first few weeks of school that s/he wanted to be considered for “highly qualified.” The district followed through with performance pay, so this decision had slight monetary implications. The idea that going for “highly qualified” was not worth the money was mentioned by two teachers. The perception that one’s quest for the highest level of rankings requires an early decision was problematic to the teacher who described it. In her/his opinion, performance goes on through the year. If that performance qualifies a teacher for “highly qualified” status, then that is fine. Having to decide during the first few weeks of school if the teacher is going to go for it creates two tiers of teachers, those who are and those who are not. This related to the acceptability of the system as a whole. In terms of implementation, a formal survey or assessment of the system after the first year could have brought these concerns to the district’s attention. Although it might just be a perception of several teachers, it seemed to de-motivate them and hurt the overall system’s acceptability. A pilot is foundational to an effective implementation of a performance appraisal system. The variance among evaluators, the issues of student growth in non-core subject areas, and the “highly qualified” confusion could be learned by all users and clarified. If the district had run a pilot program, these issues could have been identified and addressed. Because there was no pilot, any systemic problems were left unresolved, negatively influencing teacher morale. The dissonance between what the district said and what it did relates back to a formal communication plan. In addition, the variance between administrators also relates to a communication plan. If teachers knew administrators were evaluated on their appraisals, it might have given those teachers hope that central office administrators were reading the evaluations and checking them for reliability, relevance, and specificity. Building administrators met regularly to tune interrater reliability, but no teachers knew that. With a formal communication plan, these efforts by the district would not have gone unnoticed. Positive perceptions were around the district’s teachers working together and spending time understanding the new teacher evaluation system and finding ways to meet the new expectations. Though no faculty meeting time was dedicated, 127 teachers took it upon themselves to work with each other on the new system. Most of this work focused on student growth data. Another positive perception was that some of the teachers in the study said that the observations, as the district was doing them, provided administrators with a clear picture of what was occurring in a teacher’s classroom. The building administrator also felt these “drop-in” observations provided an opportunity for clear feedback and evaluation of a teacher’s performance. The implementation steps and perceptions are summarized in Table 28. 128 Table 29: District 4 Implementation and Influence on Perception of the System Implementation Steps/Lack of Implementation Steps Lack of clarity about “highly effective” designation Lack of formal communication plan Lack of reliability among administrators Teachers spending time with each other to understand teacher evaluation system Drop-in observations were effective in showing a teacher’s performance Perceptions Influenced Several teachers thought that “highly qualified” was too much work. One teacher described having to decide if one wanted to “go for” the “highly qualified” status during the first few weeks of school. The usage of performance pay and the necessity of lay-offs intensified teachers’ feelings about the issue. A pilot or an evaluation of the system would have identified these issues for the district. The district did communicate with teachers. However, it did many things, like meeting (as administrators) to improve inter-rater reliability, being trained on the system, and evaluating administrators, that teachers did not know about. Also, the issues of “highly effective” and student growth varied among buildings. Teachers felt there was a difference between what the district said and what the district did. Some teachers felt that the system was not reliable among administrators. Though the district was evaluating administrators on their evaluations, this fact was not communicated to teachers. There was a perception that evaluations among evaluators varied within the building and within the district. The district did not facilitate regular meetings among building administrators to produce reliable assessments for teachers. Within the building, teachers met to work on gaining data to display student growth data. Though no faculty meeting time was set for this purpose, teachers did meet. This was positively perceived by administrators and some teachers. Some teachers and the building administrator felt that the unplanned observations gave administrators the chance to see the teacher in a natural environment. 129 District 5: Small Rural Fringe District 5 is the smallest of the districts in the study. District and building administrators share responsibility for managing the operations of the district. District 5 used a group of volunteer teachers, some with union responsibilities, and some who just wanted to be involved for their team. In District 5, there have been no layoffs or performance pay implemented, so the implementation and perceptions are influenced by the fact that the impact of teacher rankings have been mostly a matter of pride for teachers and did not influence job action or compensation. District 5: Implementation A complicating factor for District 5 was its implementation of a new vendor-based system for teaching and learning that was implemented at the same time as the new teacher evaluation system. Efforts to train teachers and gain understanding of a common language mixed with the adapted observation protocol from previous years. In addition, the district implemented a new evaluation and observation system during the 2013-2014 school year, based on input from district leadership. The study took place during that school year, so there was a need for clarification, as some participants were confused about which implementation was to be discussed. During the initial implementation, the district used several of Grote’s implementation steps. Table 30 below describes responses by participant. DA indicates District Administrator; BA indicates Building Administrator; UCSAT indicates Union rep Core Subject Area Teacher; CSAT indicates Core Subject Area Teacher; and NCSAT indicates Non-Core Subject Area Teacher. 130 Table 30: District 5 Implementation Survey Reponses D5 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews D5- Sources (One Building Union Representative, One Central Office Administrator, and One Building Principal) 3/3- Team-Building and district administrators and teacher representatives (All) (Provided by Central Office Administrator) (Interviews with One Building Union Representative, One Building Administrator, and One HS Teacher) Names of teachers on the teacher evaluation team were submitted. 2/3 Described a volunteer team of teachers who worked with administrators on the district’s teacher evaluation system (BA, NCSAT) 1/3 Remembers very well (CSAT) 1/3 Does not remember very well (CSAT) 1/3 Does not remember at all (NCSAT) 2/3- Team discussed, but final decision was made by administration. (BA, UCSAT) 1/3- Dictated to district by Michigan Department of Education (DA) 1/3- They led the process (UCSAT) 2/3- They were informed of the process regularly in a private setting (DA, BA) N/A 2/3 Described a volunteer team of teachers who worked with administrators on the district’s teacher evaluation system. (BA, NCSAT) ---- N/A 0/3- Described central office’s involvement. 2/3 Remember well (CSAT, NCSAT) 1/3 Remember (CSAT) Formation of Implementation Team How was the decision on the model made? Central office involvement Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 131 Surveys-Do Teachers Remember the District Implementation Steps? (Surveys of two core subject teachers and one non-core subject teachers) Table 30 (cont’d) D5 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews Focus group/Survey for needs assessment 3/3- No (All) N/A 0/3- Described a focus group/survey used for a needs assessment. Pilot 3/3- No (All) N/A 0/3- Described a pilot Communication Modes 1/3- Building administrator explained at faculty meetings. (UCSAT) Teacher evaluation tool was submitted 3/3- Described the building administrator explaining it to the faculty at a meeting (All) 1/3- Through emails with evaluation procedures and tools attached (BA) 2/3- Described an instructional initiative to gain common language and teaching that used walk throughs to identify effective teaching strategies during the first year of the teacher evaluation system implementation (BA, NCSAT) 1/3- OtherQuote 1: “Building faculty meetings and/or individual private meetings with teachers” (DA) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 132 Surveys-Do Teachers Remember the District Implementation Steps? 1/3 Remembers very well (CSAT) 1/3 Does not remember very well (NCSAT) 1/3 Does not remember at all (CSAT) 1/3 Remember well (NCSAT) 1/3 Remember (CSAT) 1/3 Do not remember at all (CSAT) 1/3 Remember well (NCSAT) 1/3 Remembers (CSAT) 1/3 Does not remember at all (CSAT) Table 30 (cont’d) D5 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews Communication: How well was new system communicated to teachers? 3/3 Not Well (All) N/A Training for administrators 2/3- Through training through outside vendors (DA, UCSAT) 1/3- Through answering questions administrators had about the procedures (DA, BA) N/A 3/3- Described the principal explaining it to the faculty in meetings (All) 2/3- Described challenges in the administrator communicating and the faculty understanding differences in teaching techniques and in data (in different subject areas) (UCSAT, NCSAT) 1/3- Discussed principals working together to get on the same page for teacher evaluations during the first year of implementation (BA) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 133 Surveys-Do Teachers Remember the District Implementation Steps? ---- 1/3 Remembers (NCSAT) 2/3 Do not remember very well (CSAT) (Trainings were grouped together on teacher survey) Table 30 (cont’d) D5 Grote’s Implementation Steps Implementation Surveys Interviews Implementation Documents Modes of training for teachers 1/3- Through emails with evaluation procedures and tools attached (BA) 2/3- Other (please describe)Quote 1: “not really at all, no rubric was provided on highly effective teaching procedures” (UCSAT) Quote 2: “Building faculty meetings and/or individual private meetings with teachers” (DA) 1/3- Yes (DA) 2/3- No (BA, UCSAT) 2/3- Described a MCREL teaching initiative in which teachers joined the principal on instructional walks, so that they could see effective teaching across the building. (BA, NCSAT) Teacher evaluation tool was submitted 0/3- Described administrators being evaluated on the way they evaluate teachers N/A 1/3 Well (DA) 2/3 Not Well (BA, UCSAT) 2/3- Described teachers not being sure how to achieve rankings (UCSAT, NCSAT) 3/3- Described teachers not involved in the “teacher evaluation” team not being informed (All) N/A Evaluators evaluated on their evaluations Teachers Understand How to Perform Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 134 Surveys-Do Teachers Remember the District Implementation Steps? 1/3 Remembers (NCSAT) 2/3 Do not remember very well (CSAT 2) (Trainings were grouped together on teacher survey) 2/3 Do not remember very well (CSAT, NCSAT) 1/3 Do not remember at all (CSAT) ---- Table 30 (cont’d) D5 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews District solicited feedback on system 1/3- Yes, with a focus group (DA) 2/3- No (BA, UCSAT) N/A 0/3- Described the district gaining feedback from teachers Adjustment to system based on feedback 1/1- Not sure (DA) N/A 0/3- Described changes to evaluation system based on feedback Surveys-Do Teachers Remember the District Implementation Steps? 1/3 Remembers well (CSAT) 1/3 Remembers (NCSAT) 1/3 Does not remember at all (CSAT) 2/3 Do not remember well (CSAT, NCSAT) 1/3 Does not remember at all (CSAT) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher District 5 formed a team of volunteer teachers, union representatives, building administrators, and district administrators. A building level leader with central office responsibilities led the process. There was no needs assessment or pilot. Communication about the new evaluation system was communicated at building meetings and through email. One-onone meetings with building administrators and training on teaching model by an outside vendor were given to teachers. Administrators met to agree on the teaching model. The expectations for teaching were described as “not well” communicated to teachers by all survey participants. As mentioned above, the district implemented a system of expectations for teaching and learning that did not necessarily align with the new teacher evaluation system. In order to teach the new system of teaching and learning techniques and vocabulary, the building administrator led walk-throughs with teachers. This did clearly define some parts of the teaching and learning process. It was not meant to train teachers on how to do well on the teacher evaluation rubric. Teachers volunteered to go on the walk-throughs, but not all teachers participated. The building administrator described the two initiatives together this way: “Two separate things are going on at the same time. We have an evaluation tool, but then, another initiative, a school improvement initiative starts up, which is how we teach. The two are separate, but they quickly 135 collide, because how we teach is really instruction and assessment. It quickly nullifies things like professional responsibility and environment.” Though the walk-throughs did allow teachers to see other teachers using district-approved techniques, the walkthroughs were not followed up by discussions about how participating teachers could implement the techniques in their classrooms. One teacher, in an interview, said it actually was more stressful because if a teacher did not teach the subjects s/he saw during the walk-throughs, it was hard to transfer those techniques into his/her classroom. Formal training for teachers was given through rubrics and explanations of those rubrics. Informal feedback was given to administrators, but no formal survey or focus group was used to assess the teacher evaluation system. The “evaluation team” provided informal input on the teacher evaluation system. After the initial usage, feedback was not formally given through focus groups or surveys. No adjustments were made to the system until 2013-2014, when the whole teacher evaluation system was changed completely. District 5: Teacher and Administrator Perceptions Teacher and administrator perceptions skewed negative in District 5. Teachers described a lack of clarity on the expectations. One teacher described a dearth of timely feedback. Another teacher expressed concern over only one administrator evaluating teachers. If that administrator was inaccurate, there was no clear avenue for teachers to redress those concerns. Specificity and communication were also described by both interview and survey participants. One said that the language was too broad and the expectations for scores too high. These responses and quotes are visible below on Table 31, Table 32, and Table 33. Table 31 is organized by positional responses about the implementation’s acceptability and influence on morale. It is initial-coded. Table 32 describes specific and concrete questions about the implementation, including its most important part: what influence it had on perception, and what should have been included in it. Table 33 follows Noe’s attributes (reliability, relevance, specificity, strategic congruence, and acceptability). Table 31 is initial-coded like Table 30. Table 32 and 33’s responses are sequenced by respondent. First respondent is teacher union-representative; second respondent is teacher without an official union position; third response (only on questions of most important step, influence of implementation on perception, and what should have been included). 136 Table 31: D5 Teacher/ Administrator Survey Perceptions of Implementation’s Influence D5 Teacher Morale and Acceptability from Surveys In building, how did the changes influence morale? Implementation Surveys Implementation Documents Interviews Surveys-Do Teachers Remember the District Implementation Steps? 1/3 Neither improved nor diminished morale (DA) 2/3 Diminished morale (BA, UCSAT) N/A 2/3- Described stress with the new teacher evaluation system based on lack of understanding and change (UCSAT, NCSAT) Quote 1: “It has become an item that has not only worried staff, but also something that many members of staff actually dislike. It could be the formality of it or it could be that we don't fully understand all of the expected criteria included in evaluation process. Some items were not addressed in detail for us to understand.” (NCSAT) Quote 2: “As a teacher I feel that the tool is too broad, the language is vague...how does an administrator judge caring (effective rating) compared to genuine caring about students (highly effective rating)...we are a small school district and many activities that are listed on the evaluation tool only include a very small number of teachers...so teachers feel that they do not have a mathematical chance of being highly effective by our districts interpretation of the new eval. tool...so it has lowered morale” (CSAT) Quote 3: “There was significant frustration about the lack of specificity of the evaluation system. The principal didn't have specific benchmarks for student growth, test scores, or success rates so teachers didn't have any idea what targets they were to meet. Overall morale is significantly lower currently as teachers are frustrated that they're increasingly asked to do more (and are evaluated on that) with less and for less pay.” (CSAT) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 137 Table 31 (cont’d) D5 Teacher Morale and Acceptability from Surveys In district, how did the changes influence morale? Implementation Surveys Implementation Documents Interviews 2/3 Neither improved or diminished morale (DA, BA) 1/3 Diminished morale (UCSAT) N/A 0/3- Described teacher morale at a district level. Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 138 Surveys-Do Teachers Remember the District Implementation Steps? Quote 1: “From what I understand from other buildings, the morale of the staff was greatly reduced by the implementation of this evaluation system. I might even go as far to say that individuals were angry about the lack of transparency in evaluation and timely feedback.” (NCSAT) Quote 2: “Our district has set highly effective at a 90% rating on the new evaluation tool and many teachers took their evaluation from last year and put it into the new system and even if they received a highly effective rating last year on the new eval. tool they would not...so again lowered morale. Again the language is too broad and in small districts it is only one person evaluating teachers so it feels like it is only one persons opinion that matters.” (CSAT) Quote 3: “I don't have much district-wide communication, however I do know that the lack of specificity complaint was district-wide.” (CSAT) Table 31 (cont’d) D5 Teacher Morale and Acceptability from Surveys Implementation Surveys Implementation Documents Interviews Based on your experience overall, how acceptable have you found the district’s teacher evaluation system? Notes: 1/3 Acceptable (DA) 2/3 Unacceptable (BA, UCSAT) N/A 2/3- Said the evaluations were acceptable. (BA, CSAT) DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 139 Surveys-Do Teachers Remember the District Implementation Steps? 1/3 Neutral (NCSAT) 2/3 Unacceptable (CSAT 2) Table 32: District 5 Interview Perception Response Summary 1: Building Union Representative Question Most Important Part Summarized Responses 1. Getting everybody on board. 2. Walk-throughs to ensure everyone knew what administration was looking for 3. Making sure all administrators were looking for the same thing. Implementation influence perception 1. Negatively, because of lack of communication 2. It was stressful because any change is stressful 3. It was complicated because the district was implementing two things at the same time. Should have been included, but was not 1. More input from teachers 2. More time to better understand what was happening 3. Professional development so teachers could understand what was expected 2: Teacher 3: Administrator Quotes 1. “I would think that in order for it to go well, just in terms of everybody being on board, is getting the teachers on board and saying, ‘OK, there are certain things we have to do for the state, how can we either make sure we're doing that or how can we observe that?’” 2. “The most important part was the actual walk-through and seeing the kinds of things that the administration looks for in the classroom.” 3. “I just really felt like there needed to be a system that was implemented with some level of fidelity.” 1. “Lack of the communication makes it scary and so it's not something that teachers, necessarily, because of their uncertainty about it, it's very scary.” 2. “It also adds a bit of uncertainty. Teachers are to the point where they're not quite sure what's going on, they get frustrated, they get angry, they're worried about things.” 3. “Now all the teachers and administrators were trained on, ‘This is how we teach, and this is how we think we should measure good teaching.’ We didn't have an evaluation tool, though, with those types of words and in that language.” 1. “The admin would be better off, as the teachers would, if it wasn't more cooperative, if they had some input.” 2. “Time wasn't friendly. I would like to have seen, or I think what wasn't included, is enough time to actually see how this is working in classrooms and see how other teachers are implementing and so forth.” 3. “Serious professional development and training right out of the gate. Like I said we did it, but we put the cart before the horse, so to speak.” 140 Table 33: District 5 Interview Perception Response Summary based on Noe’s Attributes 1: Building Union Representative Question Reliable Relevant Specific Strategic Congruence to District goals Acceptability 2: Teacher 3: Administrator Responses 1. No, because it is highly subjective 2. No, because techniques vary between subject areas 1. No, positives are underaccounted and negatives are over-accounted 2. Yes and no, because what students think is not included Quotes 1. “I don't think it's reliable at all, because I believe it's very subjective.” 2. “I think in some cases the things that they look for are different.” 1. “I don't believe that all positives are accounted. I think that some of the negatives get over-accounted. I also believe that has happened to other teachers that have spoken to me as the building rep that they feel. I think I feel this way as well.” 2. “One thing that you can't necessarily rate or have a more difficult time, and this is the potential problem or hurdle, whatever you want to call it, is what students feel about the class and how they're able to use it and what they're going to do with it from this point forward.” 1. No, it is not clear on how to become highly effective. 2. No, it varies from teacher to teacher 1. No, because teachers know how to align themselves with district goals 2. Yes, it is up to the teachers to figure this out 1. “There was one of the problems for instance when I say to the person during my evaluation. So I am effective. How do I become highly effective? The answer was I do not know.” 2. “Clear? Not so much. I do feel like we have this broad overall.” 1. No, because it is not clear how to get “highly effective” 2. Yes, but some areas are debatable 3. No, teachers think it is pre-determined. 1. “I think that you are good teachers align themselves with the district's goals and objectives no matter what the evaluation tool is. That is why they are good teachers.” 2. “Obviously, it's up to us to also figure out how to implement it. Sometimes that's hard. Sometimes it's very difficult. With everything that's being asked of us, sometimes we do a really good job of it, and other times we kind of struggle with it, too.” 1. “The most important statements to me was, if I am effective how do I get to be highly effective. I do not know. That is a telling statement to me. At any category if I am minimally effective how to become effective I do not know.” 2. “For the most part, I would say yes. I do think that there are some things that are open to interpretation, with that evaluation system, that could be a means for discussion. I don't understand what you see in this particular aspect, why I was rated differently than the way I feel, positively or negatively? If it's lower than what I think, I need to know why, and what you saw, and I will explain what I feel I'm doing. If I'm rated higher, like most people, if they're rated higher they probably don't say a whole lot.” 3. “I think in short they don't think it's fair. They think they're predetermined into effective slot and that's where they stay. They probably consider themselves somewhat lucky if they get highly effective. They have a hard time defining it, and if they are given something in a minimally effective or ineffective range, they're greatly offended, typically.” 141 Interview participants felt the most important part in this district was getting everyone on the same page, walkthroughs to make sure all teachers understood what administrators were looking for, and getting all administrators together on their approach. All three participants described stress and uncertainty because the system was new. Two participants said there was a lack of clarity and communication on the system, when describing the implementation. Participants also said what should have been was more time and more input from all teachers, rather than just those who had volunteered to be on the “teacher evaluation team.” Interview participants agreed that reliability and specificity were problems in the teacher evaluation model. Reasons for this were the difference between subject areas and the variance between the way administration interprets teacher performance. Alignment to district goals had mixed response. One participant said that teachers need to figure out the alignment themselves; another participant said that teachers should align themselves to district goals without an evaluation. Relevance also had mixed results. One participant said what students think should be included and is not. Another said positive are undervalued and negatives are over emphasized. The response to acceptability from interview participants was mixed too. Two participants thought that there is not enough specificity. Another thought how to achieve different designations was not clear. In the teacher and administrator surveys, a majority of teacher perceptions were negative. In terms of morale, teacher perceptions were mixed. Administrators thought the implementation of the evaluation system did not have an influence on teacher morale. Teachers who thought the district’s response negatively impacted teacher morale were concerned that the wording and descriptors are vague, “highly effective” designation is hard to reach, and teachers did not know how to accomplish what the evaluation system required. 142 District 5: Relationship between Implementation and Perception District 5 followed some of Grote’s implementation steps, but this was confused by the implementation of a new terminology and definition of teaching and learning that did not directly align with the new teacher evaluation’s framework for observations. There was significant overlap, but the terminology was different. Teachers felt that the language in the teacher observation framework was too broad. The feedback received was not specific enough. During implementation, reducing the number of initiatives to one, teacher evaluation, could have improved what teachers had to learn during the implementation. Furthermore, a pilot or a needs assessment (since the district implemented an adapted form of its old framework) would have allowed teachers to have a say in the wording of the observation framework. It also would have allowed feedback on the implementation of two somewhat conflicting initiatives at the same time. A second perception challenge was communication. With two initiatives implemented simultaneously, communication about the overlap of the two would have allowed teachers to see them as complementary. The communication about the “teacher evaluation team” and their work could have defined how the district was working with teachers. The decisions, the discussion, and the relationship to student learning could have been defined clearly in teachers’ minds. There was no formal plan on how to communicate about the changes in the district, so one of the teachers and the administrator who participated in interviews described a lack of clarity. Another interview teacher participant described the initiatives as if they were one. The walk-throughs could have been used to strengthen teacher understanding, but, according to one teacher who participated, it caused more confusion and stress for teachers. If walk-throughs were going to be a part of the implementation, their role and their transfer to participating teachers’ instructional methods should have been defined more clearly. A third concern among teachers was the fact that the administrator was not trained, not supported, and not monitored effectively. No central office administrator was reviewing and providing feedback to the evaluators. Combined with the two evaluation and instructional initiatives, this caused a perception that feedback was not timely or specific. There was also a sense, from teachers, that if there was an inaccuracy in the observation or evaluation of performance, there was no recourse for the teacher. Therefore, this lack of monitoring by central office caused more scrutiny of the content and outcomes of the teacher evaluation system because administrators were left, as one teacher said, to “sink or swim.” Aligned with the lack of proper monitoring by central office was the fact that there was no formal or clearly communicated assessment of the teacher evaluation system. Teachers are charged with formally and informally assessing their students on a daily, weekly, monthly, and yearly basis. However, when the district implemented their teacher evaluation system, years went by without much, if any, assessment and improvement of the system. Districts 3 and 5 drastically changed 143 its teacher evaluation system to begin during the 2013-2014 school year. These changes may have been made because of informal feedback. However, the reason for the changes were not communicated to teachers, so the second teacher evaluation system seemed to be mandated from the district level without any clear rationale or explanation on why the system changed. There were not many positive perceptions about the teacher evaluation system from teachers. Administrators, on both the survey and the interview, felt that the perception of teachers was at best unaffected, and at worst, not influenced by the implementation. In the context of the interview, one administrator described more effort and conversation from teachers. Teachers also described more effort on evaluations, but they perceived that effort toward an unclear goal and without specific feedback, in addition to being for less money. There was a clear consensus, however, that teachers were putting in more work on their evaluations. This was because the system expected more of them, and, at least, the need for more work was obvious. These findings are summarized on Table 33 below. Table 34: District 5’s Implementation and Influence on Perception of the System Implementation Steps/Lack of Implementation Steps Confusion about implementation of teacher evaluation system and instructional practice initiatives Lack of pilot and needs assessment Lack of a formal communication plan More work by teachers Perceptions Influenced Some teachers confused and merged the two initiatives. Teachers were supposed to learn a new terminology and methods for teaching, but those methods and terminology were not all on the adapted observation framework. The observation framework was not as clear as the instructional practice definitions, so teachers were confused on expectations. Most teachers described confusion about expectations was present and continues today. Lack of teacher input from the beginning, outside of the “teacher evaluation team”, left teachers without understanding and a sense of some ownership in the process. Most teachers felt as if they did not understand expectations and the decisions made by the “teacher evaluation team” because the emphasis of building administrators during implementation was on the workings of the system, not on how or why the system was designed the way it was. Most teachers and administrators described more work by teachers to meet the expectations of the new teacher evaluation system. 144 District 6: Large Rural Fringe District 6 is described as a large rural-fringe district in Michigan. In this district, there have been no layoffs based on the teacher evaluation system. Performance pay has been implemented, but for a very small amount of money. The implementation and perceptions are influenced by the fact that the impact of teacher rankings have been mostly a matter of pride for teachers and a small amount of money. In addition, there has been a significant administrator turn-over at the building level during the implementation. District 6 Implementation District 6 used some of the Grote’s implementation steps. These implementation steps were complicated by a change in the building principalship during implementation. It was also complicated by the district briefly participating in a statewide pilot of teacher evaluation frameworks. During the school year 2012-2013, the district attempted to manage this pilot while still running its old system. Within the district, there has been a high level of intervention from the state to improve financial and student achievement performance. Table 35 below describes responses by participant about implementation. DA indicates District Administrator; BA indicates Building Administrator; UNCSAT indicates Union rep Non-Core Subject Area Teacher; CSAT indicates Core Subject Area Teacher; and NCSAT indicates Non-Core Subject Area Teacher. 145 Table 35: District 6 Implementation Survey Reponses D6 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews D6 Sources (One Building Union Representative, One Central Office Administrator, and One Building Principal) (Provided by Central Office Administrator) Formation of Implementation Team 3/3- Team- Central office and building administrators, teachers were on the team (All) Agendas and emails from this group were submitted. How was the decision on the model made? 1/3- Team discussed, but final decision was made by administration (UNCSAT) 1/3- Selected a tool from external consultant, with input from team (DA) 1/3- Other (please describe)- “discussed and developed our own based on Danielson model” (BA) 1/3- They led the process (DA) 1/3- They were informed of the process regularly in a public setting (board meeting, committee meeting) (UNCSAT) 1/3- Not sure how involved they were involved (BA) Agendas and emails from this group were submitted. No direct evidence on decision making. (Interviews with One Building Union Representative, One Building Administrator, and One HS Teacher) 2/3- Described the team working together on the evaluation tool (BA, UNCSAT) 1/3- Described committee adapting the Danielson model to fit the district’s needs (BA) Central office administrator sent emails and set agendas, leading the committee 1/3- Described working with central office on the development of the tool (BA) Central office involvement Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 146 Surveys-Do Teachers Remember the District Implementation Steps? (Surveys of two core subject teachers and one non-core subject teachers) 3/3 Do not remember at all (All) ---- 2/3 Remember (CSAT, NCSAT) 1/3 Do not remember very well (CSAT) Table 35 (cont’d) D6 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews Focus group/Survey for needs assessment 1/3- Yes (BA) 2/3- No (DA, UNCSAT) 1/3- Described teachers and administrators working together. No formal survey/focus group identified (BA) Pilot 3/3- No (All) Teacher evaluation committee (and union leadership) provided feedback as evaluation was developed. No separate focus group or survey was used. Emails and agendas were submitted that show the union and district working together. N/A Communication Modes 2/3- Administrators explained in faculty meetings (BA, NCSAT) 2/3- Building and/or district leaders explained in union meetings (BA, NCSAT) 1/3- Central office administrators explained in district wide meetings (BA) 1/3- Other (please describe)- “District administration and Union Leaders trained all building Union Reps and principals reviewed with staff with the help of the reps” (DA) Numerous documents and emails explaining process, rubrics, tuning, and tuning to administrators, union representatives, and teachers 2/3- Described central office and district administrators working with the union and teachers to make sure teachers understood expectations through emails, group and individual meetings (BA, CSAT) 1/3- Described unclear and changing expectations as well as different follow-through between administrators and buildings. (UNCSAT) 0/3- Described a pilot Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 147 Surveys-Do Teachers Remember the District Implementation Steps? 1/3 Remembers (CSAT) 2/3 Do not remember very well (CSAT, NCSAT) 1/3 Remembers (NCSAT) 2/3 Do not remember very well (CSAT 2) 3/3 Remember (All) Table 35 (cont’d) D6 Grote’s Implementation Steps Communication: How well was new system communicated to teachers? Implementation Surveys Implementation Documents Interviews 1/3 Well (DA) 2/3 Not well (BA, NCSAT) Numerous documents and emails explaining process, rubrics, tuning, and tuning to administrators, union representatives, and teachers Training for administrators 1/3- Through training from district employees (DA) 1/3- Through providing administrators with written documents that described the new procedures (DA) 1/3- Through answering questions administrators had about the procedures (DA) 2/3- Other (please describe)- Quote 1: “training by the ISD” (UNCSAT) Quote 2: “No training was provided about implementation/use of the tool” (BA) Numerous agendas about training teachers and administrators, rubrics, process guides, flow charts were submitted. 2/3- Described the one to one meetings as essential to being specific (BA, CSAT) 1/3- Described limited/no meetings and little clarity on expectations for non-core faculty (UNCSAT) 2/3- Brought up the communication depended on the administrator who was evaluating (BA, UNCSAT) 0/3- Described training for administrators Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 148 1/3- Described administrators meeting on their own time to “tune” their approach to teacher evaluations (BA) Surveys-Do Teachers Remember the District Implementation Steps? ---- 1/3 Remembers well (CSAT) 1/3 Remembers (NCSAT) 1/3 Does not remember at all (CSAT) (Teacher and Administrator Training were grouped together on the survey) Table 35 (cont’d) D6 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews Modes of training for teachers 1/3-Through emails describing the way teachers were expected to perform (DA) 1/3- Through mandatory meetings to begin the school year (DA) 2/3- Other (please describe)- Quote 1: “some, not all adminstrators had individual meetings” (UNCSAT) Quote 2: “No clear expectations related to ratings beyond what was stated on the tool” (BA) Numerous agendas about training teachers and administrators, rubrics, process guides, flow charts were submitted. 2/3- Described the most effective training as being when administrators met with teachers to begin the year, before the first observation (BA, CSAT) 1/3- Described some administrators did not have time to meet with teachers, so it was not clear how to meet expectations (UNCSAT) Principals/directors evaluated on their evaluations 3/3 No (All) N/A 0/3 Described administrators being evaluated on how they evaluate teachers Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 149 Surveys-Do Teachers Remember the District Implementation Steps? 1/3 Remembers well (CSAT) 1/3 Remembers (NCSAT) 1/3 Does not remember at all (CSAT) (Teacher and Administrator Training were grouped together on the survey) 3/3 Do not remember at all (All) Table 35 (cont’d) D6 Grote’s Implementation Steps Implementation Surveys Implementation Documents Interviews Teachers Understand How to Perform 1/3 Well (DA) 2/3 Not well (BA, UNCSAT) Numerous agendas about training teachers and administrators, rubrics, process guides, flow charts were submitted. District solicited feedback on system 2/3- No (BA, DA) 1/3- Not sure (UNCSAT) N/A 2/3- Described the most effective training as being when administrators met with teachers to begin the year, before the first observation (BA, CSAT) 1/3- Described some administrators did not have time to meet with teachers, so it was not clear how to meet expectations or how to perform (UNCSAT) 0/3- Described feedback being solicited. Adjustment to system based on feedback 0/0 Answered questions because no feedback was gained. N/A 0/3- Described the district gaining feedback. Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 150 Surveys-Do Teachers Remember the District Implementati on Steps? ---- 1/3 Do not remember very well (CSAT) 2/3 Do not remember at all (CSAT, NCSAT) 1/3 Remember (CSAT) 1/3 Do not remember very well (NCSAT) 1/3 Do not remember at all (CSAT) District 6 formed a “teacher evaluation team” made up of building administrators, district administrators, and teacher union representatives. It was led by a central office/district administrator. The “teacher evaluation team” provided feedback on the needs for the district. No formal focus groups or surveys were used. Communication occurred at district and building level meetings. Emails and one-on-one meetings with administrators were also used to communicate the new teacher evaluation system. Administrators were trained through the county ISD, district meetings, and trainings by outside vendors on the tools. There is clear documentation of all of these efforts. Administrators were not evaluated on the way they evaluate teachers. After initial usage, feedback was provided through teachers meeting with administrators. There were not surveys or focus groups to assess the teacher evaluation system. No adjustments were made to the teacher evaluation system. District 6: Teacher and Administrator Perceptions Teacher and administrator perceptions in interviews were mixed. In surveys, they were also mixed. One participant did not seem to understand the questions, so s/he answered with the school name and his/her building to the questions. Her/his answers were difficult to interpret for the open-ended questions about morale. One of the teacher interview participants was very positive about his/her experience with the teacher evaluation system. He said his perspective was shaped by the fact that his evaluator was a teacher in his subject area, understood it well, and was familiar with the teaching methods used to communicate it to students. The other teacher interview participant was very negative in her/his perceptions. S/he described the leadership turnover, the lack of clarity on expectations, and the absence of follow-through at end-of-the year teacher evaluation meetings. These mixed perceptions are evident in the tables below. The responses are visible below on Table 36, Table 37, and Table 38. Table 36 is organized by responses about the implementation’s acceptability and influence on morale. Table 37 describes specific and concrete questions about the implementation, including its most important part: what influence it had on perception, and what should have been included in it. Table 38 follows Noe’s attributes (reliability, relevance, specificity, strategic congruence, and acceptability). Table 36 is initial-coded like Table 35. DA indicates District Administrator; BA indicates Building Administrator; UNCSAT indicates Union rep Non-Core Subject Area Teacher; CSAT indicates Core Subject Area Teacher; and NCSAT indicates Non-Core Subject Area Teacher. Table 36 and 37’s responses are sequenced by respondent. First respondent is teacher union- 151 representative; second respondent is teacher without an official union position; third response (only on questions of most important step, influence of implementation on perception, and what should have been included). Table 36: D6 Teacher/ Administrator Survey Perceptions of Implementation’s Influence 1: Building Union Representative D6 Teacher Morale and Acceptability from Surveys In building, how did the changes influence morale? 2: Teacher 3: Administrator Implementation Surveys Implementati on Documents Interviews Surveys-Do Teachers Remember the District Implementation Steps? 3/3 Diminished morale (All) N/A 1/3- Described reduced morale because of inconsistency amongst administrators (UNCSAT) 1/3- Described reduced morale because change is hard (CSAT) 1/3- Described neutral morale influence because some administrators are fair with teachers (BA) Quote 1: ““In the beginning, moral was a bit discouraged. Teachers feel we work hard and now there is another way to criticized.”” (NCSAT) Quote 2: “The process is inconsistent based on which administrators evaluate you. This leads to reduced respect for the process, which leads to lowered teacher morale.” (CSAT) Quote 3: “Since there is no uniform evaluation being implemented, the results are not compelling.” (CSAT) Key for Position Designations- DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 152 Table 36 (cont’d) D6 Teacher Morale and Acceptability from Surveys Implementation Surveys Implementation Documents Interviews In district, how did the changes influence morale? 1/3 Neither improved or diminished morale (DA) 2/3 Diminished morale (BA, UNCSAT) N/A 0/3- Mentioned district morale Based on your experience overall, how acceptable have you found the district’s teacher evaluation system? 1/3 Neutral (DA) 2/3 Unacceptable (BA, UNCSAT) N/A 2/3- Found it acceptable because the administrator communicated clearly and understood teachers’ subject matter (BA, CSAT) 1/3- Found it to be unacceptable because the implementation was not uniform, and the system was not implemented consistently throughout the building (UNCSAT) Notes: DA- District Administrator BA- Building Administrator UNCSAT- Union representative Non-Core Subject Area Teacher UCSAT- Union representative Core Subject Area Teacher CSAT- Core Subject Area Teacher NCSAT- Non-Core Subject Area Teacher 153 Surveys-Do Teachers Remember the District Implementation Steps? Quote 1: “An increase in the disconnect from the school district” (NCSAT) Quote 2: “It has made little difference in teacher morale.” (CSAT) Quote 3: “I do not know about other schools here.” (CSAT) 2/3 Neutral (CSAT, NCSAT) 1/3 Unacceptable (CSAT) Table 37: District 6 Interview Perception Response Summary 1: Building Union Representative Question Most Important Part Summarized Responses 1. Consistency 2. Pre observation conference (meeting between administrator and teacher) 3. Clear understanding of the goal Implementation influence perception 1. Confusing 2. Not at all 3. Yes, positively Should have been included, but was not 1. Information 2. Difference between effective and ineffectiveness 3. Time 2: Teacher 3: Administrator Quotes 1. “There needs to be a follow-through and make sure everybody's doing the same and that you're not just going in checking 10 minutes on this person and spending 45 minutes with someone (else).” 2. “It was very straightforward and we could follow their guidelines.” 3. “There needs to be clear communication about the purpose of why are we doing this.” 1. “I remember it starting off a little shaky. ‘This is how we're going to do it.’ Then somebody else came in, then a little shaky again. ‘This is how we're going to do it’.” 2. “Those people became a framework for adjusting, as in ‘you guys do this or you don't get to be teaching here anymore.’” 3. “(the system) would find ways to allow for the things that they couldn't control, and make it fair and equitable” 1. “They didn't give the information to the teachers efficiently, and I think that nobody knew what they were doing.” 2. “ I think there was the idea of how to evaluate and ascribe, highly effective, was somewhat uninspired. We don't know what the rules will be.” 3. “There's just not enough time for us.” 154 Table 38: District 6 Interview Perception Response Summary based on Noe’s Attributes 1: Building Union Representative 2: Teacher 3: Administrator Question Reliable Responses 1. No, leadership changes hurt 2. No, not uniform Quotes 1. “ The thing that has really hurt us is the change in leadership.” 2. “It's not the case. It is not uniform.” Relevant 1. No, because it depends on the kids in the class 2. Yes, lots of good questions about good teaching Specific 1. No, not enough information on how to do their job well 2. Yes, this was through preobservations Strategic Congruence to District goals 1. No, we do well with kids. Evaluations do not show this 2. Yes, suggestions are a way to collaborate on district goals 1. “She (another teacher) said because of the test scores and some of the special education her class and that she had a higher number of special education kids in her class than prior years.” 2. “It did have lot of questions in regards to the environment. Does it look inviting, are students engaged? Are students welcomed? Things like that are very important.” 1. “They didn't give the information to the teachers efficiently, and I think that nobody knew what they were doing.” 2. “I have always thought they have been incredible and use that pre-observation thing, as a very good way to specifically say what they are looking for and thing that they are concerned about that they think you need to work on. I thought, well communicated.” 1. “We have done really well with kids. Some of our kids have done wonderful things outside and I'd like that to continue, not as many as I can. They're not looking at one thing in particular.” 2. “What it does more is if you are a really good teacher, evaluation is a way to collaborate with administrator and have him give you suggestions for improvement.” Acceptability 1. No, basic needs are not being met 2. Yes, there are good discussions 3. Yes, no one challenged decisions 1. “Because most of them feel like they're being judged not fairly. They don't have what they need.” 2. “Because the discussion is also, we've to sign this thing. We've talked about it and if we don't agree with anything that is on there, we can add notes, add to that, to the file.” 3. “Nobody challenged the decisions that got made.” Two interview participants described an attempt to get all administrators on the same page on approach to teacher evaluations as the most important part. One said it did not happen, and that was the most important thing. The one-on-one meetings before observations also offered specificity in the teacher evaluation system and were important. Perceptions of the implementation varied. One participant said the meetings between teachers and administrators offered insight. Another participant said it was made clear from the beginning what teachers were supposed to do. Another participant said the communication was unclear and that lack of clarity negatively impacted teacher perception of the implementation. What should have been included was more time to make things clear, more clarity about effectiveness and ineffectiveness, and more specificity about how to be effective. 155 The reliability of the system was a place of agreement between perspectives. Both interview participants described a difference between administrators in their actualization of the teacher evaluation system. This hurt reliability. There were mixed responses to the attributes of specificity, relevance, alignment to district goals, and acceptability. Concerns in these areas included changes in leadership, subject area knowledge, and differences between administrators. Each of these factors came up in interviews about specificity, relevance, alignment to district goals, and acceptability. Depending on the administrator, the participant could say all of the attributes of effective performance appraisal systems were being met or were completely missing. The survey responses also varied between “had no influence” and “negatively influenced.” No reasons were given for negative influence. At a building level, all of the survey participants said that the implementation negatively influenced teacher morale. At the district level, there was a lack of knowledge about perceptions around the district. District 6: Relationship between Implementation and Perception What mattered to the teachers in District 6 was making it clear to them, well in advance, what would be included in their evaluation and a way to report back to them in timely fashion how they performed during their administrator’s observation of them. Implementation of an evaluation model that has the buy-in of all its stakeholders can be successful in both perception and reality. The teachers of this district, like the other five, were concerned about how the process was actually working. The first area of concern in perceptions was the difference between evaluators. The teacher who was most positive described much of his/her positive perceptions in relationship to his/her evaluator. Another teacher interview participant had a far more negative perception of the overall system. The two perceptions about evaluators came together in the description of the difference among administrators. Several factors influenced these perceptions. The first was the changes in principal leadership. When the lead administrator in a school changes during an implementation, the expectations and norms of an evaluation system can change or become secondary. The second issue teachers described was the content knowledge within the subject they taught. If the administrator understood the subject matter, then s/he would be more likely to evaluate the teacher of that subject accurately. According to the building administrator, s/he and his/her team met regularly to keep the reliability amongst themselves consistent; however, that was not communicated to the teachers. Administrators met outside of their contracted work-day, at least in the last year, 156 to try to be similar. Such inconsistency upon implementation, and the breakdown in communication once these efforts occurred, led teachers to believe that the evaluations were not reliable. Another matter that some teachers described was a lack of clear expectations across the building. It should be noted that the teacher with an administrator who was experienced in his/her subject area described the expectations as very clear. The concerns about the expectations were intensified by the district’s financial difficulties and state intervention. The expectations were also muddied by the district’s involvement in a pilot program. The pilot ended up to be too confusing and time-consuming. One year into implementation, however, a brief attempt at running two evaluation systems simultaneously caused some confusion on what administrators’ expectations were. The lack of a formal needs assessment or a pilot contributed to teachers not being sure of what was expected of them. The absence of a formal communication plan, outside of the personnel filling the administrator posts, left teachers feeling that the teacher evaluation system was not uniform. Although the district made several efforts to provide teachers with a clear understanding, there was no formal plan for communicating with teachers, so the understanding of expectations was dependent on the administrators who drew each teacher for evaluation. The institution and usage of performance pay also caused some concern among teachers. The amount of performance pay was minimal. Teachers who received it almost felt insulted by the low monetary amount. The usage of it, with little funds to support the initiative, seemed like a joke to the one teacher who did receive it. In terms of positive perceptions, several teachers felt there was little impact of the new evaluation system on the surveys. In addition, the teacher who described having a strong administrator felt the process helped him/her improve as a teacher. The problem was that this feeling was not replicated across administrators and teachers. These findings are summarized in Table 38 below. 157 Table 39: District 6’s Implementation and Influence on Perception of the System Implementation Steps/Lack of Implementation Steps Lack of training and communication of reliability among administrators Lack of specificity in expectations Lack of communication plan Institution of performance pay Strong administrator knowledge of subject area and teaching techniques Perceptions Influenced Some teachers felt there was a lack of reliability among administrators. Though considerable district resources were used to ensure reliability, the perception of teachers was accuracy and acceptability varied considerably. If teachers received an acceptable evaluation, they attributed more to their evaluating administrator’s expertise, rather than a system that fairly evaluated each teacher’s performance. Some teachers felt that the administrators did not provide clear and specific feedback on their performance. The changes in leadership and the lack of teacher feedback on the evaluation system led to teacher feeling as if the expectations were not clear. Some teachers felt there was a difference between administrators in expectations. They felt there was a lack of clear communication about what was expected from teachers. One teacher felt the amount and distribution of a small amount of performance pay made the whole act of bestowing performance pay a joke. As the district did not have enough financial resources to give significant performance pay, the small amount made it funny to the teacher. One teacher felt that the whole teacher evaluation system was strong, based on the clear expectations given by his/her administrator. This feeling was consistent throughout his/her perceptions. This perception was based on the administrator being a former teacher in the subject area and having technical knowledge of the subject and how to teach that subject. 158 FORMAL AND INFORMAL COMMUNICATION ABOUT KEY ISSUES OF IMPLEMENTATION Grote wrote extensively on the importance of communication (1996). Needs assessments, plans, consistent evaluations on the way users understood the system, and adaptations to the system were involved in Grote’s implementation steps. Communication encumbers many parts of the teacher evaluation system. If a system is only clear to some users, but not all, the whole system struggles to be accepted. In the tables and narratives above, following Noe and Grote’s frameworks, some challenges and gaps to district’s communication emerged. Communication can be analyzed into two categories, formal and informal. As used in Table D, formal communication describes what building and district administrators said to teachers officially. Informal communication, as used elsewhere in Table D, describes the communication among teachers about a particular subject. These descriptions come from interviews and open-ended survey questions from both administrators and teachers. In some districts and on some topics, the formal and informal communication are aligned. In others, the communication is different. In the areas of confidence, of the “highly effective” status and performance pay, and of positionality to the state and the system, informal communication among teachers described how teachers understood their formal communication. The areas of communication are described in the paragraphs that follow. The first challenging area is the amount of follow-through on what the district said it was going to do in teacher evaluations and what it actually did. At the introduction of the teacher evaluation system, the reality of running a school beset administrators. They faced attending extra-curricular activities, the average blow-ups and clean-ups, and the daily grind of managing a school--all parts of an administrator’s job. In the process, some expectations of the system that were stated early were not enacted. Inter-rater reliability, which is hard enough if there is time to meet, to discuss, and to bring together different backgrounds of evaluators, became difficult to achieve. In several districts, the end of the year came quickly, and some administrators did not have the time to meet with teachers on their evaluations. Those appraisals that were controversial with the teacher involved seemed to dominate the endof-the-year meetings, and time ran out. Although this is understandable, it caused a disconnect with teachers in their perceptions. If end-of-the-year meetings were promised, they were expected. Other examples are described in Table D. A second emergent and unexpected area of communication was the confidence. At first glance, confidence in the system seems like it would bolster implementation. If administrators are confident, then it would seem to facilitate confidence and acceptance among teachers. When analyzing the data of teacher perceptions, a different relationship became apparent. 159 Administrators who communicated their lack of confidence in the system and its implementation seemed to engender more acceptability and neutral influence on morale. The differences in the confidence communicated by administrators are described in the Table D. A third interesting area of communication and clarity were the ways “highly effective” and performance pay were communicated to teachers. These were two controversial portions of the teacher evaluation law because defining “highly effective” in frameworks that are more developmental than judgmental is a challenge. According to administrators in four districts, “highly effective” was a “place teachers visited occasionally, but did not always live” because of the amount of work needed to reach these levels of performance. The communication clarity of general job expectations also can be included here, as teachers were unsure of how to perform to achieve the ratings they wanted. A fourth interesting area was the stakes and/or potential stakes of the teacher evaluation system’s rankings. Districts that enacted systems with lofty targets, almost impossible to reach, then laid some teachers off or gave others performance pay because they attained those levels of performance, sent mixed messages to their teaching staff. Whether it was negative stakes (possibility of getting laid off) or positive stakes (performance pay), usage and/or the threat of higher stakes increased anxiety and nervousness about the teacher evaluation system. Using or threatening to use higher stakes also put the district and its building administrators closer to the state, rather than giving the district some degrees of distance from the state. The state wanted districts to utilize or have agreements in place to utilize performance pay and lay-offs based on rankings. If districts complied too swiftly or without offering criticism of the state’s mandates, their compliance and communicated closeness with the state’s mandates seemed to correlate negatively with their teachers’ perceptions. Teachers, as referenced in the literature, are a group who generally have achieved high levels of performance in previous experiences. If they are told that it is very hard to reach “highly effective” status and are subsequently threatened and rewarded for their accomplishments, the result will be stress in their ranks. Therefore, communication and clarity about both “highly effective” and performance pay, which itself is an intensifier of the issue of “highly effective”, pose a large challenge. Districts’ varying communication on these matters are described in Table 40. The fifth area of communication belied a district perspective on how to position their implementation against the state mandate. Districts had a range of choices. They could embrace the state mandate and believe that the state had the whole system’s best interests in mind. They could accept the state’s mandate and believe that the state legislators had a reason for doing what they did. Districts could also say that they did not believe in the state mandate and did enough to comply, but nothing more than that. Saying to teachers, “the state made me do it,” seems like it would negatively influence 160 teacher perception. Districts 1 and 2 enacted this approach. They are the districts with the most positive perceptions from teachers. The tables came from an analysis of the transcripts on these issues. Without direct questions about these issues, much of the descriptions come from inference from both teacher and administrator interviews. When asked about implementation coherence and perceptions, the teachers and administrators brought up these issues and described the areas of confusion and discontent. The analysis and descriptions come from those responses. They are described in Table 40. Table 40: Formal and Informal Communication about Key Implementation Issues Formal and Informal Communication on Key Topics District 1 District 2 District 3 District 4 District 5 District 6 Formal Communication (from Administrators) on District Follow Through of Processes of Teacher Evaluation System Admins stated most participant would receive consistent followthrough on teacher evaluation system Admins stated most participan t would receive consistent followthrough on teacher evaluation system Admins stated most participant would receive consistent follow-through on teacher evaluation system Admins stated most participant would receive consistent follow-through on teacher evaluation system Admins stated most participant would receive consistent followthrough on teacher evaluation system Admins stated most participant would receive consistent followthrough on teacher evaluation system Informal Communication (between teachers) on District Follow Through of Processes of Teacher Evaluation System Processes were slightly different, but pretty similar. Processes varied among evaluators . Some teachers did not meet with their evaluating administr ators at the end of the year. Processes varied extensively among administrators. The areas of variation were length and number of observations, length of postconference meetings, length of performance appraisal, and scrutiny of performance. Process was similar among administrators. The issue of “highly effective” being described as “almost impossible,” then being given to several teachers with performance pay seemed to cause distrust in the system. Process was similar among admins Process varied among administrator s. This started with beginning and end of the year meetings. It also varied based on the evaluator’s knowledge of teachers’ subject area Notes: Admins- Administrators 161 Table 40 (cont’d) Formal and Informal Communication on Key Topics District 1 District 2 District 3 District 4 District 5 District 6 Formal Communication (from Administrators) on Administrator Confidence in the Implementation and Usage of District’s Teacher Evaluation System Admins stated they were trying to figure it out as it went along. They were doing the best they could. Admins stated they would do their best to figure out the state law, but it was something they would have to work with teachers to understand. Admins stated they were confident in their system and thought it gave teachers a fair appraisal of their performance. Admins stated they believed their system was clear to teachers. They were confident teachers understood the system. Admin stated he/she believed the system brought about good conversations and gave teachers a fair appraisals. Admins stated they believed the system brought about improved work on teaching and gave teachers a fair appraisals . Informal Communication (between teachers) on Administrator Confidence in the Implementation and Usage of District’s Teacher Evaluation System Teachers believed the Admins were working with the teachers collaborativ ely and with each other on the teacher evaluation system. They believed the Admins were doing their best. Teachers believed the district had to comply with state law, and the way to have confidence in the system was to work with teachers on understandi ng it. Teachers believed Admins were confident in their abilities and the administrators completely believed in the district’s teacher evaluation system. Teachers believed Admins were confident in their abilities and the Admins completely believed in the district’s teacher evaluation system. Teachers believed Admins were confident in their abilities and the Admins completely believed in the district’s teacher evaluation system. Teachers believed Admins were unsure of the system, but the Admins did not communic ate their lack of confidenc e to teachers. Notes: Admins- Administrators 162 Table 40 (cont’d) Formal and Informal Communication on Key Topics District 1 District 2 District 3 District 4 District 5 District 6 Formal Communication (from Administrators) about the clarity and consistency of achieving “highly effective” ratings Admins stated they used selfevaluations to determine “highly effective” status during the initial year of implementing . In successive years, they defined how to achieve “highly effective” status. Teachers felt it was hard to reach, but achievable. Admins stated all teachers could reach “highly effective” status. Teachers felt it was hard to reach, but achievable. Admins first stated teachers would have to apply for “highly effective” status. The Admins changed their minds and attempted to define clearly the expectations for “highly effective.” Teachers felt this was an extra job that they “would have to apply for” and the criteria was unclear also. The lead district admin said at a district meeting that “highly effective” would be almost impossible. According to teachers, a number of teachers achieved this status, more than would have been anticipated, based on the lead district admin’s comments. Admins stated all teachers could reach “highly effective” status. Admins stated they clearly communicat ed how to achieve “highly effective” status. Teachers thought it had to do with how much their evaluating admin knew about the subject area or truly analyzed Informal Communication (between teachers) about the Designation of “Highly Effective” status and performance pay Teachers did understand the importance of selfevaluations during the first year. “Highly effective” achievement was more clear in the second year. It is considered very timeconsuming to achieve. Teachers believed “highly effective” status could be achieved with a lot of work. Teachers were upset by the application for “highly effective” because it was not clear. This made “highly effective” unachievable Teachers believed “highly effective” status was almost impossible in the high school building. They believed “highly effective” was predetermined. Teachers believed “highly effective” status was extremely difficult to reach. One teacher stated that the designations were predetermined and biased. Teachers believed achieving “highly effective” status was unclear and dependent on the evaluating admin’s understandi ng of the subject area. They also believed the amount of performance pay was insulting. Notes: Admins- Administrators 163 Table 40 (cont’d) Formal and Informal Communication on Key Topics District 1 District 2 District 3 District 4 District 5 District 6 Formal Communication (from Administrators) about the stakes and potential stakes of rankings Admins did not communicate about performance pay or lay-offs at all. District did not need to consider teacher layoffs. Admins did not communicat e with teachers about performance pay or teacher layoffs at all. District did not need to consider teacher layoffs. Admins communicate d the importance of the rankings, as the district considered teacher layoffs over the last two years. Perform. pay was not described by administrator s or teachers. District did consider, but did not lay-off teachers. Admins sent mixed messages about performance pay, issuing significant performance pay during the first year and quite smaller amounts the second year, to “highly effective” teachers. District did layoff teachers based on evaluation rankings Admins communicated the district might be laying-off teachers because of declining enrollment and reduced funding. Admins explained late in the first year a small amount of performance pay would be given to “highly effective” teachers. Performance pay’s inclusion, a part of the initial law, was said to be included because of state involvement in the district. Informal Communication (between teachers) about the stakes and potential stakes of rankings Teachers did not communicate or describe performance pay or teacher layoffs, other than to say it was a lousy part of the law. Teachers did not communicat e or describe performance pay or teacher layoffs. Teachers were worried about lay-offs based on the teacher evaluation system. With declining enrollment and funding, teacher layoffs seemed inevitable. Teachers were unclear about the specifications on achieving performance pay. During the first year, when the amount was very high, teachers thought “highly effective” was unattainable and therefore performance pay was unattainable too. The lack of clarity for “effective” and “highly effective” occurred because of the mixing of initiatives of an instructional framework and the new teacher eval system. They also thought that rankings were predetermined by compliance to admin initiatives. Teachers believed the amount of performance pay was insulting. They also believed achievement needed varied greatly depending on the admin evaluating them. 164 Table 40 (cont’d) Formal and Informal Communication on Key Topics District 1 District 2 District 3 District 4 District 5 District 6 Formal Communication (from Admins) on the District’s Positionality to the State Law Admins stated to teachers this was something the state was “making them do.” Together, the teachers and administrato r would “build the plane, while flying it.” Admins clearly stated this was a state mandate and not one the district would have chosen itself. They stated and showed teachers, union representativ es, and Admins would spend time understandin g the system together. Admins stated the teacher evaluation system was a best practice given by the state. The district would work to meet the mandates and it would improve teacher practice. Admins stated the teacher evaluation system was a best practice given by the state. The district would work to meet the mandates and it would improve teacher practice. Admins stated the teacher evaluation system was a practice given by the state. The district would work to meet the mandates and it would improve teacher practice and conversations about practice. Admins stated the teacher evaluation system was a practice given by the state. The district would work to meet the mandates and it would improve teacher practice and conversations about practice. Informal Communication (between teachers) on the Designation of “Highly Effective” status and performance pay Teachers believed Admins made as few changes as possible. They trusted that the Admins would make the best decisions. The teachers, union representati ves, and Admins would figure out the system together. Teachers did not know about the importance of selfevaluations during the first year. They were clear on how to achieve “highly effective” status in successive years. Teachers believed the building Admins believed in the state guidelines and mandates. They saw the state and Admins as being on the same page. Teachers believed the building Admins believed in the state guidelines and mandates. They saw the state and Admins as being on the same page. Teachers believed the building Admins believed in the state guidelines and mandates. They saw the state and Admins as being on the same page. Teachers believed Admins said they were properly implementing the state mandates, but teachers believed there were great variations. What was professed as close following was not really close following to teachers in the district. Notes: Admins- Administrators 165 These comparisons between districts show the “how” of communication and decision-making. Without a formal plan, like Grote (1996) suggested, communication by the district and building administrators varied between the building and the district, in some cases, and between administrators within the building, in bigger districts. The “how” of communication and each district’s approach to the state law are portrayed. The reader should remember that teachers viewed the evaluation systems as most acceptable in Districts 1 and 2, so their formal and informal communication seemed to have an important relationship to teacher perceptions. In terms of district follow through on their announced plans, teachers in Districts 2, 3, and 6 described variations in what the district said was going to happen with respect to three evaluation markers: reliability among evaluators, consistency among evaluators, and end-of-the-year evaluation meetings. When using this lens to look at perceptions, District 2 did not seem to have such negative perceptions. Districts 3 and 6 had very negative teacher perceptions. According to both teachers interviewed, District 4’s central office administrator made a statement at the introduction of the teacher evaluation system indicating that attaining a “highly effective” rating was “almost impossible” or “next to impossible.” According to the teachers interviewed, several teachers in other buildings received “highly effective” ratings and significant performance pay bonuses. It was the high school teachers’ perception that other building administrators used “highly effective” ratings to reward teachers who went along with their initiatives or were seen as generally supportive of the administrator. The difference between what was said to begin the year and what was done to end the first year of implementation caused distrust toward the system and its implementation. In Districts 1 and 2, district administrators formally told teachers that they were all figuring out the system together. Even though District 1 heavily weighted self-evaluations without telling the teachers they were going to do so during the first year, their teacher perceptions were the highest in the sample. The lead administrator in District 1 expressed significant frustration with the state’s evaluation mandate, devoid of guidance on “how to do it.” The District 1 administrator told faculty and staff “we are building the plane while we are flying it.” The District 2 administrator was not as overt in his/her admission of less confidence in the system. S/he did give class time and communicated to teachers, “we will figure this out together.” The usage of faculty meeting time to allow teachers to talk with one another, within disciplines, about how to meet the mandates of the new teacher evaluation law seemed to indicate, to teachers, that this would be a combined effort, not one that administrators would figure out on their own and then tell teachers. 166 In District 3, the administrators also used faculty meeting time. The tone of those meetings, however, was set by the administrators designing activities and discussions so that teachers could understand the teacher evaluation system like the administrators did. This might be a slight difference, both Districts 2 and 3 dedicating faculty meeting time to evaluation, but the tone was important. The evaluation system was not open for question or debate in District 3. It was to be learned from those who understood it, the administrators. Districts 4, 5, and 6 teachers shared the belief that the system was set and clearly explained. In District 4, the system was built into the school’s improvement plan. By including it in the school improvement plan, it showed a sense of confidence in the system. It was what the school was doing, and administrators communicated that they were confident in the system and its implementation. In Districts 5 and 6, the evaluation system was not part of the school improvement plan, but it was what the state ordered the district to do. It needed to “get done.” In Districts 3, 4, 5, and 6, this confidence in the system through communication seemed to influence teacher perceptions of the system and its implementation as a whole. Clarity and consistency about achieving “highly effective” status as well as the stakes or potential stakes those ratings carried with them seemed permeate discussions in Districts 4 and 6, where performance pay was implemented based on teachers achieving highly effective status. In District 4, there was a perception, in the high school building, that very few teachers, if any, would achieve this ranking. At the end of the year, as faculty members between buildings met, it became clear, to the teachers in the sample, that “highly effective” was bestowed more in other buildings. That year, performance pay was considerable. In following years, the money in District 4 was far less, as more teachers achieved “highly effective” status. Furthermore, one teacher in District 4 felt that if teachers did not sign up to do a large amount of work, the highest status they could achieve was “effective.” This pre-determination of “highly effective,” based on what teachers said in the beginning-ofthe-year meeting, seemed to influence teacher perception negatively. Here is how one teacher from District 4 described it: “I am a teacher in the high school. In our first year, no one received a highly effective rating, as the district lawyers told the superintendent that would make it hard to remove them from their position. That clearly impacted morale. In year two, four teachers in our building received highly effective ratings; almost everyone else was rated effective. Again, that caused a problem with morale, as some teachers receiving an effective rating were clearly more dedicated and effective than others receiving the rating. I think that, as a whole, teachers are less dedicated to professional areas like professional development and serving the district on committees, as they can receive an effective rating without demonstrating these areas of professionalism.” 167 District 4 also laid teachers off based on the rankings of the teacher evaluation system. Negative job action raised the stakes and the importance of the teacher evaluation system. District 4 was the only district to lay off teachers during the implementation. Although these lay-offs may have been correct and law-compliant, it raised the level of concern about the system and its implementation. District 6 implemented performance pay based on “highly effective” status. The amount was under 25 dollars, so teachers thought it was a joke. The amount of performance pay was not communicated to begin the year. Once it was given by administrators, it was so low that teachers saw it as a joke. At the same time it was a joke, it also communicated to teachers that positive compensation amounts would be awarded based on the teacher evaluation system, which raised the stakes of the system’s importance. District 3 struggled with their communication on teacher evaluations because they first said teachers interested in “highly effective” status needed to apply. The list of expectations was great. The idea that a teacher had to “apply” brought criticism. After a district-wide survey showed teacher and administrator contempt for the application process, the district changed the application requirement, but the expectations did not change much, if at all. Therefore, “highly effective” seemed to demand a lot more work with no extra pay. This change of expectations, along with the long list of expectations, lowered morale and acceptability in the system and its implementation. Districts 1, 2, and 5 steered clear of the issue of performance pay and applying for “highly effective” status. They also did not have to lay off teachers, this for many reasons. First, the enrollment in these districts may not have dictated laying off teachers, but also not having the money to deliver on performance pay. It also might have been a system-wide decision to take time to comply with the law. Districts 1 and 2 may have enjoyed more positive teacher perceptions because of this district decision. No questions were asked about why the districts did this, so reasons can only be speculated. Districts 1 and 2 also made it clear that the state made them do this, distancing themselves from the mandates, performance pay, and the stakes of “highly effective” status. Building administrators communicated to teachers that the “state made them do it.” This really was true in all of the districts. Administrators in both Districts 1 and 2, however, told the teachers they would not have done this, had it been their choice. They also communicated some disdain with the state and, as they saw it, the state’s inadequate guidance on the mandate. To paraphrase: “The state made us do this. We don’t want to do it. We have to do it. The law is very unclear in several sections. We will figure it out together.” In Districts 3, 4, 5, and 6, the positionality to the state started similarly; however, it changed later in the explanation. To paraphrase: “The state made us do 168 it. It is what we are doing. We need to get it done. We will make our system clear to you. If you have questions, we will explain it to you.” By not explaining the degree of distance from the state, Districts 3, 4, 5, and 6, whether they wanted to do this or not, became emissaries of the state. Table 40 displays a map of the different degrees of separation among district administrators with respect to the state’s mandate. Districts 1 and 2 were on the “far away from the state” end of the table; Districts 3, 4, 5, and 6 seemed to position their actions “close to the state.” 169 Table 41: Degrees of Administrator Distance from the State’s Teacher Evaluation Mandate Degrees of Distance from the State Closest To State Closer to the state Close to the state Away from state Far away from state Furthest away from the state Formal Communicati on from building The state made us do it, but it is a good idea The state made us do it The state made us do this, and we would not have chosen this. However, administrator s will figure it out what the state wanted and tell teachers how to do it The state made us do this, and we would not have chosen this. However, administrat ors will do the bear minimum to meet what the state wants districts to do on paper. The state made us do this; we did not want to do it; the state did not provide guidance or direction; we only will do as much as we have to do in order to comply; we (teachers, administrators, union representatives) all will figure this out together. The state made us do this; we are not going to do what the state says. Informal communicatio n among teachers Administrat or and the state are one Administrator s will interpret the laws for teachers. Administrator s don’t totally agree, but they are the experts. They know what the state wants. They will show us how to do it. District will only do the bear minimum to meet mandates on paper. Administrators and teachers are in the same boat. Teachers should give administrators the benefit of the doubt. The district is not complying with state mandates. During the time of implementation, the state effectively forced districts to cap teacher health benefits, increased teachers’ contributions to their retirement system, and precluded salaries increases by holding per-student funding the same, despite falling enrollments in many of the sample districts. Districts 1 and 2 administrators, who kept several degrees of distance between themselves and the state mandate, achieved greater acceptability and impact on morale. District administrators created a sense that the law was happening to them; they did not believe in it; and teachers and administrators would figure it out together. District administrators were not emissaries of the state. The themes and communications about confidence, positionality with respect to the state, clarity and consistency in “highly effective” designations, and the stakes of these ratings are more fully described in the synthesis chapter that follows. 170 CHAPTER 5: SYNTHESIS OF FINDINGS Michigan’s legislature passed a series of laws between 2008 and 2011. These laws mandated school districts and educational entities to change the way that they evaluated teachers. The laws were influenced by various factors, including an increase in control of state funding since 1994’s Proposal A, Michigan’s failure in Race to the Top, a desire to increase educational efficiency by both political parties, and several assumptions about teachers, administrators, and school districts. Districts across a county in southeastern Michigan responded to these changes. These district responses, the independent variable in the study, varied for several reasons: the district’s financial context; the union-administration relationship and history; limited knowledge of implementation steps and human resources research; lack of guidance from the state on framework, tools, or implementation; collaboration not encouraged by the laws; and an overall emphasis on getting it done to comply with state law, fearing state intervention if the district did not cooperate. This study has framed district responses by reference to Grote’s (1996) implementation steps for performance appraisal systems. Teachers and administrators developed perceptions of the system and its implementation over the next two to three years. The perceptions are the dependent variable in the study. Such perceptions may have been influenced by the teacher’s own ranking, teaching assignment, official union position, experience level, age, gender, race, compensation level, background, party affiliation, view of his/her own job performance, and negative opinion of the state. These factors may have influenced individual responses among teachers. The factors that were measured— experience level, gender, teaching assignment—varied across the study. Perceptions were organized initially by Noe’s (2006) attributes of an effective performance appraisal system. Several key findings emerged from the perceptions of teachers. First, because of a lack of knowledge of Grote’s recommended implementation steps, a generally negative or neutral perception was held by teachers about the implementation of the teacher evaluation system and the system itself. By reviewing Grote’s steps (1996) and Noe’s attributes (2006), some differences between districts emerged. Two districts, Districts 1 and 2, achieved neutral and comparatively positive perceptions because of several factors. These included administrators’ communicating confidence, establishing consistency in their designation of “highly effective” status, establishing clarity for teachers regarding how they should do their jobs in the new evaluation system, and being sensitive to teachers’ perceptions of the stakes associated with performance pay and teacher lay-offs. In Figure 1 below, a conceptual map displays the variables with possible causes, intervening or antecedent variables, and a synthesis of perceptions. 171 Figure 1: Conceptual Map of Teacher Response to Michigan’s Teacher Evaluation Laws To synthesize the findings, I will review the implementation steps that districts did and did not follow. I will also consider districts’ rationale for not following implementation steps recommended by Grote. And I will review factors that were related to more positive perceptions of teachers. Grote’s implementation steps will be bolded to help the reader identify them. Implementation Steps Districts Followed Closely All of the districts included central office administrators, building administrators, and teachers in their “evaluation teams.” The methods of enlisting administrator and teacher involvement varied, but each district included employees at all three levels of the organization: central office administration, building administration, and teachers. Most districts utilized their union contacts to gain participation from primary and secondary teachers. Some districts asked for participation, and a majority of the teacher participants held official union positions. Communication about selection, about discussions in the “teacher evaluation team,” and about the impact of those discussions was inconsistent and really varied by district. All of the districts did have participation from all levels of their organizations. 172 All districts also had administrators with central office responsibilities were involved with the teacher evaluation teams by getting the teams together, formulating agendas, and facilitating the meetings. The district administrators had copies of agendas of the “teacher evaluation team” meetings. Minutes or follow-up on these meetings were not submitted by district administrators. Central office administrators, though they may have been reading and evaluating administrators’ evaluations, did not clearly communicate their roles in the implementation of the teacher evaluation system. At minimum, this is how building administrators and teachers described central administrators’ role. All of the districts provided training for sharing feedback to teachers on how they performed. Training on the frameworks varied among districts. Much of the variance had to do with many of the districts using some version of the Danielson (2007) framework for teacher evaluation. Without major changes to the framework, districts depended on previous training on teacher evaluation frameworks with their administrators and focused on the feedback tools. In addition, several districts had building administrators meet with each other to “tune” their approaches to teacher observations and evaluations. The training on frameworks and tools, however, were not communicated broadly to teachers, so teachers were generally unaware of the training and tuning that building administrators did to improve their reliability on teacher evaluations. All of the districts communicated to their teachers the process, framework, and tools of the teacher evaluation system in general terms. Teachers seemed to understand the logistics of the teacher evaluation system. Districts did struggle, however, to define the “how to” of the frameworks. Teachers had trouble demonstrating their proficiency, or mastery of descriptors, on the framework. Many of the frameworks were too general because they were used from kindergarten through 12th grade, as well as for special education teachers, social workers, and counselors. While the logistics and broad descriptors of the frameworks were communicated to teachers and staff, challenges arose with the actual usage. Implementation Steps Districts Did Not Follow Closely or At All Based on the informal manner districts undertook understanding what their teachers wanted or a needs assessment, teachers not on the team or connected with the union did not know about or understand the districts’ attempts to identify areas to improve the current teacher evaluation system. When the “teacher evaluation teams” met, minutes were not taken or distributed to keep teachers at large apprised of the team’s discussion or progress. Therefore, many teachers felt they did not have initial input into the creation of the teacher evaluation system. Lack of specificity, relevance to their jobs, and a lack of acceptability were the resultant teacher perceptions of the evaluations. 173 None of the districts showed evidence of a formal communication plan. All of the districts communicated informally. There was no formal plan on how to communicate with teachers about the new teacher evaluation system. Thus, much of the communication depended on the communication abilities of building administrators. In districts with more building administrators, the idiosyncratic nature of the communication caused differences in the way teachers understood expectations and perceptions of the teacher evaluation system. Only one of the districts had any assessment of the teacher evaluation system itself. Only District 2 re-convened its teacher evaluation team. Administrators described a sense that the state eventually would dictate a system for all schools to use. At the time of this writing, the state had not done that, so districts were waiting for the state to tell them what system to use and how to use it. That did not occur at the time of this writing, and it has been three years since the initial changes to the law. While waiting for the state to act, districts’ teacher evaluation teams were disbanded when they could have been useful to improve the districts’ teacher evaluation systems. Teachers meanwhile perceived the system to be set and held little hope that it would improve. Some steps in Grote’s (1996) implementation process were not utilized at all. These included a pilot and accountability of building administrators for their evaluations of teachers. Such completely missed steps contributed to teacher perceptions being “neutral” at best. None of the districts studied piloted their system. A pilot year to study, to understand, and to improve the system was absent. The lack of a pilot seemed to influence teacher perception in several ways. First, the system’s logistics could have been worked out. Second, teachers would have felt that their input to the system was heard and the system could be improved as the district continued to use it. Third, teachers could have gotten to know the system and to understand their administrators’ expectations within that system. Administrators could learn the system and make mistakes because the system itself was being piloted. The word “pilot” itself indicates a test which would not necessarily produce “highly effective,” “effective,” “minimally effective,” or “ineffective” designations. The state required each district to submit to it a designation of “ineffective”, “minimally effective”, “effective”, and “highly effective”. District leaders may have felt pressure to comply with the new laws and give each teacher a ranking among these options. Since future layoffs were a clear possibility for some, it may have been necessary for districts to issue lay-offs given funding trends during the implementation. Though a pilot is what business and human resource management might recommend, the possibility of lay-offs made a pilot without designations impossible. A consistent finding was that teachers did not understand that their evaluators’ performance was being assessed as they evaluated teachers by the district’s central office. Administrators, in teachers’ view, were not accountable for their 174 appraisals of teacher performance. Aligned with this concern from teachers was the feeling that no administrators at central office were reading the teacher evaluations. The only evaluations that received any analysis were the ones that were contested by teachers. This led to the feeling among teachers that they were accountable for more work, yet their administrators were not accountable for their assessments of that work at all. Teachers, in their view, were doing a lot of work for the new evaluation system, and were held accountable for things outside of their control, while administrators were not accountable for their evaluations. Among teachers this stirred up considerable stress and confusion about the evaluation system and its implementation. Rationale for Lack of Implementation Coherence There are several possible reasons for lack of coherence to Grote’s (1996) implementation steps. They include, but are not limited to, a lack of time for implementation, a lack of guidance from the state, a lack of human resources or labor experts in individual districts, and a lack of knowledge about Grote’s implementation steps. With respect to the time variable, business research indicates that three to six years are optimal, whereas most districts had less than one year. These possible reasons for a lack of compliance are described below. The lack of time for implementation was a key factor influencing district response to teacher evaluation legislation. According to Grote (1996), most businesses implementing a new performance appraisal system would implement it in a period of three to five years. They would have time for a formal needs assessment, time for inclusion of stakeholders, time for a communication plan, time for a pilot, time for developing an assessment of the system, and time for adaptations of the system. Districts did not have even one full year. Without time, districts could not use approaches that emphasized collaboration or were defined as “best practices” by Grote. Teachers saw this as a lack of collaborative spirit from the districts. But from the district administrators’ perspective, there was no time to do such things. Districts were also promised guidance from the state. The perceptions of district and building administration in the study was that the state eventually would dictate the framework and the tools, so why should they spend time “re-inventing the wheel”? That guidance came in the form of recommendations from the Michigan Council for Educator Effectiveness (MECC), led by university professors and practitioners. Legislators did not heed their counsel, however. Pilots were run by the MECC; recommendations made by the MECC; and the legislators did not act. The promise from the state was never fulfilled, and district and building personnel moved onto to the next issue. 175 Five of the six districts in the study lacked a full-time human resources person to guide the teacher evaluation system. In the study, only one district could afford such a dedicated position. With the financial distress of Michigan schools, the focus was on getting things done and making sure that those things were legally compliant. The districts would have been better served thinking about what would work best from a human resource perspective. There was really nowhere else for them to turn for guidance on how to implement the system. Each one needed a human resource professional. Without state guidance, or the funding to afford in-house guidance, districts focused on simply meeting the requirements of the law. Grote’s implementation steps were not communicated at all to district or building administrators. In addition, many features of the evaluation process were defined as “non-negotiable” in the collective bargaining agreements under the new state laws. Districts knew that they were not required to negotiate with employees regarding how they implemented the evaluations, so they tended to disregard attention to forms of teacher involvement that was may have proven more successful for the operations. Lawyers said one thing; district administrators believed another thing; and building administrators thought still something else with respect to collaboration. Without knowledge of steps to guide them or clear definition by the law, districts did what they thought was best. That limited their effectiveness in implementation. The changes in Michigan’s teacher evaluation laws made this clear that: evaluation procedures/systems could not be collectively bargained. This seemed to encourage or to require districts to obtain teacher feedback, but they needed to make the final decision on the evaluation framework, tools, and system because of the short time line allowed by the state. Because collective negotiations were not allowed, a focus on “getting it done” overshadowed the teacher evaluation teams. It may also have contributed to the lack of follow-up, assessment, or improvement of the system. With a complex and chaotic environment in Michigan schools, administrators felt they did not have time to facilitate collaboration and were not allowed to do it anyway. Districts’ financial straits were another rationale for the lack of compliance. There was no funding available from the state for developing, or for buying, a new teacher evaluation system. In subsequent years, such funding has been discussed, but it has not yet occurred. Therefore, districts had to fund all of this work out of their general monies or utilize already budgeted professional development dollars (typically earmarked for improving teaching and learning) if they were to utilize Grote’s steps. Therefore, the limited resources seemed to be a reason for a lack of implementation coherence. 176 Challenges with Grote’s Framework There were some challenges with applying Grote’s framework and applying it to school districts. Grote’s framework assumed three to five years to implement (1996). Districts did not have that, so it may be asking too much to hold them accountable for an implementation that was between one and two years. The number of meetings Grote suggested for the implementation team arguably were not reasonable. Working with union representation, especially when negotiation is not legal, was not addressed at all. Given the many businesses that operate in collaboration with unions, the lack of operating within a union context was surprising. Finally, steps like ongoing collaboration, communication, pilots, and ongoing improvement were more important, based on Grote’s writing on them; however, there was no explicit description of the strength or importance of any of the steps. This made some of the analysis difficult because all steps were not equal in importance, so the importance had to be inferred. Findings About All Districts and The Differences Between Districts A consistent finding among districts was that perceptions of this implementation varied by employee position type. Whereas most district and building administrators viewed the teacher evaluation system, as positive or neutral, most teachers viewed the system negatively. In the study’s interviews, union representatives described their perceptions in more negative terms than did teachers without official union positions. None of the six union representatives characterized the evaluation system as “acceptable.” Of the six teachers interviewed, who had no union position, five of six said that the system was acceptable or that it depended on the teacher. In the online surveys, many more teachers, without union positions, viewed the system and its implementation negatively. District 1 and District 2 were the exceptions in terms of the way that teachers described communication, and how well teachers understood expectations. In those districts, more teachers viewed the implementation neutrally, in terms of its acceptability and impact on morale, than in other districts. In Districts 1 and 2, teachers seemed to trust their administrators and give them the benefit of the doubt. Despite the misapplication or total omission of Grote’s implementation steps, Districts 1 and 2 had higher acceptability and influence on teacher morale. Why was that? Table 42 below lists teacher perceptions in the areas of acceptability, teacher morale at the building level, and teacher morale at the district level, all from electronic surveys. 177 Table 42: Teacher Survey Perceptions of Acceptability and Morale at the Building Level District Number Union Representative Perceptions (Survey) 1/1- Unacceptable 1/1- Diminished morale Teacher perceptions with no union position (Survey) 1/1- Acceptable N/A Teacher perceptions (survey) 3/3-Neutral 3/3- Neutral 1/1- Unacceptable 1/1- Acceptable District 2- Morale (Building Level) District 3- Acceptability District 3- Morale (Building Level) District 4- Acceptability 1/1- Diminished morale N/A 1/1- Unacceptable 1/1- Greatly diminished morale 1/1- Unacceptable 1/1- Unacceptable N/A 2/3- Acceptable 1/3- Unacceptable 2/3- Neutral 1/3- Unacceptable 3/3- Unacceptable 3/3- Negative District 4- Morale (Building Level) District 5- Acceptability 1/1- Diminished morale N/A 1/1- Unacceptable 1/1- Acceptable District 5- Morale (Building Level) District 6- Acceptability 1/1- Diminished morale N/A 1/1- Unacceptable 1/1- Acceptable District 6- Morale (Building Level) 1/1- Diminished morale N/A District 1- Acceptability District 1- Morale (Building Level) District 2- Acceptability 1/1- Neutral 1/3- Acceptable 2/3- Unacceptable 1/3- Neutral 2/3- Unacceptable 1/3- Neutral 2/3- Unacceptable 3/3- Negative 2/3- Neutral 1/3- Negative 2/2- Negative In the interview phase of research, the perceptions were very mixed. One could assume that the face-to-face interview with a researcher who was also a high school principal led teachers to soften their perspectives because they may have feared repercussions. All the interviews took place during teachers’ preparation periods or after school, in their rooms or the school library/media center, so there was a chance that the building’s administrator could walk into the interview. It seems plausible that this factor could have impacted teachers’ willingness to be completely frank in their comments. The mixed results of the interviews are detailed in Table 43. 178 Table 43: Teacher Interview Perceptions of Acceptability District Number Perceptions of Acceptability 1 Yes- 1, No-1 2 Yes-1, Mixed-1 3 No-2 4 Mixed- 1, No-1 5 Yes-1, No-1 6 Yes-1, No-1 These findings indicated that Districts 1 and 2’s implementation were associated with somewhat higher acceptability. When teacher interview responses were linked to electronic survey responses, three teachers changed their perceptions of acceptability during the interview portion of the study. Therefore, the electronic survey responses will be used to differentiate among districts because the details and descriptions were more plentiful in the electronic surveys. Based on the electronic survey results, what was different about the way Districts 1 and 2 implemented their system? The following section first reviews factors that did not seem to relate strongly to the more positive teacher perceptions in Districts 1 and 2. Then, aspects of communication that seemed to relate to District 1 and District 2’s positive perceptions will be described and discussed. Formal and informal communication within the district, in terms of communicated confidence, clarity and consistency on designating “highly effective” status and overall job expectations, the potential stakes of ratings, and positionality to the state’s mandates, all seem to relate to the district’s positive perceptions. The section will also review some interesting, yet limited findings, on how well teachers understood they needed to perform. The decisions made in these areas, and the ensuing communication from administration to faculty, and from faculty to faculty, revealed major disconnects in districts with more negative teacher perceptions. Districts 1 and 2 approached these topics, and their communication about them, similarly. Districts 3, 4, 5, and 6 did as well. Several other variables—including district follow-through on promised processes, the stability of district leadership and district economic context did not show any systematic association with teacher perceptions. Follow-through on the promised processes of the district occurred in Districts 1, 4, and 5. District 1, as mentioned above, did receive a higher acceptability and morale response. Districts 4 and 5 did not. Several teachers reported that District 179 2 administrators did not all follow through on their outlined process. Therefore, follow-through did not seem to relate to higher acceptability perceptions. Stability of leadership initially seemed to be a factor in trust and acceptability. In Districts 1, 2, 5, and 6, the high school building administrators were employed by the district for a minimum of four years. Those building administrators may have held other roles, but they were working in the district. It seemed that they may have built trust by working in the district or working their way up to leadership roles within the district. In Districts 1 and 5, building administrators had been in their roles for longer than five years. Despite having administrators who worked several years in the district prior to the system’s implementation, some districts with experienced leaders or employees did not have positive teacher perceptions about it. Economic context is somewhat difficult to quantify, in terms of a variable for this study, because all districts in the study sustained reduced funding. In this analysis, lay-offs and state-mandated debt-reduction plans serve as a measure of each district’s economic well-being. One might imagine that impending lay-offs because of reduced student enrollment and state funding could cause evaluations to take on a different tone and importance, lowering the acceptability by teachers and morale of teachers. District 4 experienced lay-offs at the high school level during the implementation. District 6 was on a statemandated debt-reduction plan, but had not reduced its teaching staff during implementation. Districts 1, 2, 3, and 5 did not experience a debt-reduction plan or lay off teachers. Yet, Districts 3 and 5 still had more negative teacher perception responses. Aspects of formal and informal communication, both formal and informal, seemed to show a relationship to teacher perceptions. These dimensions of how much confidence administrators communicated they had in the district’s teacher evaluation system, the usage of “highly effective” designations and performance pay, and distance created between the state’s mandates and the district’s philosophy and action. The discussion will start with communicated confidence. Whether a building administrator is dealing with parents about a discipline issue, teachers about the best way to teach, or district administrators on how to manage building operations, confidence and bravado may seem essential to his or her success. A building administrator may need to exude confidence, in the system, in the state’s intentions, and in their own/their own team’s ability to implement consistently. In implementing a new teacher evaluation system, one might think that the same kind of confidence is necessary. In Districts 1 and 2, however, building administrators admitted that the system was not perfect and that their confidence in their performance was not absolute. They remarked that they would need to work with their teachers to figure out how to use the 180 system. One District 1 administrator described the growth data side of the evaluation this way: “We don't have answers for it. We're confused, too.” In Districts 3, 4, 5, and 6, building administrators may have felt a lack of confidence in their ability to implement evaluations, the state’s intentions, the districts’ interpretations of the law, or their own administrative teams. In these districts, however, there was no admission of flagging administrator confidence in their own abilities or in the system. It was part of the school improvement plan; it was something the principal needed to teach teachers; it was part of instructional initiative; or it was something that the state had to do. None of these messages communicated to teachers that administrators and teachers would be working together to figure out the system. In discussing the role of the teacher evaluation team, a District 3 official union described how the district ignored suggestions by the team because of confidence in “their” way of doing things: “They listened. They took notes, then they did what the hell they wanted to do anyway.” This show of confidence by administrators, whether they actually felt confident or not, gave teachers the sense that administrators were sure of their evaluations, even in the first year of the system. Without the district communicating a system for reviewing evaluations, teachers may have felt they did not want to “rock the boat.” In Districts 3, 4, 5, and 6, administrators were confident; there was no system of checking the evaluations; and the laws seemed to be against teachers. This confidence then seemed to influence teacher perceptions negatively. Clarity and consistency on “highly effective” status and the increased stakes of distributing add-on performance pay were areas of district decision-making and communication that seemed to relate to teacher perception. Section 380.1250, 1 of Public Act 205 states: “A school district, public school academy, or intermediate school district shall implement and maintain a method of compensation for its teachers and school administrators that includes job performance and job accomplishments as a significant factor in determining compensation and additional compensation. The assessment of job performance shall incorporate a rigorous, transparent, and fair evaluation system that evaluates a teacher's or school administrator's performance at least in part based upon data on student growth as measured by assessments and other objective criteria.” Districts interpreted this law in several different ways, similar to the interpretations of how to designate “highly effective” status. In Districts 1 and 2, “highly effective” status and performance pay were handled carefully by building administrators. In District 1, teachers’ self-evaluations weighed heavily the first year and still were important in the second year. Although one teacher described not knowing the weight of the self-evaluations in District 1, overall, teachers felt it was clear how to achieve a “highly effective” rating. Performance pay was not mentioned or used. In District 2, the same was true. The “highly 181 effective” rating, if work-intensive, was achievable and clear to teachers. Add-on performance pay was not mentioned, other than that the state wanted districts to do it. Therefore, the stakes were lower in these districts. In Districts 4 and 6, add-on performance pay by the second year was a menial amount of money, under 50 dollars total if a teacher received “highly effective.” Those closely complying with the state mandate caused teachers to think that the amount of money was insulting, rendering the whole endeavor silly. It gave the teacher evaluation system a negative tone in those districts. A core teacher without an official union position described the performance pay this way: “I was considered a highly effective, outstanding teacher, and I got a $10 addition to my pay. . .I have been insulted by state law.” In Districts 3 and 5, teacher lay-offs were threatened (based on rankings from the teacher evaluation system), but were not implemented. In Districts 3, 4, 5, and 6, performance pay and lay-offs raised the stakes, as well as the scrutiny, of the district’s whole teacher evaluation system. With these higher stakes in place, each district might have wanted to be more clearer and more consistent in their expectations and communication of those their expectations. According to teachers in those districts, this did not happen. Furthermore, by increasing the stakes in the way that the state mandated, districts seemed to be an emissary of the state without any distance created between the state’s mandates and the district’s action. In District 3, the district and building administrators at first said that teachers had to apply for “highly effective” status. After teachers expressed concern about the need to “apply”, and the perception this gave teachers, the district changed the requirement that teachers should “apply.” Still, the district kept rigorous expectations for teachers to reach this status. Whether teachers’ perceptions were shaped by the district’s altered approach to applying or by the stringent requirements, teachers felt that the “highly effective” status was out of reach. A District 3 non-core teacher with an official union position described the “highly effective” situation this way: “But that was a cluster. They shot themselves in the foot. That was a huge mistake on their part.” Because of decreasing enrollment and reduced funding, Districts 3, 4, 5, and 6 threatened teacher lay-offs based on the teacher evaluation system. Only District 4 followed through on teacher lay-offs based on the teacher evaluation system. In District 5, the dual initiatives, a new way to approach teaching and learning and a new teacher evaluation system, left the teachers in the study unsure on what was expected of them. They described the teacher evaluation system as too broad, perhaps not as specific as the other initiative on common language and practices for teaching and learning. There was also the fact that an overlap existed between the initiatives, but two sets of language, two sets of goals (one assessing teacher performance, and the other, teaching teachers a new way to teach) caused confusion. However the initiatives meshed and 182 tangled, teachers were not clear on how to achieve “highly effective” status. A core-area teacher with an official union perspective described the lack of clarity this way: “As teachers we wanted to have concrete vocabulary of what a highly effective teacher does, or looks like, or says, or is doing in class.” District 4’s teachers also described issues with “highly effective” status. One teacher remembered the superintendent saying that no one was going to get “highly effective” status the first year. A second teacher recalled the superintendent predicting that it would cost teachers their jobs. S/he described the superintendent’s statement this way: At one point our supe said, "I'm going to tell you straight up, folks. There would not be a lot of highest ratings, because if that's the case, you talk yourself out of a job, if we are directed to, pulling the corners, and there's nowhere else to put you, because you are so uniquely, highly qualified in your area." As high school teachers understood it, several teachers throughout District 4did achieve “highly effective” status that first year, despite what the superintendent had said to start the year. In addition, performance pay was given to those who achieved this status. The disconnect between what was said by the superintendent said at the outset of the first year and what teachers heard informally about the “highly effective” designation seemed to influence teacher perceptions negatively. Grote’s claim that employee clarity in regard to how they should perform their duties is important for their perceptions of the evaluation system is certainly plausible. Union representatives, building administrators, and district administrators were asked about this in each district. So in each district, the union representative was the only teacher who addressed how well teachers understood how to perform. While administrators may be prone to put a positive spin put on the implementation, the research had limited data from teachers. Despite the limited nature of these findings, official union representatives in Districts 1 and 2 indicated that teachers understood well the expectations of how they should perform, no systematic association between this variable and teacher perceptions was evident across districts More research on this area could reveal a stronger relationship. A fourth area of communication and decision-making was “positionality” with respect to the state’s mandate. In Districts 1 and 2, building administrators clearly communicated that this was what the state was making the district do. They made it clear to teachers that the district and they themselves would not have chosen this method to assess teacher performance. One core-area teacher with an official union position described the way building administrators described their position to the state mandate: “It was clear they didn't buy into what the state was saying.” 183 In Districts 3, 4, 5, and 6, what the state said was what happened. There was no degree of distance between district or building action and what the state was requiring. In Districts 3, 4, 5, and 6, what the state and district each wanted merged into what the district itself wanted. One non-core teacher with an official union position described their administrator’s position to the district’s system this way: “(our administrator) particularly seems to be very, very married to the process. The Charlotte Daniels system specifically takes it right to the letter. Letter of the law, literally.” In other districts, administrators did not question the need, the method, or the particulars of the mandate. The state said it, and it had to be done. Further damaging in the approach to positionality was when building administrators explicitly or implicitly communicated that they would figure it out for teachers and then communicate it. This “figuring it out for teachers” made it seem like the state and the district were locked in step. This approach and its lack of distance seemed to relate negatively to teacher perceptions. Privately, administrators from these districts might have described concerns or a desire to distance themselves from what the state was mandating. Formally, they communicated that the system was valid. Informally, teachers communicated with one another that the district was taking the state mandate literally and implementing it to the “letter” of the law. It is difficult to separate all the other moves the state made at the same time from mandating teacher evaluation law. Whether it was holding funding steady, while forcing teachers to pay more for retirement benefits, reducing those retirement benefits, or requiring districts to force teachers to pay more for health insurance, teachers felt under attack. Teacher evaluations were an area where districts had a choice where they stood and to what degree they wanted to agree with the state. Regardless of rationale, if the district did not publicly question the wisdom of the state’s mandates and create some degrees of distance between themselves and the state, it seemed to influence teacher perception negatively. Closely intertwined with the positionality issue, is that Districts 1 and 2 implied to teachers that the state did a poor job mandating teacher evaluations, and that the building administrators would work with teachers to figure out how to implement the system. A District 2 teacher with an official union position described working with his/her principal to gain understanding: “We did have a lot of conversations last year about it as a union rep. The other rep and I had multiple meetings with the principal. Talking about it and clarifying.” Districts 3, 4, 5, and 6 seemed to be closely aligned with the state and its wishes. This seemed to relate to negative teacher perceptions on the system and its implementation. There are also intangible factors not researched in this study that seemed to relate to teacher perceptions. These include legitimacy and background of building administrators, trust in building and district administrators, teachers’ own 184 rankings and their influence on perceptions, and the history of union and administrative relationships in the district. These will be described below. The first intangible leadership factor that was hard to measure in this study was the legitimacy of the building leader/s. Was this person ever a teacher? What was her/his knowledge of teaching and learning? These issues seemed to permeate teacher responses, yet were difficult to ascertain by the questions asked. Expectations, communication, and followthrough seemed to relate to building legitimacy. How the administrator/s talked to teachers, how they administrator reacted to differences in opinion, and how they administrator led the building all appear seemed to be related to teacher perceptions. They were difficult to quantify into a variable, but seemed to be related to the way teachers perceived the teacher evaluation system and its implementation. Teachers’ own evaluation ratings seemed to influence their perceptions. If a teacher was rated “highly effective,” then the framework, the process, the tools, and the implementation seemed to matter less to them. Were teachers who achieved higher rankings or status more likely to take part in the study? Did the amount of time and the overall risk a teacher took by participating in the study influence their perceptions? The research cannot firmly answer these questions. In further research, a relationship between teacher rankings and their perceptions could be explored. Finally, each district’s relationship to its teachers’ union seemed to matter. Distrust or a general lack of stability seemed to influence perceptions. It was hard to quantify this lack of trust with district or building administration. Districts had to ask teachers to pay more for benefits, while holding their compensation steady. This fact seemed to influence teacher perceptions. It is difficult to say how much this relationship influenced teacher perception, but there were suggestions in the interviews and surveys that it did. 185 CHAPTER 6: CONCLUSIONS AND IMPLICATIONS FOR PRACTICE The data has been analyzed and viewed through several different lenses, including by question, by position, and by district. Although many of the districts followed similar steps, as framed in 1996 by Grote, Districts 1 and 2 enjoyed more positive perceptions from teachers, albeit perceptions deemed “neutral” on questions of acceptability and influence on teacher morale. Leadership variables such as administrators honestly appraising their lack of confidence in themselves, the state, or the law; careful handling of difficult topics like “highly effective” status and performance pay, district’s communication clarity on expectations, and the district’s positionality (far away is preferable) to the state’s mandates all seemed to make the difference for Districts 1 and 2. Several conclusions can be drawn from the data analysis and synthesis. Perhaps unwittingly, state legislators placed districts in a most unfavorable position to implement the law. Leadership and trust factored heavily into the success or failure of implementation, as did communication. Taking time to focus and execute could have helped all parts of implementation. Districts could have used the main ingredients of Grote’s implementation framework to make the teacher evaluation system work. The state put districts in a very difficult position to implement their teacher evaluation systems effectively. The first area that showed up throughout the study was the promise that the state would define an observation framework and a teacher evaluation system shortly after the passage of the original law. Answers to tricky questions about student achievement growth, “highly effective” status, performance pay, and weighting of different components of the system were all assumed by districts to be “right around the corner.” A group of researchers was formed and called the Michigan Council for Educator Effectiveness, or MCEE. They studied five frameworks and made several sets of recommendations in their 70-page report, distributed in July of 2013. These recommendations are summarized by Wendy Zdeb-Roper, executive director of the Michigan Association of Secondary School Principals here (2013):      “Districts would have four frameworks to choose from: Danielson's Framework for Teaching (2011), Marzano Teacher Evaluation Model, The Thoughtful Classroom, and the 5 Dimensions of Teaching and Learning. The recommendation includes a waiver process for districts that would like to keep their current framework, though it appears to require significant documentation. One of the four frameworks will be selected as the state tool, based on a bidding process. Should a district choose one of the other three recommended tools, it will be required to fund any additional costs that exceed the expense of the state-funded tool. All observers must be trained in the framework observation protocol that any district selects. That training would have to be delivered by the observation tool vendor. Multiple observations must be conducted across the school year with at least one being unannounced. In addition to being trained on the framework observation protocol, observers must also receive training in coaching and providing feedback. The recommendation is for three rating categories: Professional, Provisional, and Ineffective (rather than Highly Effective, Effective, Minimally Effective, and Ineffective). 186    The state should continue to select and develop assessments aligned with the state-adopted content standards in all core content areas and in "high-volume" non-core content areas where standards exist. The state should provide guidelines for evaluating third-party or locally-developed assessments, as well as training on the development and measurement of student learning objectives. The state should produce value-added modeling scores for educators on state-provided assessments in the core content areas. Schools may also use building value-added measures for not more than five percent of the total evaluation.” Michigan’s legislators have not acted on any of these recommendations, other than to set aside a small amount of money for training administrators on teacher evaluations. They did this a full year after the MCEE produced its recommendations. Legislators promised guidance, tools, and assistance to match their mandate. None of that materialized. Districts are left to ask themselves, Do we follow the MCEE guidelines? Do we wait? Do we make it up ourselves? What and when will the state act? It does not seem like even the legislators know when anything more will occur. A second obstacle that state legislators gave to schools was a limited timeline. The importance of a pilot was clear in the perception interviews. Many teachers and administrators wanted more time to clarify expectations, to describe to teachers how to achieve effectiveness, and to become acclimated to the new evaluation environment. This was especially pertinent if performance pay and lay-offs of teachers hung in the balance. The legislation mandated that the systems be implemented as soon as the district’s contract expired. The financial conditions of many districts around the state were stressed, among them five of the six districts in the study. Most districts had short contract durations, which meant that they had a year or less to implement their new performance appraisal system. According to Grote (1996), it takes an average of three to five years to implement such a system of quality caliber. Thus, one year hardly seems like enough time to do things correctly. A third obstacle was the lack of guidance on implementation. Grote’s work (1996) could have given districts a clear roadmap about how to implement the system. Noe’s attributes (2006) also would have helped districts target the outcomes they wanted. Grote and Noe’s work, from the business field, would allow districts to slow down, look past “getting it done”, and understand how to “get it done well.” Without the roadmap, districts missed key steps that would have allowed them to improve teacher perceptions of the system. For the future, the state law-makers and policy-makers should focus on delivering what they promised. If they are going to allow districts to collaborate and to adapt their own framework, tools, and system to meet the state law, that process should be simplified and expedited. If the state wants to require usage of one of the MCEE-recommended observation frameworks, then the state should provide funding for initial purchase of the copyrighted frameworks and tools. If the state wants to allow districts to use either their own frameworks or the recommended frameworks, the cost would be lessened. 187 Districts should have choices on how they want to implement with clarity, and there should be a strong recommendation that they collaborate with their teachers on the systemic direction taken. The MCEE recommended major changes to the state’s teacher evaluation law. How should student growth be measured? How should student growth be measured for special education teachers, non-core subject areas, counselors, and social workers? What student assessment tools should be used? What is an “effective” or “highly effective” amount of growth? Should the state define some basic parameters that measure student growth and encourage districts to collaborate with specialized groups of employees to choose effective measures that track student growth? The stakes associated with the teacher evaluation law constitute another difficult issue. If compensation or teacher lay-offs are based on the teacher evaluation system, how can a district meet the legal requirements of the increased stakes? What about those increased stakes? One must consider percentages of student growth, relevant training, and significant contributions. Can a district, fully following state law, have its performance assessments and subsequent job actions challenged and lose? What is the minimum standard for districts to meet? Or, instead of the state defining it, should a minimum amount of collaboration with teachers be required for districts to use specifics on the law? With such high stakes, these ambiguous areas must be clarified. The state should provide districts with a roadmap for implementation. Grote’s implementation steps (1996) could serve as a helpful place to start, with Noe’s attributes (2006) as the goal for all districts’ teacher evaluation systems. Validated questions for teacher evaluation surveys, as well as recommended vendors for these surveys could also aid districts in gaining feedback from all teachers on their systems. If there will be a change to a new system or required adaptations to the old system, a step-by-step guide to follow would help districts and thereby teachers collaborate on a system that is understood well, communicated clearly, and improving steadily. Either way, districts have been required to implement the mandates of the state law without the support of the law’s promises. Even at this late point, the state legislators could support and fulfill promises it made within the wording of the current law. Districts would be able to use that fulfillment to meet teacher needs, improve teacher perception, and build teacher motivation, objectives which have been hard-hit over the past three years. Leadership and trust factor heavily into implementation. I analyzed the ways that building administrators showed teachers they were worthy of trust, but I also described intangible factors that could influence a leader’s legitimacy. The manner in which administrators talked to teachers, how they reacted to differences of opinion, and how they led the building 188 all seemed to be related to teacher perceptions. Teachers, in general, seemed to focus more on legitimacy, as principals made evaluation assessments that were linked to compensation and job security. This makes sense. If a principal gives performance rankings that are insignificant in weight and unrelated to pay or lay-offs, background in the classroom, or willingness to support teachers in stressful times, the rater’s history in the building or in the district would matter less. With the stakes raised significantly higher, teachers want to make sure they are getting a fair shake. They base this on what they know about the principal. If they know a lot and have seen the principal be fair and be supportive, it seemed like they would give the principal the benefit of the doubt, even if mistakes were made in the implementation. This benefit of the doubt seems to be based on administrators honestly appraising their lack of confidence in themselves, the state, or the law; careful handling of difficult topics like “highly effective” status and performance pay, district’s communication clarity on expectations, and the district’s positionality (far away is preferable) to the state’s mandates. Building and district can use the findings from this study to build trust and legitimacy among teachers. These avenues to trust and to legitimacy seem as if they would be helpful in implementing any state legislative mandate in schools. For policy-makers, it is impossible to ensure that all schools in a state or in a county have principals viewed as legitimate by their faculties. States could schools with transparent and effective leadership behaviors to guide district and building administrators into building trust and legitimacy, even if it did not exist previously. For districts, building administrators should be allowed the time to have the extensive conversations that are necessary for teachers who may not understand, as well as for building union representatives who have a broader range of communication and can spread the message of clear expectations to all teachers in a building. They must also be held accountable for following a roadmap of collaboration. This accountability should be through teacher surveys, broken down by building and by evaluator, so that districts can see how teachers are perceiving the evaluations and identifying areas in need of improvement. To extend the legitimacy of evaluators, districts should formally have a system in place for annually reviewing either a sampling of teacher evaluations or all of them. Teachers would feel like their evaluators were more legitimate if a central office administrator was reading all teacher evaluations and offering feedback to administrators. A clear communication of the process for review, what will happen if an issue is identified, and a process for teachers to bring up concerns (even if they are happy with their results) would allow a healthy vetting of the evaluations. It would show that administrators are not only 189 responsible for improving test scores in the building or district, but are also accountable for fairly and accurately evaluating teachers. Communication, communication, communication. One of the most interesting findings from the survey is that several districts did a lot to improve inter-rater reliability, consistency, and validity. , District administrators and building administrators did not tell teachers about their efforts, despite the fact that districts provided extensive and expensive training for evaluators. Within buildings and among buildings, administrators met extensively to try to “get on the same page,” to sacrifice their own backgrounds and pre-dispositions to achieve a better district approach to evaluations, and to “do the right thing for teachers.” Without teachers at large knowing, it left teachers to think that the training, the ongoing work, and the goals of the administrators were missing. In Districts 3, 4, and 6, this lack of communication about the efforts that building and district administrators were making seemed to influence teacher perceptions negatively. From the policy-making level, it seems important enough to require a communication plan about teacher evaluations as a part of the school improvement plan. At least half the districts in the study could have achieved more positive perceptions if they had more thoroughly approached this crucial step, communication. If districts took the time to write even a short plan, they could follow it, allowing teachers and other certified faculty and staff to understand the discussions, the decisions, and the expectations of the teacher evaluation framework and system. From a district level, writing and using a plan about teacher evaluations would increase the conversation and collaboration with teachers, but it would also allow district and building administrators to reap the benefits from work they are already doing. From a building level, weekly updates should consistently include information about the building administrator’s and union’s conversations, collaboration, and communication about the evaluations. This communication could identify, clarify, and correct unclear areas and improve focus on teacher evaluations throughout the year. Take time to focus and implement. When a mandate, especially with perceived or real threats, comes to a school district, the push can be to “get it done” and meet whatever minimums the state has set in place for the initiative. Occasionally, there are real teeth to those threats, and the minimums must be met; however, districts can often slow down and implement mandates over longer periods. In the case of teacher evaluation laws, this would have been possible, but districts did not take the time to see the flexibility and time available in the law. Taking time, from a building and district perspective, would have allowed communication, pilots, assessment, and improvement. 190 Following Grote’s (1996) implementation steps would also have forced districts to focus on just the teacher evaluation for the initial implementation period. In Districts 5 and 6, other initiatives and directions by the district made the implementation of the teacher evaluation system unclear, a result specifically described by teachers. The number of initiatives going on in a district can stretch administrators thin, make expectations for all of them less clear, and render their assessment and adaptation nearly impossible. Districts and building administrators whose teachers had more positive perceptions of the implementation seemed to take the time to implement well and to focus on that initiative, at least in the first year of operation. Districts should use the main components of Grote’s framework, regardless of state action. All districts missed portions of Grote’s framework. There is no going back on their missed opportunity to conduct a pilot or assessment of needs. Still, districts are able to use assessment and ongoing improvement, reciprocal accountability, collaboration, and communication. If the State of Michigan stands pat or even adopts new adjustments to the law, districts should be sure to use Grote’s key actions. The ongoing assessment and upgrade of the system can allow teachers to have a voice and input, which can then be aligned to the district’s needs and direction, shaping an evaluation system that recognizes effective teacher behavior and assesses it in a clear and efficient manner. Districts must have a system of accountability for building and district administrators, so it is clear that all evaluations are being scrutinized, and all evaluators are held responsible for their appraisals of teacher performance. Communication, as mentioned above, is the foundation of building trust and legitimacy of building leadership. Ongoing collaboration could take the form of regular meetings of the teacher evaluation team. Some districts have “professional councils” that review district and building professional development, provide feedback on calendar, on teacher handbooks, and on school improvement. These committees could take on the work of consistently improving the teacher evaluation system. Such continuous collaboration could be a part of the district and building school improvement plan. A formal plan for buildings and districts to have ongoing conversation and action around teacher evaluations and other professional topics would alleviate some of the mistakes of the implementation and improve teacher perceptions of the system. In conclusion, state policy-makers, district administrators, building administrators, and teachers can learn much about how to move forward from what may have been, in many cases, a very flawed implementation. Policy-makers at the state level need to consider how best to equip districts with financial support, clarifications, and concrete plans for implementation and communication. In this way, the state’s policy can better match the lofty ideals of the original law. 191 District leaders now know better and can learn from their mistakes. They can communicate, assess, improve, establish reciprocal accountability, and collaborate. They can slow down, focus, and have the goal be “getting things done well,” rather than just “getting things done.” Building leaders can also ascertain where they went wrong. They can be honest with their faculties, build the sense that “we’re all in this together,” communicate, highlight how they are accountable, create several degrees of distance from the state, clarify how they define “highly effective,” and focus on improving evaluations until the system is acceptable to all users. It would be in the best interests of all teachers if district and building administrators took all helpful steps to ensure that the high-stakes teacher evaluation system is understood well and followed carefully. In districts around the State of Michigan, educators’ careers are riding on teacher evaluation systems. I found that how districts responded to the state’s mandate, influenced how teachers felt about the evaluation system and by extension their jobs. As summarized above, this study has identified several practical steps for state policy makers, on the one hand, and district and school administrators, on the other, that would improve the state’s teacher evaluation system and increase the prospects that the beneficial outcomes envisioned by state policy makers when they passed the evaluation laws are actually realized. Making the system better, however, will clearly require a careful and disciplined approach from all stakeholders. 192 APPENDICES 193 APPENDIX A Admin/Union Electronic Survey Questions 194 Appendix A: Admin/Union Electronic Survey Questions Welcome. Thank you for agreeing to take part in the research project “District Response to Michigan’s Teacher Evaluation Law.” The independent variable in the proposed study is the extent of congruence between school district practices and Grote’s framework of evaluation system implementation. The dependent variable is teacher and administrator perceptions of the evaluation system. The goals of the study are to describe each district’s implementation of their teacher evaluation system, to understand teacher’s perceptions of the implementation of the system, and to relate the implementation to teacher perception of the district’s evaluation system. Your answers on the questionnaire will reflect your understanding of your district’s implementation of Michigan’s Teacher Evaluation Laws. Your identity and the answers you provide will be kept confidential. Nowhere on the questionnaire will you include your name. You will be assigned a number kept confidential by the researchers. All materials will be kept locked and secured in Kevin Mowrer’s home office and computer for three years and will be destroyed immediately after the three years. Information about you will be kept confidential to the maximum extent allowable by law. All information is subject to being a part of the final product, which involves written or oral presentations and manuscripts. You will not directly benefit from your participation in this study. However, your participation in this study may contribute to the understanding of future teacher evaluation systems. The risks to you are minimal. I do not anticipate any risks to you participating in this study other than those you encounter in day-to-day life. Participation in this research project is completely voluntary. You have the right to say no. You may change your mind at any time and withdraw. There are no consequences of withdrawal or incomplete participation. You may choose not to answer specific questions or to stop participating at any time. You will be told of any significant findings that develop during the course of the study that may influence your willingness to continue to participate in the research. The researchers declare there are no conflicts of interest on their part in this research project as there is no material gain involved and no external sponsors of this research other than Michigan State University. The data being collected and analyzed are for educational purposes only. The researcher conducting this study is Kevin Mowrer. The advisor of the project is Dr. David Arsen, professor at Michigan State University. Please ask any questions you have now. If you have questions now or later, you may contact Kevin Mowrer at kmowrer@gmail.com or at 1-734-385-6022. If you have any questions or concerns regarding your rights as a subject in this study, you may contact the Michigan State University’s Institutional Review Board (IRB) at (517) 355-2180, access their website at http://www.humanresearch.msu.edu/, or email them at irb@msu.edu. Statement of Consent: I have read the above information, and have received answers to any questions I asked. By clicking agree, we assume that you understand and agree to the conditions and information included in this consent presentation summary. We look forward to hearing from you. The online consent form will be kept by the researcher for at least three years beyond the end of the study and was approved by the IRB on 5/11/2013. The title of the study will appear at the top of every page of the survey. ACCEPT DECLINE 1. What primary position do you hold in the district? A. B. C. Teacher/Union Representative (if this, then please indicate subject area that your chief assignment) Building administrator District administrator 195 2. How many years of experience do you have working as a teacher or administrator in the public schools? A. B. C. D. E. F. 0-5 years 6-10 years 11-15 years 16-20 years 21-30 years 31 years or more 3. A. B. What is your gender? Female Male 4. Who selected the teacher evaluation tool and system or for your district? A. B. C. A single person A team I do not know 5. If answer A for #4, who was that person? 6. If answer B for #4, who was on the team (check all that apply)? A. B. C. D. E. F. School board members District administrator/s Building administrator/s Teachers Teacher union representatives Other (open for participant completion) 7. If answer B for #4, in what manner did that team select a teacher evaluation tool/system? A. B. C. D. team E. F. Cooperatively, with stakeholders on team Team discussed, but final decision was made by administration Selected with a tool from external consultant, with input from team Selected with a tool from external consultant, with input from selected members of Dictated to district by Michigan Department of Education Other (open for participant completion) 8. How were superintendents and assistant superintendents involved in the process of choosing and implementing the new teacher evaluation system? A. They led the process B. They were informed of the process regularly in a private setting C. They were informed of the process regularly in a public setting (board meeting, committee meeting) D. They were not involved E. Not sure how involved they were involved 196 9. How much are superintendents and assistant superintendents involved in the process of monitoring the acceptability to employees of the new evaluation system? A. B. C. Very much involved Moderately involved Not involved 10. In selecting the districts’ current teacher evaluation system, did the developer(s) utilize a survey or focus group to gather input from teachers on desired features of the evaluation system? A. B. C. Yes No Not sure If you answered “Yes” to #10, who was contacted to provide input regarding the new teacher evaluation system OR changes to the teacher evaluation system? (check all that apply) A. B. C. D. E. F. G. School board members District administrator/s Building administrator/s Teachers Community Members/Parents Teacher union representatives Other (open for participant completion) 11. Was the new teacher evaluation tools and procedure piloted with a subset of district employees before its districtwide implementation? (If you answer No, skip to question 15) A. B. C. Yes No Not sure 12. What subset of district employees were involved in the pilot? A. B. C. Building (write building name) Department (write department name) Other 13. How, if at all, did the district use the results of the pilot (check all that apply)? A. B. C. D. Changed language/descriptions in evaluation tool Provided more training for teachers regarding how they were expected to teach Provided more training for administrators/evaluators on performing teacher evaluations Other (Explain) 14. How did the district communicate the new evaluation procedures to district employees prior to its implementation (check all that apply) A. B. C. D. E. Administrators explained in faculty meetings Emails to all certified faculty and staff Building and/or district leaders explained in union meetings Central office administrators explained in district wide meetings Other (Explain) 197 15. In your opinion, how well did the district communicate the new evaluation procedures to district employees prior to its implementation? A. B. C. D. E. Very well Well Not well Not at all I do not know 16. Prior to implementation, how did the district train teachers on how they should perform their jobs in order to rate highly on the new teacher evaluation system? (Check all that apply) A. B. C. D. E. Through emails describing the way teachers were expected to perform Through emails with evaluation procedures and tools attached Through mandatory trainings before the year started Through optional trainings before the year started Other (Explain) 17. Prior to implementation, how did the district train administrators/evaluators to perform the new teacher evaluations? (Check all that apply) A. B. C. D. E. Through training by outside vendors Through training by district employees Through providing administrators with written documents that described the new procedures Through answering questions administrators had about the procedures Other (Explain) 18. Prior to implementation, how well did you understand how teachers were expected to perform their duties under the new teacher evaluation system? A. B. C. D. E. Very well Well Not well Not at all I do not know 19. Does your district’s new teacher evaluation system include features to assure that administrators are accountable for performing their evaluations well? A. B. C. Yes No Not sure 198 20. Did the district solicit feedback from teachers on the new evaluation system? If so, what form did it take? A. B. C. D. Yes, with a survey Yes, with a focus group No Not sure 21. If you answered “yes” to question #21, how was this feedback used to modify the evaluation procedures, if at all? A. B. C. D. Changed language/descriptions in tool Provided more training for teachers on how they were supposed to teach Provided more training for administrators/evaluators on performing evaluations Other (Explain) 22. In the building to which you are most closely connected, how has the new teacher evaluation system/ influenced teacher morale? A. B. C. D. E. Greatly improved morale Improved morale Neither improved or diminished morale Diminished morale Greatly diminished morale 23. In the district as a whole, how has the new teacher evaluation system influenced teacher morale? A. B. C. D. E. Greatly improved morale Improved morale Neither improved or diminished morale Diminished morale Greatly diminished morale 24. A. B. C. D. E. Based on your experience overall, how acceptable has the district’s new teacher evaluation system been to you? Highly acceptable Acceptable Neutral Unacceptable Highly unacceptable 199 APPENDIX B Non-Union Representative Electronic Teacher Surveys 200 Appendix B: Non-Union Representative Electronic Teacher Surveys Welcome. Thank you for agreeing to take part in the research project “District Response to Michigan’s Teacher Evaluation Law.” The independent variable in the proposed study is the extent of congruence between school district practices and Grote’s framework of evaluation system implementation. The dependent variable is teacher and administrator perceptions of the evaluation system. The goals of the study are to describe each district’s implementation of their teacher evaluation system, to understand teacher’s perceptions of the implementation of the system, and to relate the implementation to teacher perception of the district’s evaluation system. Your answers on the questionnaire will reflect your understanding of your district’s implementation of Michigan’s Teacher Evaluation Laws. Your identity and the answers you provide will be kept confidential. Nowhere on the questionnaire will you include your name. You will be assigned a number kept confidential by the researchers. All materials will be kept locked and secured in Kevin Mowrer’s home office and computer for three years and will be destroyed immediately after the three years. Information about you will be kept confidential to the maximum extent allowable by law. All information is subject to being a part of the final product, which involves written or oral presentations and manuscripts. You will not directly benefit from your participation in this study. However, your participation in this study may contribute to the understanding of future teacher evaluation systems. The risks to you are minimal. I do not anticipate any risks to you participating in this study other than those you encounter in day-to-day life. Participation in this research project is completely voluntary. You have the right to say no. You may change your mind at any time and withdraw. There are no consequences of withdrawal or incomplete participation. You may choose not to answer specific questions or to stop participating at any time. You will be told of any significant findings that develop during the course of the study that may influence your willingness to continue to participate in the research. The researchers declare there are no conflicts of interest on their part in this research project as there is no material gain involved and no external sponsors of this research other than Michigan State University. The data being collected and analyzed are for educational purposes only. The researcher conducting this study is Kevin Mowrer. The advisor of the project is Dr. David Arsen, professor at Michigan State University. Please ask any questions you have now. If you have questions now or later, you may contact Kevin Mowrer at kmowrer@gmail.com or at 1-734-385-6022. If you have any questions or concerns regarding your rights as a subject in this study, you may contact the Michigan State University’s Institutional Review Board (IRB) at (517) 355-2180, access their website at http://www.humanresearch.msu.edu/, or email them at irb@msu.edu. Statement of Consent: I have read the above information, and have received answers to any questions I asked. By clicking agree, we assume that you understand and agree to the conditions and information included in this consent presentation summary. We look forward to hearing from you. The online consent form will be kept by the researcher for at least three years beyond the end of the study and was approved by the IRB on 5/11/2013. The title of the study will appear at the top of every page of the survey. ACCEPT DECLINE The following questions will allow you to describe your perspective of the district’s implementation process. Your district implemented its implementation system during the (INSERT YEAR HERE) school year. 1. A. Which term best describes the majority of your teaching assignment? Core 201 B. Non-core 2. A. B. C. D. E. F. How many years of experience do you have working in public schools? 0-5 years 6-10 years 11-15 years 16-20 years 21-30 years 31 years or more 3. What is your gender? A. B. Female Male 202 4. The following activities are components of a performance appraisal system. Please rate how well you remember these activities occurring in your district as the new teacher evaluation system was implemented during the (INSERT SCHOOL YEAR HERE). Check the box next to the activity described. Implementation Activity Remember Very Well Remember Well Remember Do Not Remember Very Well Do Not Remember Very Well Do Not Remember Very Well Do Not Remember At All Do Not Remember At All Do Not Remember At All Formation of Implementation Team Superintendent and Central Office Administrators Engaged in Process Survey/focus groups to identify needs for new evaluation system Adaptation of teacher evaluation tool and process The adapted teacher evaluation system was piloted with a department or building Communication about adapted evaluation system leading up to initial usage Training for teachers, principals, and administrators on understanding and usage of teacher evaluation system prior to initial usage Principals and directors evaluated on their performance in evaluating employees District solicited feedback about the adapted evaluation system during and after initial usage Adaptation and/or adjustment of teacher evaluation system after initial usage Remember Very Well Remember Well Remember Remember Very Well Remember Well Remember Remember Very Well Remember Well Remember Do Not Remember Very Well Do Not Remember At All Remember Very Well Remember Well Remember Remember Very Well Remember Well Remember Do Not Remember Very Well Do Not Remember Very Well Do Not Remember At All Do Not Remember At All Remember Very Well Remember Well Remember Do Not Remember Very Well Do Not Remember At All Remember Very Well Remember Well Remember Do Not Remember Very Well Do Not Remember At All Remember Very Well Remember Well Remember Do Not Remember Very Well Do Not Remember At All Remember Very Well Remember Well Remember Do Not Remember Very Well Do Not Remember At All Remember Very Well Remember Well Remember Do Not Remember Very Well Do Not Remember At All 203 5. In the building to which you are most closely connected, how has the new teacher evaluation system influenced teacher morale? 6. In the district as a whole, how has the new teacher evaluation system influenced teacher morale? 7. Based on your experience overall, how acceptable have you found the district’s teacher evaluation system? A. Highly acceptable B. Acceptable C. Neutral D. Unacceptable E. Highly unacceptable 204 APPENDIX C Administrator Interview Questions 205 Appendix C: Administrator Interview Questions Introduction: Welcome. Thank you for your time to share with me your perceptions of your district’s teacher evaluation system during the ________ school year. The goals of the study is to describe each district’s implementation of their teacher evaluation system, to understand teacher’s perceptions of the implementation of the system, and to relate the implementation to teacher perception of the district’s evaluation system. I am using frameworks from the field of human resources and business to analyze these descriptions. This interview will provide validation and detail of the district’s implementation effort. By “implementation effort”, I mean all steps from choosing or developing a new teacher evaluation system to making adaptions to the evaluation system to improve it. The survey will provide the opportunity for you to describe how acceptable teachers in your building find the results of the teacher evaluations produced by the system. For the description of the interview process, please think back to the time during the (INSERT SCHOOL YEAR HERE) the system was implemented. When asked about perceptions, please use your current thoughts on the system as your basis for answering. Do you have any questions about the goals of the study or of the interview? We will now begin with the questions. 1. How many years of experience teaching do you have in a traditional public school setting? 2. How many years of experience do you have in the administration in a traditional public school setting? 3. Describe your district’s implementation of the new teacher evaluation system during the (INSERT SCHOOL YEAR HERE)) from beginning to end. Please include all parts of the process, including who was involved in the process, communication, training, and changes to the process. 4. What did you believe to be the most important part of the implementation process (for the evaluation system implemented in the (INSERT SCHOOL YEAR HERE)) school year)? Why? 5. What was not included in the implementation of the system that should have been? 6. How did the implementation of the new teacher evaluation system influence teacher perception of the system in your building? 7. After receiving an evaluation using the new evaluation system(implemented in the (INSERT SCHOOL YEAR HERE) school year), do teachers in your building accept the results as fair? Why or why not? 8. Did the implementation of the system influence this acceptability? Why or why not? 206 APPENDIX D Teacher Interview Questions 207 Appendix D: Teacher Interview Questions Introduction: Welcome. Thank you for your time to share with me your perceptions of your district’s teacher evaluation system during the ________ school year. The goals of the study are to describe each district’s implementation of their teacher evaluation system, to understand teacher’s perceptions of the implementation of the system, and to relate the implementation to teacher perception of the district’s evaluation system. I am using frameworks from the field of human resources and business to analyze these descriptions. This interview will provide validation and detail of the district’s implementation effort. By “implementation effort”, I mean all steps from choosing or developing a new teacher evaluation system to making adaptions to the evaluation system to improve it. The survey will provide the opportunity for you to describe your perceptions of the implementation and of the district’s teacher evaluation system. For the description of the interview process, please think back to the time during the (INSERT SCHOOL YEAR HERE) the system was implemented. When asked about perceptions, please use your current thoughts on the system as your basis for answering. Do you have any questions about the goals of the study or of the interview? We will now begin with the questions. 1. How many years of teaching experience do you have in education? In the district? 2. What subject area is your chief assignment? 3. Describe your district’s implementation of the current teacher evaluation system (implemented in the (INSERT SCHOOL YEAR HERE) year) from beginning to end. Please be sure to include all parts of the process, which could include who was involved in the process, communication, training, and changes to the process. 4. What did you believe to be the most important part of the implementation process? Why? 5. What was not included in the implementation of the system that should have been? 6. How did the implementation of the new teacher evaluation system influence teacher perceptions of the new system (implemented in the (INSERT SCHOOL YEAR HERE) school year)? 7. One way to measure an evaluation system is how reliable it is between different evaluators and between different work contexts within the system. If a teacher performs his/her job the same way in two different buildings or two different administrators, he or she, if the system is reliable, the teacher should receive a similar rating and feedback. Do you feel that the current evaluation system (implemented in the (INSERT SCHOOL YEAR HERE) school year) is consistent among various administrators within your building? Explain. Did the implementation of the system influence this perception? Why or why not? 8. Another way to assess performance evaluation systems is to measure if they encompass all of the relevant, and only the relevant, aspects of performance. Do teachers in your building feel that the current evaluation system (implemented in the (INSERT SCHOOL YEAR HERE) school year) is valid in assessing all the relevant and only the relevant aspects of performance? Why or why not? Did the implementation of the system influence this perception? Why or why not? 208 i9. Specificity in performance appraisal systems assesses the extent to which a performance measure tells employees what is expected of them and how they can meet those expectations. Do teachers in your building feel that the current evaluation system (implemented in the (INSERT SCHOOL YEAR HERE) school year) clearly communicates what is expected of them and clearly express how to meet those expectations? Why or why not? Did the implementation of the system influence this perception? Why or why not? 10. Do teachers in your building feel that the current evaluation system (implemented in the (INSERT SCHOOL YEAR HERE) school year) motivates them to perform their job in a way that is aligned with the district’s goals and objectives? Why or why not? Did the implementation of the system influence this perception? Why or why not? 11. Acceptability in performance evaluation systems gauges the extent to which the people who use the performance measure accept it. After receiving an evaluation using the new evaluation system(implemented in the (INSERT SCHOOL YEAR HERE) school year), do teachers in your building accept the results as fair? Why or why not? Did the implementation of the system influence this perception? Why or why not? 209 APPENDIX E Methodology- Notes from the field 210 Appendix E: Notes from the field The topic of teacher evaluation system implementation is very personal to teachers. It is about the way school districts evaluate those teachers’ performance. Teachers’ livelihoods are based on the evaluations of their performance by their building principals. Therefore, the proposed methods morphed in several ways from the proposed study. The dilemmas evident in the research are described in the following sections. Fear of Repercussions Several teachers, once the microphone was turned off and the interviews concluded, expressed concern with their responses to the interview questions. Evaluations by principals seemed personal enough to both the teacher and the administrator that teachers were worried that their honest responses would get back to their principals. They worried that their own job status could be influenced by their participation. Therefore, it is possible that teachers may have been more positive, especially in interviews, than they might have been if there had been more participants involved in the study. Participants were assured that their identities would be kept confidential, but they still worried about their futures, particularly if they were less experienced and had a lot of time left to work. These fears should not be underestimated and should be considered when viewing the results of the interview. The Seeming Power of the Anonymous Survey in the Study When the research design was first developed, interviews were utilized because of the personal and intimate details of a district and a building’s implementation could be best describe face to face. However, when analyzing the anonymous surveys teachers took, it seemed like their responses to the larger scale, anonymous surveys were more detailed and more critical than many of the interviews. The survey system seemed to depersonalize the responses, leaving more room for teachers to “tell it like it is” to open-ended questions about morale and acceptability. How Well Teachers Did and How That Influenced Their Perceptions. As teachers described the implementation of the new teacher evaluation system, it was hard to separate their perceptions from the designations their principals gave them. If teachers received a “highly effective” ranking from their principals, it seemed as though they were more positive about the implementation and the system. If, in the end, the implementation and system produced results the teacher found acceptable, then their perceptions of that system and implementation seemed to be mor positive. 211 Drawing the Sample Because of the random selection of participation for teachers, there were teachers who opted not to take part in the study. If teachers agreed to take the anonymous survey, they understood there was a chance of them being selected for an interview. Because of the time involved and the personal nature of the subject, many possible subjects declined participation, when randomly selected. Table 44: Number of Teachers Who Declined Participation By District District Number 1 2 3 4 5 6 Number of Teachers Who Declined Participation in the Study 1 0 1 6 2 2 All of the administrative and union representatives agreed to take part in the study immediately. It is hard to say what those who declined to take part in the study would say. Some participants did not reply at all. After three emails and phone calls, I moved on to the next highest randomly selected number. As I moved forward with the research, those who did not participate recurred in my thoughts, as I wondered what they might have said. The Small Sample The small sample of 18 initial survey participants, 18 interview participants, and 18 teacher participants limit the generalizability of the study. The amount of time necessary to obtain consent from the district, obtain consent from participants, facilitate completion of surveys and interviews, and then analyze the data made that number of participants seem large to the researcher. Educators are busy throughout the year and convincing them to take part in survey and then possibly take part in extensive interviews was considerable. The personal nature of the responses and the need for time to record and to analyze these findings made the small sample seem necessary. My hope is that future researchers can use the framework that I selected to further research implementation of teacher evaluation systems. 212 Establishing Rapport I chose the high school setting because that is the context where I work. After I interviewed teachers, many of them asked about the teacher evaluation system we implemented in my district. I kept my responses brief, accepting the difficulty each district and each teacher found themselves. The hope that my research would bring about positive change in district action seemed to motivate teachers to give me feedback on the way their district implemented the system. The common language and perceptions of high school educators did allow me to establish a level of rapport with the participants. Place of Interview: The School In my initial research design, I hoped to meet teachers at a neutral place, where teachers would not worry about others overhearing their responses. The truth was that very few, if any, of the participants knew me. Meeting me at a public coffee house or library was a non-starter. It seemed like a lot to ask teachers to talk to me about the evaluation system. To get them to arrange a time to meet me outside the school day was a non-starter. I interviewed most teachers during the school day, on their preparatory period. Because the districts gave me consent to talk to their teachers, I felt comfortable interviewing them in the school day. Most teachers wanted to close the door or have me interview them in a place like the office or the media center in their schools. I let teachers take the lead in setting a place they felt comfortable within their school. In terms of their responses, the place may have inhibited their honesty. If their building or district administrator had walked into the interview location, the teacher may have been put in an awkward and potentially dangerous situation. Because teachers would not meet me outside the school, I had to settle for interviewing within the school, but I felt that teachers might have been less honest than they could have been because of the place of the interview. Respondents Not Understanding The Questions One survey respondent did not understand the questions about building and district morale. S/he inputted responses that did not correspond with the questions. I vacillated back and forth about re-contacting the respondent to get a more accurate response to the question. After encouragement from my dissertation committee, I re-contacted the respondent and sent her the survey portions that were not accurately completed. S/he completed the survey, answering the questions. Her answers were given five months after his/her initial responses. The slight difference in time may have influenced her responses. Because of the small number of responses, obtaining a more accurate response for him/her seemed important. When Teachers Asked Questions 213 When teachers asked questions about what I knew about the teacher evaluation system, I tried to answer as briefly as possible. When teachers, non-verbally, indicated they were interested in my opinion of what the district did, I nodded and agreed with their perspective, regardless of what I truly felt about their perspective or their way of describing district action. When asked, I wanted to keep teachers talking about their perspective, I agreed with them. When I thought my perspective would stunt the flow of the interview, I deferred until after the interview was over to answer questions. In some cases, the interviewee really wanted to have the need to know more or to share experience. In those cases, I answered questions and tried to relate to the interviewees as best I could. 214 BIBLIOGRAPHY 215 BIBLIOGRAPHY Ackley, M. (2010, August 1). Michigan Merit Curriculum High School Graduation Requirements. Michigan Merit Curriculum High School Graduation Requirements. Retrieved December 7, 2012, from www.michigan.gov/documents/mde/FAQ_-_Entire_Document_12.07_217841_7.pdf Ball, D. (2013, July 24). Michigan Council for Educator Effectiveness. Building an Improvement-Focused System of Educator Evaluation in Michigan: Final Recommendations. Retrieved July 3, 2014, from http://www.mcede.org/reports Brocka, B., & Brocka, M. S. (1992). Quality management: implementing the best ideas of the masters. Homewood, Ill.: Business One Irwin. Cortazzi, M. (1993). Narrative analysis. London: Falmer Press. Danielson, C. (2007). Enhancing Professional Practice a Framework for Teaching (2nd ed.). Alexandria, VA: Association for Supervision and Curriculum Development, 2007. ERS (2000). The principal, keystone of a high-achieving school: Attracting and keeping the leaders we need. Educational Research Service, 1, 1-100. Conley, S., & Glasman, N. S. (2008). Fear, the School Organization, and Teacher Evaluation. Educational Policy, 22(1), 63-85 Feistritzer, C. E. (2011, July 29). PROFILE OF TEACHERS IN THE U.S.2011. PROFILE OF TEACHERS IN THE U.S.2011. Retrieved November 7, 2012, from www.edweek.org/media/pot2011final-blog.pdf Gibbs, R. (2009, November 4). Fact Sheet: The Race to the Top | The White House. The White House. Retrieved November 5, 2012, from http://www.whitehouse.gov/the-press-office/fact-sheet-race-top Gilliland, S., & Langdon, J. (1998). Performance appraisal: state of the art in practice. San Francisco: Jossey-Bass Publishers. Grote, D. (1996). The Complete Guide to Performance Appraisal. New York, NY: American Management Association. Ilgen, D., & Hollenbeck, J. (1991). The Structure of Work: Job Design and Roles. Handbook of Industrial and Organizational Pscyhology (pp. 165-208). Palo Alto, CA: Consulting Psychologists Press. Kennedy, M. M., & Cohen, D. (2010). Teacher assessment and the quest for teacher quality: a handbook. San Francisco: JosseyBass. Koch, K. (1999). National Standards: Will they restrict local schools' flexibility?. CQ Researcher, 9(18), 401-429. Lane, A. (2010, January 10). Gov. Jennifer Granholm signs "Race to the Top" bills to reform Michigan schools | Crain's Detroit Business. Crain's Detroit Business. Retrieved November 5, 2012, from http://www.crainsdetroit.com/article/20100104/FREE/100109990# Medley, D. M., Coker, H., & Soar, R. S. (1984). Measurement-based evaluation of teacher performance: an empirical approach. New York: Longman. Mohrman, A. M., Resnick-West, S. M., & Lawler III, E. E. (1989). Designig Performance Appraisal Systems. San Francisco, CA: Jersey-Boss Inc. Nathan, B., Mohrman, A., & Millman, J. (1991). Interpersonal Relations as a Context for the Effects of Appraisal Interviews on Performance and Satisfaction: A Longitudinal Study. Academy of Management Journal, 34 , 352-369. Nimtz, J. (2009). A Thesis Presented to the Faculty of California State University, Chico . Chico, CA: California State University. Noe, R. A. (2006). Human resource management: gaining a competitive advantage (5th ed.). Boston, MA: McGraw-Hill. 216 Nye, B., Konstantopoulos, S., & Hedges, L. (2004). How Large are Teacher Effects?. Educational Evaluation and Policy Analysis, 26(3), 237-257. Primoff, E. S. (1975). How to prepare and conduct job element examinations. Washington, DC: U.S. Civil Service Commission, Personnel Research and Development Center. Riessman, C. K. (1993). Narrative analysis. Newbury Park, CA: Sage Publications. Schuler, R., & Jackson, S. (1987). Linking Competitive Strategies with Human Resource Practices. Academy of Management Executive, 1, 207-219. Wright, P. W., Wright, P. D., & Heath, S. W. (narr2004). No child left behind. Hatfield, VA.: Harbor House Law Press. Zdeb-Roper, W. (2013, July 24). MCEE Releases Final Recommendations. Mymassp.com. Retrieved July 5, 2014 from http://mymassp.com/content/mcee_releases_final_recommendations. 217