"This study uses data from the Measures of Effective Teaching (MET) project to examine the differences in teachers' observational measures across content under various contexts: 1) the principled choice of the instrument, 2) the score aggregation methods to generate the final ratings, and 3) the performance frameworks under which teachers are categorized. Specifically, this study examines whether the same teachers' observational measures in distinctive subjects (ELA vs. mathematics) as well as subject areas within mathematics (e.g., Algebra vs. Geometry) are different, and hence influencing their evaluation results non-trivially. For the generalist teachers, this study finds that there are statistical differences as well as practical differences in the same teachers' observational measures between ELA and mathematics. Such differences are present for both the generic instruments and the subject-specific instruments, and do not depend on the grade level. For the mathematics teachers, this study compares the consistency of their observational measures between the two generic instruments, and between the generic instruments and a math-specific one. The results show that the two generic instruments have much higher consistency with each other than with the subject-specific one respectively. Moreover, almost none of the differences between the same mathematics teachers' observational measures across unlike subject areas are statistically significant. Under the relative performance framework, however, analyses using the rank scores demonstrate a large volatility between teachers' two observational measures across areas of mathematics. In conclusion, there is a lack of consistency between the same teachers' observational measures when they are observed in diverse content, especially different subjects, across all three contexts: instrument choice, score aggregation methods, and performance framework for categorization. The consistency, or the lack thereof, when teachers are observed in different areas of mathematics, however, have some associations with the performance framework to put teachers into quality categories. These findings all together suggest the need to take into consideration the content of the lessons the evaluator chooses to observe, and provide some empirical evidence on the implementation of the observation component in current teacher evaluation systems."--Pages ii-iii.
Read
