Challenging the current discordance in orientation between student assessment models and teacher/school value-added models, this study aims to present the educator performance function (EPERF)-based teacher effect estimation method which utilizes the nature of student criterion-referenced assessment, to evaluate its feasibility and usefulness by comparison with the currently prevailing methods - the education production function (EPROF)-based value-added model. Specifically, this study (1) investigated how different the teacher effect estimates of the EPERF-based method are from those of the EPROF-based method, (2) examined whether the model fit of the EPERF is acceptable, and (3) simulated whether the EPERF-based method is robust to the locations of cut-scores and number of performance levels. A northern state's student-teacher linked data set was used, and the student challenge index, which is defined as the degree of difficulty that teachers face in teaching a student to attain a desired/higher performance standard, was constructed as a summary quantity of individual students' characteristics. The main findings from comparison between the two different teacher effect estimates - the educator performance level (EPL) from the EPERF-based method, and the value-added measure (VAM) from the EPROF-based method - were as follows: First, rank correlations between the two estimates were above .82 for mathematics. In particular, the EPL from the polytomous EPERF were very close to the VAM estimates in terms of ranking teachers, showing above .8 rank correlations. Second, in consistent and considerable ways, the relationship of the teacher effect estimates to student and teacher characteristics did not differ between the EPL and VAM estimates. Third, intra-teacher rank correlations across different subjects and different grade levels were also similar between the EPL and VAM. These observations implied that the teacher ranking information resulting from the EPERF-based methods did not differ noticeably from the results of the EPROF-based method. The EPERF-based methods, however, produced several useful areas of information for understanding how average or individual teachers perform with their students. For the second question, the EPERF showed a reasonable model fit in mathematics but not in reading. The conditional independence assumption of student success was violated. The amount of conditional dependency within each teacher was reasonable, and tended to be larger than in the EPROF-based models. Regarding the third question, it was found that, as a result of real-data simulations, the EPL based on the polytomous performance levels was quite robust to the location of cut-scores, and the number of performance levels also did not substantially change the teachers' ranking. These mixed results of model-fit and the robustness of the estimates bring into question on whether the EPL estimates change when student challenge index indicators are added, or when more generalized EPERF models are applied. This study appraised a part of the validity evidence of using the EPERF-based method, including if the method is executable and if the estimated teacher effects are trustworthy, along with the comparison with the EPROF-based method. Implications of applying the EPERF-based teacher effect estimation and future directions for expanding the method are discussed.
Read
