Second language (L2) oral performance assessment always involves raters' subjective judgments and is thus subject to rater variability. The variability due to rater characteristics has important consequential impacts on decision-making processes, particularly in high-stakes testing situations (Bachman, Lynch, & Mason, 1995; A. Brown, 1995; Engelhard & Myford, 2003; Lumley & McNamara, 1995; McNamara, 1996). The purposes of this dissertation study were twofold. First, I wanted to examine rater severity effects across two groups of raters, English-as-a-Second-Language (ESL) teachers and American undergraduate students, when raters evaluated international teaching assistants' (ITAs) oral proficiency, accentedness, and comprehensibility. Second, I wanted to identify and compare rater orientations, that is, factors that drew raters' attention when judging the examinees' oral performances. I employed both quantitative and qualitative methodologies to address these issues concerning rater effects and rater orientations in the performance testing of ITAs at a large Midwestern university. Thirteen ESL teachers and 32 American undergraduate students participated in this study. They evaluated 28 potential ITAs' oral responses to the Speaking Proficiency English Assessment Kit (SPEAK). Raters evaluated the examinees' oral proficiency, accentedness, and comprehensibility, using three separate holistic rating scales. Raters also provided concurrent written comments regarding their rating criteria and participated in one-on-one interviews that explored raters' rating orientations. I employed a many-facet Rasch measurement analysis to examine and compare rater severity across rater groups using the computer program FACETS. I compared the written comments across groups to identify major rating criteria employed by the ESL teachers and the undergraduates. I analyzed the interview data to explore the reasons for rating discrepancies across groups. Results of the study suggested that the ESL teachers and the undergraduate raters did not differ in severity with respect to their ratings of oral proficiency. However, the comparisons of ratings in accentedness and comprehensibility were both statistically significant. The undergraduate raters were harsher than the teacher raters in their evaluations of examinees' accentedness and comprehensibility. Additionally, the analysis of the written comments identified six major rating criteria: linguistic resources, phonology, fluency, content, global assessment, and nonlinguistic factors. Cross-group comparisons of the rating criteria indicated that the undergraduate raters tended to evaluate the examinees' oral performances more globally than the ESL teachers did. In contrast, the ESL teachers tended to use a wider variety of rating criteria and commented more frequently on specific linguistic features. The interview protocols revealed that raters' experience with accented speech, perceptions of accent as an important rating criterion, and approaches to rating (i.e. analytical or global), had important bearings on raters' judgments of ITA speech.
Read
