My dissertation research examines constructs of writing proficiencies in state and national assessments through content analysis of writing prompts and rubrics; predicts students' writing performance on the National Assessment of Educational Progress (NAEP) from assessment variations using multi-level modeling; and explores genre demands in state writing assessments through syntactic analysis of writing prompts to identify the ambiguity and implicit expectations and content analysis of rubrics and state standards to identify the genres specified.Through content analysis of 78 prompts and 35 rubrics from 27 states' writing assessments, and three representative prompts and rubrics from the NAEP, the research presented in Chapter 1 finds that state writing assessments and the NAEP seem to align in their adoption of the writing process approach, their attention to audience and students' topical knowledge, their accommodations through procedure facilitators, and their inclusion of organization, structure, content, details, sentence fluency, and semantic aspects as well as general conventions, such as punctuation, spelling, and grammar in their assessment criteria. However, the NAEP's writing assessment differs from many states' by having explicit directions for students to review their writing, giving students two timed writing tasks, making informative composition--which was rarely included in state assessments--one of the three genres assessed, and including genre-specific components in their writing rubrics. The fact that all of the NAEP's writing rubrics are genre-mastery rubrics with genre-specific components can be considered one of its biggest differences from most state writing assessments. To examine the impact of the variations between state and national writing assessments through Hierarchical Linear Modeling, the research presented in Chapter 2 examines the relationship between students' NAEP performances and the amount of difference between state and NAEP direct writing assessments using content analysis of the state and NAEP prompts and rubrics detailed above. This study finds that students' preparedness for the tasks, namely the similarity between the assessments of their home states and the NAEP, plays a role in students' performance on the NAEP. Students from those states with writing assessments similar to the NAEP performed significantly better than students from states with writing assessments that differed markedly from the NAEP. Through syntactic analysis of the same set of state prompts and content analysis of rubrics and standards, the research presented in Chapter 3 explores genre demands in state writing assessments. In total, this study found that 23% of prompts possessed one of two problematic features: 14% of prompts were ambiguous, and 9% of prompts had implicit genre expectations. Almost one third of those prompts that possessed problematic features were used with genre-mastery rubrics. The content analysis of state writing standards also suggests that 22% of them do not cover all the genres assessed in their corresponding writing assessments. The ambiguity and implicit genre expectations in writing prompts and the limited congruence of state writing assessments with learning expectations pose potential threats to the valid interpretation and use of these writing assessments.
Read
