Alignment index values are often presented as evidence that test content is representative of the performance domain defined by a written curriculum. Alignment measures bear on the validity of state achievement test score interpretations, and on test fairness. While alignment reports have been used to document test content distribution, and to generate recommendations for test form improvement that appear sensible to assessment professionals, there is little external evidence that alignment index values are valid quantitative measures of tests’ content representativeness. Using eleven states' mathematics achievement test item and Surveys of Enacted Curriculum (SEC; Porter, 2002) content analysis data, I examine external validation evidence for the curriculum emphasis measures underlying the coarse-grained SEC test-curriculum alignment index. I then use fractional logit regression models to assess the relationship between state-level test-curriculum alignment and proportion-correct item difficulty on corresponding test items, controlling for other state and test item characteristics that may affect achievement. I find that the content analysis proportions that summarize SEC alignment panelists' judgments about curriculum objectives' topics and cognitive demand requirements seem to relate in expected ways to other measures of state curricular emphasis in Grade 4, providing weak external validation support for the mathematics curriculum content analysis data. However, there is no evidence of a statistically or substantively significant relationship between an alignment measure based on SEC coarse-grained content analysis data, and test item difficulty in Grade 8, regardless of the extent to which a particular item's content type is emphasized by the curriculum. I conclude that although external validation studies for the SEC alignment index have tended to focus on the relationship between curricular alignment and test performance, other types of evidence may yield clearer conclusions, and perhaps be more crucial for demonstrating these indices' validity. Specifically, I suggest future research to evaluate the content classification schemes implemented by popular alignment methods, and to establish the reproducibility of overall and item-level alignment results across independent panels of qualified experts trained by different facilitators.
Read
