The NCSSFL-ACTFL (2015) Can-Do Statements describe what language learners can do in the target language at the various ACTFL Proficiency sublevels. Unlike the extensive work that has been done to scale and refine the proficiency descriptors and corresponding can-do statements associated with the CEFR (Council of Europe, 2001) using Rasch modeling (North, 2000; North & Schneider, 1998), the NCSSFL-ACTFL statements have yet to be empirically tested. Both the scale and its foreign language performance indicators were constructed using language teachers’ beliefs and experiences (Shin, 2013). While this is a logical starting point, concerns include whether the difficulty levels of the skills described in the statements match their assigned ACTFL proficiency levels, and whether each statement accurately measures the underlying construct: language proficiency on the ACTFL subscales.This study addresses these concerns by analyzing a self-assessment instrument composed of fifty NCSSFL-ACTFL (2015) Can-Do Statements targeting spoken language proficiency. American university students of varying proficiency levels in Spanish language classes (N = 382) rated the Can-Do Statements as: 1 (I cannot do this yet), 2 (I can do this with some help), 3 (I can do this with much help), 4 (I can do this). I analyzed their item responses using a Rasch rating scale model (Andrich, 1978; Rasch, 1960/80). I compared the difficulty levels estimated by the model to the proficiency levels assigned to the statements, and assessed each item’s fit to the model by considering the item’s measures of infit and outfit. The mean item difficulties estimated by the Rasch model matched the difficulty level predicted by the ACTFL scale at the major threshold proficiency levels, and these differences were statistically significant. The mean item difficulties did not show statistically significant increases at the ACFTL proficiency sublevel, and there was a decrease in difficulty from Advanced Low (M = 1.72, SD = 1.51) to Advanced Mid (M = 1.37, SD = 1.32) items, rather than an increase. The analysis also revealed 14 items that did not fit the model measuring spoken proficiency. In the second phase of the study, I revised the self-assessment instrument based on the findings of the first phase, and the revised assessment was used for a second round of testing. Spanish language learners (N = 886) rated the items in the revised instrument of the ACTFL (2015) Can-Do Statements. I analyzed their item responses using an exploratory factor analysis (EFA) and a Rasch rating scale model. The results of the EFA revealed two possible models of spoken language proficiency as represented by the Can-Do Statements that were included in the instrument: a unidimensional model in line with ACTFL’s unitary and hierarchical model of spoken proficiency, and a two-factor model. The Rasch analysis revealed that some of the items and some of the test takers in the analysis did not behave as expected. The analysis also replicated the finding that the mean item difficulties estimated by the Rasch model matched the difficulty level predicted by the ACTFL scale at the major threshold proficiency levels, and that these differences were statistically significant. The mean item difficulties in the revised assessment also ascended according to the ACTFL sublevels: There were significant differences between items from the lower proficiency sublevels, but the instrument did not discriminate well between statements pegged at higher proficiency sublevels. Findings are discussed in terms of how the NCSSFL-ACTFL (2015) Can-Do Statements can be used to self-assess spoken language proficiency, and how the statements should be assessed for content validity and psychometric value.
Read
