"Recent research in the areas of second language testing and learner corpus research has provided increased insight into linguistic features of various score levels and into the meaning of a test score (Cushing, 2017; Knoch & Chapelle, 2018). However, language testing researchers have asserted the need to select linguistic features that are relevant to the test construct for test validation purposes (Egbert, 2017; Xi, 2017). In addition, the Common European Framework of Reference (CEFR) has been widely adopted in testing contexts and provides level descriptions for linguistic abilities, but empirical validation of its use in various testing contexts is critical (Wisniewski, 2017, 2018). Addressing these two limitations, I drew upon learner-produced written English from a large-scale English exam, the Certificate of English Language Competency (CEFR B2-level certification). The aim of the study was to (a) investigate specific grammatical features and overall linguistic accuracy of second language English texts to reveal patterns of language use at different score levels, and (b) examine how well rating rubric descriptors reflect characteristics of examinee texts and differentiate between score levels to find evidence for test validity. In order to provide concrete, context-relevant grammatical features for investigation, I selected 14 grammatical features from the English Profile studies (English Profile, 2015; Hawkins & Filipovic, 2012), which have also been documented in L2 writing research. Data included 560 texts written on three different topics and ranging across five levels of performance. I extracted the occurrences of 14 grammatical features from the corpus using Natural Language Processing tools and analyzed occurrences of these features attested in the corpus. Additionally, a subset of the texts was manually coded for error to examine overall accuracy of each texts. Consistent with the findings in existing literature, I found significant differences in the frequencies of certain clausal features across lower score levels. Both the frequencies of the 14 grammatical features and the overall number of different types of these features used in each text were moderately useful in predicting the grammar subscore. I identified co-occurring patterns of the target grammatical features by performing a principal components analysis. The results showed that grammar structures that are of similar types (e.g., finite, non-finite) and functions (e.g., complement, noun modifier) tended to occur together and exhibited (cross-sectional) developmental patterns. For a subset of data coded for errors, the error-free clause ratio was calculated, which significantly distinguished between each pair of adjacent levels. This study's findings highlight the need for empirical investigation of how learner language has been described by experts in proficiency descriptors (e.g., Council of Europe, 2001, 2018) and how reliably the constructs of rubric descriptors attest in test performance data. I suggest that writing assessment materials can benefit from reference to the tangible characteristics of L2 development found in writing development research (e.g., phrasal complexity, morphological accuracy, and association strength between a construction and its lexis)."--Pages ii-iii.
Read
