Language tests must be cognitively valid and are expected to elicit from test takers the same type and level of cognitive processing as similar real-world tasks (Bax & Weir, 2012; Bax, 2013; Field, 2009, 2011). However, recent test validations of language tests have heavily relied on correlation-based psychometric methods, which primarily assess test outcomes rather than test-taking processes. Such a statistical approach is limited because numbers do not convey conceptual information and test scores do not explain how test takers derive their answers (Weir, 2005). Psychometricians have emphasized the importance of cognitively based test validation and development (Borsboom, 2005; Gorin, 2007). Though language testers have agreed that understanding test takers' cognition is important for test validation, empirical studies of high-stakes academic reading tests are relatively scarce (e.g., Bax, 2013; Bax & Weir, 2012; Chapelle, Enright, & Jamieson, 2008; Cohen & Upton, 2007). The present study explores cognitive evidence of the validity of the TOEFL iBT reading test by examining Chinese ESL students' test-taking behaviors. More specifically, it investigates whether the test exploits construct-relevant cognitive resources and activates the intended cognitive processes while concomitantly investigating the effect of the construct-irrelevant variable (e.g., test-taking strategy). The guiding research questions are 1) to what extent does the TOEFL iBT reading test measure the reading construct?; 2) to what extent does the TOEFL iBT reading test activate the cognitive processes being measured?; 3) to what extent does test format affect students' test scores and test-taking processes? Ninety Chinese ESL students attending a large midwestern university participated in the main experiment, and 206 Chinese students in the US completed the reading difficulty survey questionnaire online. During the experiment period, participants took two reading testlets, one with multiple-choice questions and the other with open-ended questions, on a computer whose screen was fitted with a Tobii TX300 eye-tracker. Their eye movements were recorded while taking the test. Participants also completed a vocabulary test, a grammar test, a lexical processing task, a sentence processing task, a working memory task, a strategy questionnaire, and a stimulated recall interview. The results of three different analyses were found to be inconsistent: the regression analysis showed that the most important construct-relevant variables (vocabulary knowledge, grammar knowledge, and word recognition skill) best predicted test scores, while the effect of the construct-irrelevant variable (e.g., test-taking strategy) was not significant. However, according to the eye-tracking analysis, some of the test items failed to activate the intended reading processes: expeditious reading skills were rarely activated, especially at the global level, and participants did not necessarily read to find the key answer, especially in the vocabulary and reading-to-learn questions. Furthermore, successful readers were not always faster at reading and locating key information. Lastly, the investigation on the test format effect revealed that participants scored higher on the multiple-choice questions and used different reading processes depending on the test format. Because the effects of test format suggest the possible involvement of test-taking strategies (Rogers & Harley, 1999), the result of the third analysis may contradict the result of the earlier regression analysis. Discussed are the implications for validity arguments, test development, and further research.
Read
