In recent years, computerized adaptive testing (CAT) has been widely used in educational and clinical settings. The basic idea of CAT is relatively straightforward. A computer is used to administer items tailored for individuals to maximize the measurement precision of their proficiency estimates. However, the administration of CAT is not so simple. Those who administer CATs must, while trying to optimize an item selection criterion, consider a variety of practical issues such as test security, content balancing, the purpose of testing, and other test specifications. Such extraneous factors make it possible that a CAT might have so many constraints that in practice it is barely adaptive at all. This concern is at the forefront of the current study, which poses two key questions: How adaptive is a highly adaptive test really? How can the level of adaptation be improved? This study aims to develop three new statistical indicators to measure the amount of adaptation conditional on the examinees' proficiency levels in CAT. It also aims to evaluate the feasibility and utility of these adaptation measures in helping to diagnose and improve adaptivity that occurs during the CAT administration. Extending work done by Reckase, Ju, and Kim (2018), the proposed measures are based on three components-the differences in the locations between the selected items and the examinee's current proficiency estimates, the variations in the item locations administered to each examinee, and the magnitude of information that the test presents to each examinee. Hence, they can be used to assess adaptivity during the CAT process, as well as to identify differences in the level of adaptation for individuals or subgroups of examinees. To demonstrate the performance of the proposed adaptation indices, this study conducted analyses of real operational testing data from a healthcare licensure examination, as well as comprehensive simulation studies under various conditions that affect adaptivity in a CAT. The key findings of the study suggest that the proposed adaptation indices are likely to function as intended to sensitively detect the magnitude of adaptivity for a CAT over the proficiency continuum. These new measures shed light on how much adaptation of a given test occurs across individual proficiency levels or subpopulations. With some guidelines for the interpretation of these measures recommended in this study, the adaptation indices can also readily serve as diagnostic tools in practice for helping test practitioners design item pools and adaptive tests that support high adaptivity.
Read
