The interest in multidimensional computerized adaptive testing (MCAT) has grown considerably over the last few years. While a significant amount of research has been conducted on item selection and ability estimation methods for MCAT, few studies specifically addressed the item pool design for MCAT. To ensure a proper functioning of MCAT, a well-designed item pool is imperative. A well-designed item pool should consist of a number of well-balanced items that achieve appropriate test precision, item usage, as well as lower the cost of item creation. One method to develop such an item pool is the p-optimality method, which is proposed by Reckase (2003 & 2007) for unidimensional CAT. This paper aims to develop p-optimal item pools for MCAT by extending the Reckase's method to a multidimensional context. The extension includes the generation of a multidimensional optimal item based on the D-Optimality item selection creation, the definition of the MDIFF-bin to describe multidimensional item succinctly for item pool design, and the interpretation for the p-optimal item pool in a multidimensional context. In this paper, a total of 24 p-optimal item pools were designed and then developed for different test specification, with different correlation among dimensions, based on different bin size, and under the condition with or without item exposure control. The characteristics for the 24 p-optimal item pools are summarized. A simulation study was conducted to evaluate the performance of the p-optimal item pools against baseline pools existing in research literature. Results show that p-optimal item pools achieve similar levels of measurement accuracy as baseline pools, but they consist of fewer items and perform better in terms of item pool usage and test security. The characteristics and the performance of the p-optimal item pools are affected by factors such as test specification, correlation among dimensions, bin size, and item exposure control. The results in this study can provide a general guideline for the item pool development for MCAT. More importantly, because the p-optimal item pool is specifically tailored to the MCAT programs, the p-optimal item pool design procedure described in this study can be adapted to other MCAT programs with different features and purposes. The end product of the p-optimal item pool design can be used as an instructive guide for item creation, item pool development, and item pool management.
Read
