In this study, an index to quantify the adequacy of an item pool of an adaptive test for a given set of test specifications and examinee population is introduced. This index is called the Item Pool Utilization Index (IPUI). The IPUI ranges from 0 to 1, with values close to 1 indicating the item pool can provide optimum items to examinees throughout the test. This index can be used to compare different item pools or diagnose the deficiencies of a given item pool by quantifying the amount of deviation from a perfect item pool. Simulation studies were conducted to evaluate the capacity of this index for detecting the inadequacies of both simulated and operational item pools. The added value of this index was compared to the existing methods of evaluating the quality of computerized adaptive tests (CAT). Results of the study showed that the IPUI can detect even slight deviations of the item pools from an optimal item pool. It can uncover the shortcomings of an item pool that other outcomes of CAT cannot detect. Additionally, it can be used to diagnose the weaknesses of the item pool and guide test developers to improve their item pools.
Read
