The aim of this research is to investigate the differential effects of multiple-choice items with three, four, and five options on a high-stakes, English language listening test. Three-option multiple-choice items on an English listening test were compared with items with four and five options in terms of average total score assignments, average item facility, average item discrimination, overall test reliability, and processing time. Three prep English listening tests for the CSAT (College Scholastic Aptitude Test), each with five-option items, were adapted into parallel forms with four- and three-option items by eliminating the least plausible options, as selected by 73 Koreans. A total of 264 Korean EFL learners, divided into three groups, participated in the study. Each group took tests with five-, four-, and three-option items. The test administrations were based on Latin squares to control for order and practice effects. Results indicated that the average scores between tests with three- and five-option items differed significantly. However, there was no significant difference in the average item facility, average item discrimination, and overall reliability among the tests with the different number of options. Regarding time on exam, the three-option-item test took 11% less time than the five-option-item test. Survey data from the 265 test takers revealed that 54% preferred the three-option-item format. Also, 68% agreed administering the CSAT with three-option items would be preferable. Results are discussed from the perspective of statistical, cognitive, emotional and contextual factors in determining the optimal number of options.
Read
