Sequential Design for Computerized Adaptive Testing that Allows for Response Revision

TL;DR

This paper introduces a novel computerized adaptive testing (CAT) method that uses polytomous models and allows test-takers to revise responses, improving flexibility and data utilization while maintaining statistical consistency.

Contribution

It proposes a new CAT design based on the polytomous nominal response model that permits response revisions, with proven consistency and asymptotic normality.

Findings

Estimator is strongly consistent under any strategy.

Estimator is asymptotically normal with Fisher information maximization.

Simulation supports theoretical asymptotic results.

Abstract

In computerized adaptive testing (CAT), items (questions) are selected in real time based on the already observed responses, so that the ability of the examinee can be estimated as accurately as possible. This is typically formulated as a non-linear, sequential, experimental design problem with binary observations that correspond to the true or false responses. However, most items in practice are multiple-choice and dichotomous models do not make full use of the available data. Moreover, CAT has been heavily criticized for not allowing test-takers to review and revise their answers. In this work, we propose a novel CAT design that is based on the polytomous nominal response model and in which test-takers are allowed to revise their responses at any time during the test. We show that as the number of administered items goes to infinity, the proposed estimator is (i) strongly consistent for any item selection and revision strategy and (ii) asymptotically normal when the items are selected to maximize the Fisher information at the current ability estimate and the number of revisions is smaller than the number of items. We also present the findings of a simulation study that supports our asymptotic results.