SPRITE: A Response Model For Multiple Choice Testing

TL;DR

SPRITE is a new response model for multiple-choice testing that effectively handles both ordered and unordered categories, offering better data fitting and interpretability compared to existing IRT models.

Contribution

Introduces SPRITE, a novel unordered categorical IRT model that improves data fitting and interpretability in educational testing scenarios.

Findings

SPRITE outperforms existing models on synthetic data.

SPRITE demonstrates superior fit on real-world datasets.

Model provides interpretable parameters for categories.

Abstract

Item response theory (IRT) models for categorical response data are widely used in the analysis of educational data, computerized adaptive testing, and psychological surveys. However, most IRT models rely on both the assumption that categories are strictly ordered and the assumption that this ordering is known a priori. These assumptions are impractical in many real-world scenarios, such as multiple-choice exams where the levels of incorrectness for the distractor categories are often unknown. While a number of results exist on IRT models for unordered categorical data, they tend to have restrictive modeling assumptions that lead to poor data fitting performance in practice. Furthermore, existing unordered categorical models have parameters that are difficult to interpret. In this work, we propose a novel methodology for unordered categorical IRT that we call SPRITE (short for stochastic polytomous response item model) that: (i) analyzes both ordered and unordered categories, (ii) offers interpretable outputs, and (iii) provides improved data fitting compared to existing models. We compare SPRITE to existing item response models and demonstrate its efficacy on both synthetic and real-world educational datasets.