Introducing Flexible Monotone Multiple Choice Item Response Theory Models and Bit Scales

TL;DR

This paper introduces a flexible monotone multiple choice IRT model fitted with autoencoders, demonstrating improved data fit and the development of bit scales for better score interpretation in test analysis.

Contribution

The paper presents a novel monotone multiple choice IRT model and a method to transform latent trait scales into ratio scales called bit scales.

Findings

MMC model outperforms traditional nominal response IRT in data fit

Autoencoder fitting enables flexible modeling without strict distribution assumptions

Bit scales facilitate easier interpretation and comparison of IRT scores

Abstract

Item Response Theory (IRT) is a powerful statistical approach for evaluating test items and determining test taker abilities through response analysis. An IRT model that better fits the data leads to more accurate latent trait estimates. In this study, we present a new model for multiple choice data, the monotone multiple choice (MMC) model, which we fit using autoencoders. Using both simulated scenarios and real data from the Swedish Scholastic Aptitude Test, we demonstrate empirically that the MMC model outperforms the traditional nominal response IRT model in terms of fit. Furthermore, we illustrate how the latent trait scale from any fitted IRT model can be transformed into a ratio scale, aiding in score interpretation and making it easier to compare different types of IRT models. We refer to these new scales as bit scales. Bit scales are especially useful for models for which minimal or no assumptions are made for the latent trait scale distributions, such as for the autoencoder fitted models in this study.