Deep Computerized Adaptive Testing

TL;DR

This paper introduces a novel Bayesian multivariate IRT-based adaptive testing system that accelerates item selection and employs reinforcement learning to optimize test strategies, addressing complex real-world data structures.

Contribution

It develops a multivariate latent trait CAT framework with faster sampling and a deep Q-learning approach for optimal item selection, advancing adaptive testing methodologies.

Findings

Accelerates item selection by direct sampling from posterior distributions.

Demonstrates improved efficiency and effectiveness in simulation and real-data studies.

Highlights the potential of reinforcement learning in adaptive testing.

Abstract

Computerized adaptive tests (CATs) play a crucial role in educational assessment and diagnostic screening in behavioral health. Unlike traditional linear tests that administer a fixed set of pre-assembled items, CATs adaptively tailor the test to an examinee's latent trait level by selecting a smaller subset of items based on their previous responses. Existing CAT frameworks predominantly rely on item response theory (IRT) models with a single latent variable, a choice driven by both conceptual simplicity and computational feasibility. However, many real-world item response datasets exhibit complex, multi-factor structures, limiting the applicability of CATs in broader settings. In this work, we develop a novel CAT system that incorporates multivariate latent traits, building on recent advances in Bayesian sparse multivariate IRT. Our approach leverages direct sampling from the latent factor posterior distributions, significantly accelerating existing information-theoretic item selection criteria by eliminating the need for computationally intensive Markov Chain Monte Carlo (MCMC) simulations. Recognizing the potential sub-optimality of existing item selection rules, which are often based on myopic one-step-lookahead optimization of some information-theoretic criterion, we propose a double deep Q-learning algorithm to learn an optimal item selection policy. Through simulation and real-data studies, we demonstrate that our approach not only accelerates existing item selection methods but also highlights the potential of reinforcement learning in CATs.