Bayesian analysis of dynamic item response models in educational testing

TL;DR

This paper introduces Dynamic Item Response (DIR) models, a new class of state space models for analyzing longitudinal educational testing data, accommodating ability changes, response dependencies, and item difficulty uncertainty.

Contribution

The paper develops DIR models that handle dynamic abilities, response dependencies, and uncertain item difficulties, with applications to real-time and retrospective educational testing analysis.

Findings

Models perform well on simulated data

Applied successfully to large reading test datasets

Enable real-time prediction and retrospective analysis

Abstract

Item response theory (IRT) models have been widely used in educational measurement testing. When there are repeated observations available for individuals through time, a dynamic structure for the latent trait of ability needs to be incorporated into the model, to accommodate changes in ability. Other complications that often arise in such settings include a violation of the common assumption that test results are conditionally independent, given ability and item difficulty, and that test item difficulties may be partially specified, but subject to uncertainty. Focusing on time series dichotomous response data, a new class of state space models, called Dynamic Item Response (DIR) models, is proposed. The models can be applied either retrospectively to the full data or on-line, in cases where real-time prediction is needed. The models are studied through simulated examples and applied to a large collection of reading test data obtained from MetaMetrics, Inc.