Item Quality Control in Educational Testing: Change Point Model, Compound Risk, and Sequential Detection

TL;DR

This paper develops a statistical framework for sequentially detecting abrupt changes in test item properties in educational testing, balancing false alarms and missed detections, using a Bayesian change point model and item response theory.

Contribution

It introduces a novel multi-stream Bayesian change point model and sequential decision rules for monitoring item quality, accounting for population changes and controlling compound risk.

Findings

The proposed method effectively detects item changes in simulated operational test settings.

The decision rules balance false detection and missed detection rates.

The framework is applicable to real-time quality control in educational assessments.

Abstract

In standardized educational testing, test items are reused in multiple test administrations. To ensure the validity of test scores, the psychometric properties of items should remain unchanged over time. In this paper, we consider the sequential monitoring of test items, in particular, the detection of abrupt changes to their psychometric properties, where a change can be caused by, for example, leakage of the item or change of the corresponding curriculum. We propose a statistical framework for the detection of abrupt changes in individual items. This framework consists of (1) a multi-stream Bayesian change point model describing sequential changes in items, (2) a compound risk function quantifying the risk in sequential decisions, and (3) sequential decision rules that control the compound risk. Throughout the sequential decision process, the proposed decision rule balances the trade-off between two sources of errors, the false detection of pre-change items and the non-detection of post-change items. An item-specific monitoring statistic is proposed based on an item response theory model that eliminates the confounding from the examinee population which changes over time. Sequential decision rules and their theoretical properties are developed under two settings: the oracle setting where the Bayesian change point model is completely known and a more realistic setting where some parameters of the model are unknown. Simulation studies are conducted under settings that mimic real operational tests.