Reinforcement Learning Measurement Model

TL;DR

The paper introduces the Reinforcement Learning Measurement Model (RLMM), a scalable framework for analyzing sequential process data in assessments, improving estimation efficiency and interpretability over existing models.

Contribution

It proposes a novel RLMM that decouples person-level choice sensitivity from task-level value representation, enabling efficient analysis of larger, more realistic process data.

Findings

RLMM achieved higher accuracy than MDP-MM in simulations.

RLMM had substantially lower runtime as task complexity increased.

Estimated person parameters correlated positively with performance metrics.

Abstract

Interactive assessments generate sequential process data that are not well handled by conventional item response models. Existing MDP-based measurement approaches, such as the Markov decision process measurement model (MDP-MM, LaMar, 2018), link action choices to state-action values, but their reliance on person-specific tabular value functions makes them difficult to scale beyond small, fully enumerated tasks. We propose the Reinforcement Learning Measurement Model (RLMM), a measurement framework that decouples person-level choice sensitivity from task-level value representation through a shared parametric action-value function, making estimation more computationally efficient for larger process-data settings. The model combines a Boltzmann choice rule with normalized advantages, a soft Bellman consistency penalty, and a block-coordinate MAP procedure for joint estimation, while also yielding step-level influence diagnostics for identifying behaviorally critical decisions. In peg-solitaire simulations, the RLMM achieved higher estimation accuracy and substantially lower runtime than the original MDP-MM, with advantages increasing as task complexity grew. In AQUALAB gameplay logs, the estimated person parameter was positively associated with cumulative reward, task completion, and behavioral efficiency. These results show that the RLMM extends decision-process-based psychometric models to larger and more behaviorally realistic environments while preserving an interpretable latent trait tied to decision making steps.