SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Prediction

TL;DR

SMART introduces a novel approach that uses simulated students aligned with IRT to predict question difficulty, reducing reliance on real student responses and improving accuracy in educational assessments.

Contribution

The paper presents SMART, a new method that aligns simulated student responses with IRT for better difficulty prediction, especially for unseen items.

Findings

SMART outperforms existing difficulty prediction methods.

The alignment improves accuracy in open-ended item difficulty estimation.

Extensive experiments validate the effectiveness of SMART.

Abstract

Item (question) difficulties play a crucial role in educational assessments, enabling accurate and efficient assessment of student abilities and personalization to maximize learning outcomes. Traditionally, estimating item difficulties can be costly, requiring real students to respond to items, followed by fitting an item response theory (IRT) model to get difficulty estimates. This approach cannot be applied to the cold-start setting for previously unseen items either. In this work, we present SMART (Simulated Students Aligned with IRT), a novel method for aligning simulated students with instructed ability, which can then be used in simulations to predict the difficulty of open-ended items. We achieve this alignment using direct preference optimization (DPO), where we form preference pairs based on how likely responses are under a ground-truth IRT model. We perform a simulation by generating thousands of responses, evaluating them with a large language model (LLM)-based scoring model, and fit the resulting data to an IRT model to obtain item difficulty estimates. Through extensive experiments on two real-world student response datasets, we show that SMART outperforms other item difficulty prediction methods by leveraging its improved ability alignment.