Synthetic Student Responses: LLM-Extracted Features for IRT Difficulty Parameter Estimation

TL;DR

This paper presents a method to estimate question difficulty in educational assessments using LLM-extracted features and neural network modeling, eliminating the need for extensive student testing.

Contribution

It introduces a novel approach combining linguistic and pedagogical features from LLMs to predict IRT difficulty parameters without student responses.

Findings

Achieved a Pearson correlation of 0.78 on unseen questions.

Demonstrated effectiveness with over 250,000 student responses.

Integrated linguistic and pedagogical features for improved prediction.

Abstract

Educational assessment relies heavily on knowing question difficulty, traditionally determined through resource-intensive pre-testing with students. This creates significant barriers for both classroom teachers and assessment developers. We investigate whether Item Response Theory (IRT) difficulty parameters can be accurately estimated without student testing by modeling the response process and explore the relative contribution of different feature types to prediction accuracy. Our approach combines traditional linguistic features with pedagogical insights extracted using Large Language Models (LLMs), including solution step count, cognitive complexity, and potential misconceptions. We implement a two-stage process: first training a neural network to predict how students would respond to questions, then deriving difficulty parameters from these simulated response patterns. Using a dataset of over 250,000 student responses to mathematics questions, our model achieves a Pearson correlation of approximately 0.78 between predicted and actual difficulty parameters on completely unseen questions.