Reconstructing Item Characteristic Curves using Fine-Tuned Large Language Models

TL;DR

This paper presents a novel method using fine-tuned Large Language Models to simulate student responses and reconstruct Item Characteristic Curves, reducing reliance on costly field testing for IRT calibration.

Contribution

It introduces a new approach that leverages LLMs and LoRA to implicitly model psychometric properties and generate synthetic ICCs for assessment items.

Findings

Method competes with baseline approaches

Effective at modeling item discrimination

Reduces need for expensive field testing

Abstract

Traditional methods for determining assessment item parameters, such as difficulty and discrimination, rely heavily on expensive field testing to collect student performance data for Item Response Theory (IRT) calibration. This study introduces a novel approach that implicitly models these psychometric properties by fine-tuning Large Language Models (LLMs) to simulate student responses across a spectrum of latent abilities. Leveraging the Qwen-3 dense model series and Low-Rank Adaptation (LoRA), we train models to generate responses to multiple choice questions conditioned on discrete ability descriptors. We reconstruct the probability of a correct response as a function of student ability, effectively generating synthetic Item Characteristic Curves (ICCs) to estimate IRT parameters. Evaluation on a dataset of Grade 6 English Language Arts (ELA) items and the BEA 2024 Shared Task dataset demonstrates that this method competes with or outperforms baseline approaches. This simulation-based technique seems particularly effective at modeling item discrimination.