Exploring the Potential of Large Language Models for Estimating the Reading Comprehension Question Difficulty

TL;DR

This study explores the use of large language models, GPT-4, to automate the estimation of reading comprehension question difficulty, aiming to enhance scalability and personalization in educational assessments.

Contribution

It demonstrates that GPT-4 can effectively estimate question difficulty levels, aligning with traditional psychometric measures, and highlights its potential for scalable, adaptive educational systems.

Findings

GPT-4's difficulty estimates correlate with IRT parameters

Models show sensitivity to extreme item characteristics

Potential for scalable, automated assessment in education

Abstract

Reading comprehension is a key for individual success, yet the assessment of question difficulty remains challenging due to the extensive human annotation and large-scale testing required by traditional methods such as linguistic analysis and Item Response Theory (IRT). While these robust approaches provide valuable insights, their scalability is limited. There is potential for Large Language Models (LLMs) to automate question difficulty estimation; however, this area remains underexplored. Our study investigates the effectiveness of LLMs, specifically OpenAI's GPT-4o and o1, in estimating the difficulty of reading comprehension questions using the Study Aid and Reading Assessment (SARA) dataset. We evaluated both the accuracy of the models in answering comprehension questions and their ability to classify difficulty levels as defined by IRT. The results indicate that, while the models yield difficulty estimates that align meaningfully with derived IRT parameters, there are notable differences in their sensitivity to extreme item characteristics. These findings suggest that LLMs can serve as the scalable method for automated difficulty assessment, particularly in dynamic interactions between learners and Adaptive Instructional Systems (AIS), bridging the gap between traditional psychometric techniques and modern AIS for reading comprehension and paving the way for more adaptive and personalized educational assessments.