Estimating Exam Item Difficulty with LLMs: A Benchmark on Brazil's ENEM Corpus

TL;DR

This paper benchmarks ten LLMs against official exam data to assess their ability to estimate question difficulty, revealing moderate accuracy but limitations in calibration and context sensitivity, especially for multimodal items.

Contribution

It introduces a comprehensive benchmark of LLMs on Brazil's ENEM questions, highlighting their current capabilities and limitations in difficulty estimation for educational content.

Findings

LLMs achieve moderate rank correlation with official difficulty measures.

Models tend to underestimate question difficulty systematically.

Limited and inconsistent adaptation to student demographic cues.

Abstract

As Large Language Models (LLMs) are increasingly deployed to generate educational content, a critical safety question arises: can these models reliably estimate the difficulty of the questions they produce? Using Brazil's high-stakes ENEM exam as a testbed, we benchmark ten proprietary and open-weight LLMs against official Item Response Theory (IRT) parameters for 1,031 questions. We evaluate performance along three axes: absolute calibration, rank fidelity, and context sensitivity across learner backgrounds. Our results reveal a significant trade-off: while the best models achieve moderate rank correlation, they systematically underestimate difficulty and degrade significantly on multimodal items. Crucially, we find that models exhibit limited and inconsistent plasticity when prompted with student demographic cues, suggesting they are not yet ready for context-adaptive personalization. We conclude that LLMs function best as calibrated screeners rather than authoritative oracles, supporting an "evaluation-before-generation" pipeline for responsible assessment design.