Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory

TL;DR

This paper introduces an item response theory framework to evaluate LLM-based automated short answer grading, providing detailed insights into response difficulty and model performance variations beyond traditional aggregate metrics.

Contribution

The study applies IRT to analyze LLM grading performance, revealing how accuracy varies with response difficulty and identifying factors influencing grading errors.

Findings

Models with similar overall performance differ in accuracy decline with response difficulty.

Errors on difficult responses often involve the partially_correct_incomplete label.

Higher response difficulty correlates with weaker semantic alignment and greater semantic isolation.

Abstract

Automated short answer grading (ASAG) with large language models (LLMs) is commonly evaluated with aggregate metrics such as macro-F1 and Cohen's kappa. However, these metrics provide limited insight into how grading performance varies across student responses of differing grading difficulty. We introduce an evaluation framework for LLM-based ASAG based on item response theory (IRT), which models grading correctness as a function of latent grader ability and response grading difficulty. This formulation enables response-level analysis of where LLM graders succeed or fail and reveals robustness differences that are not visible from aggregate scores alone. We apply the framework to 17 open-weight LLMs on the SciEntsBank and Beetle benchmarks. The results show that even models with similar overall performance differ substantially in how sharply their grading accuracy declines as response difficulty increases. In addition, confusion patterns show that errors on difficult responses concentrate disproportionately on the \texttt{partially\_correct\_incomplete} label, indicating a tendency toward intermediate-label collapse under ambiguity. To characterize difficult responses, we further analyze semantic and linguistic correlates of estimated difficulty. Across both datasets, higher difficulty is associated with weaker semantic alignment to the reference answer, stronger contradiction signals, and greater semantic isolation in embedding space. Overall, these results show that item response theory offers a useful framework for evaluating LLM-based ASAG beyond aggregate performance measures.