LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias

TL;DR

This paper evaluates how well instruction-tuned LLMs align with human essay scoring, revealing biases and the impact of prompt design, and suggests bias correction strategies for educational assessment.

Contribution

It systematically assesses LLM scoring agreement, bias, and prompt effects across multiple datasets, highlighting the importance of bias correction over fine-tuning.

Findings

LLMs achieve moderate to high agreement on holistic scores (QWK about 0.6).

Models show stable negative bias on Lower-Order Concern traits like Grammar.

Concise keyword prompts outperform longer rubric prompts in analytic scoring.

Abstract

Despite growing interest in using Large Language Models (LLMs) for educational assessment, it remains unclear how closely they align with human scoring. We present a systematic evaluation of instruction-tuned LLMs across three open essay-scoring datasets (ASAP 2.0, ELLIPSE, and DREsS) that cover both holistic and analytic scoring. We analyze agreement with human consensus scores, directional bias, and the stability of bias estimates. Our results show that strong open-weight models achieve moderate to high agreement with humans on holistic scoring (Quadratic Weighted Kappa about 0.6), but this does not transfer uniformly to analytic scoring. In particular, we observe large and stable negative directional bias on Lower-Order Concern (LOC) traits, such as Grammar and Conventions, meaning that models often score these traits more harshly than human raters. We also find that concise keyword-based prompts generally outperform longer rubric-style prompts in multi-trait analytic scoring. To quantify the amount of data needed to detect these systematic deviations, we compute the minimum sample size at which a 95% bootstrap confidence interval for the mean bias excludes zero. This analysis shows that LOC bias is often detectable with very small validation sets, whereas Higher-Order Concern (HOC) traits typically require much larger samples. These findings support a bias-correction-first deployment strategy: instead of relying on raw zero-shot scores, systematic score offsets can be estimated and corrected using small human-labeled bias-estimation sets, without requiring large-scale fine-tuning.