Autoscoring Anticlimax: A Meta-analytic Understanding of AI's Short-answer Shortcomings and Wording Weaknesses

TL;DR

This meta-analysis reveals that current LLMs struggle with short-answer scoring tasks, especially on easier items for humans, and highlights technological and bias-related shortcomings affecting their performance.

Contribution

The study provides a comprehensive meta-analytic assessment of LLM short-answer scoring, identifying key technological limitations and biases affecting performance.

Findings

LLMs underperform compared to humans on easy scoring tasks.

Decoder-only architectures lag behind encoder models by 0.37 in agreement.

Tokenizer vocabulary size shows diminishing returns, indicating undertraining issues.

Abstract

Automated short-answer scoring lags other LLM applications. We meta-analyze 890 culminating results across a systematic review of LLM short-answer scoring studies, modeling the traditional effect size of Quadratic Weighted Kappa (QWK) with mixed effects metaregression. We quantitatively illustrate that that the level of difficulty for human experts to perform the task of scoring written work of children has no observed statistical effect on LLM performance. Particularly, we show that some scoring tasks measured as the easiest by human scorers were the hardest for LLMs. Whether by poor implementation by thoughtful researchers or patterns traceable to autoregressive training, on average decoder-only architectures underperform encoders by 0.37--a substantial difference in agreement with humans. Additionally, we measure the contributions of various aspects of LLM technology on successful scoring such as tokenizer vocabulary size, which exhibits diminishing returns--potentially due to undertrained tokens. Findings argue for systems design which better anticipates known statistical shortcomings of autoregressive models. Finally, we provide additional experiments to illustrate wording and tokenization sensitivity and bias elicitation in high-stakes education contexts, where LLMs demonstrate racial discrimination. Code and data for this study are available.