Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking

TL;DR

This paper reveals that standard confidence intervals in LLM evaluation underestimate true variability, leading to unreliable benchmarks, and proposes methods to account for this hidden measurement error to improve evaluation accuracy.

Contribution

It decomposes sources of uncertainty in LLM pipelines, distinguishes variance types, and introduces a design-study approach to reduce total evaluation error and improve benchmarking reliability.

Findings

Naive standard errors are 40-60% smaller than TEE-corrected SE.

TEE-corrected coverage remains at 95% as sample size grows, unlike naive CIs.

Design-guided pipelines reduce benchmark gaming and improve estimation accuracy.

Abstract

LLM evaluations drive which models get deployed, what safety standards get adopted, which research conclusions get published, and how projections of AI's labor-market impact get made. Yet standard confidence intervals ignore variability from judge model choice, model temperature, and prompt phrasing, producing under-coverage that worsens with more data. The omitted variance can shift results enough to reverse conclusions \citep{baumann2025llmhacking, huang2026dropping}; pipelines that fail to average over it leave the surface that ``benchmark hacking'' exploits \citep{singh2025leaderboard}. This paper decomposes LLM pipeline uncertainty into its sources, distinguishes variance that shrinks with more data from sensitivity to researcher design choices, and uses design-study projections to reduce total evaluation error (TEE). Across the demonstrations, naive standard errors are 40 - 60\% smaller than the TEE-corrected SE. Using Chatbot Arena data, we show naive 95\% CI coverage drops as $n$ grows while TEE-corrected coverage holds at 95\%, and TEE-guided pipelines restrict the benchmark gaming surface from 56 to 32 Elo ($K=27$), below the human-leaderboard baseline. We show further that a small pilot recovers honest CIs and projects which design changes most improve precision. Acting on those projections halves MMLU estimation error against the answer key at equivalent cost, and raises per-match agreement with human votes by 7.9 percentage points on Chatbot Arena.