Benchmark Illusion: Disagreement among LLMs and Its Scientific Consequences

TL;DR

This paper reveals that similar benchmark accuracy among large language models can mask significant disagreements, which can critically impact scientific research and reproducibility.

Contribution

It uncovers the 'benchmark illusion' where comparable accuracy conceals epistemic divergence among LLMs, affecting scientific data annotation and inference.

Findings

LLMs disagree on 16-66% of benchmark items.

Switching models can change estimated effects by over 80%.

Model choice influences scientific reproducibility.

Abstract

Benchmarks underpin how progress in large language models (LLMs) is measured and trusted. Yet our analyses reveal that apparent convergence in benchmark accuracy can conceal deep epistemic divergence. Using two major reasoning benchmarks - MMLU-Pro and GPQA - we show that LLMs achieving comparable accuracy still disagree on 16-66% of items, and 16-38% among top-performing frontier models. These discrepancies suggest distinct error profiles for different LLMs. When such models are used for scientific data annotation and inference, their hidden disagreements propagate into research results: in re-analyses of published studies in education and political science, switching the annotation model can change estimated treatment effects by more than 80%, and in some cases reverses their sign. Together, these findings illustrate a benchmark illusion, where equal accuracy may conceal disagreement, with model choice becoming a hidden yet consequential variable for scientific reproducibility.