The Evaluation Trap: Benchmark Design as Theoretical Commitment

TL;DR

This paper critiques current AI benchmarks for reinforcing existing paradigms by unexamined assumptions, introducing Epistematics as a meta-evaluation methodology to ensure benchmarks genuinely measure claimed capabilities.

Contribution

It presents a novel meta-evaluation approach, including an audit procedure, failure mode taxonomy, and design criteria to assess the coherence of capability evaluations.

Findings

Identified how benchmarks can entrench dominant paradigms.

Demonstrated the audit procedure on a specific benchmark proposal.

Showed that some benchmarks reproduce the assumptions they aim to challenge.

Abstract

Every AI benchmark operationalizes theoretical assumptions about the capability it claims to assess. When assumptions function as unexamined commitments, benchmarks stabilize the dominant paradigm by narrowing what counts as progress. Over time, narrow evaluation reorganizes capability concepts: architectures and definitions are selected for benchmark legibility until evaluation ceases to track an independent object and instead produces a version of the target defined by its own operational assumptions. The result is a trap: evaluation frameworks treat self-reinforcing assessments as valid, both creating and obscuring structural limits on what the current paradigm can accomplish. We introduce Epistematics, a methodology for deriving evaluation criteria directly from technical capability claims and auditing whether proposed benchmarks can discriminate the claimed capability from proxy behaviors. The contribution is meta-evaluative: an audit procedure, a failure mode taxonomy, and benchmark-design criteria for evaluating capability-evaluation coherence. We demonstrate the procedure through a worked audit of Dupoux et al. (2026), a proposal that revises the dominant paradigm's theoretical assumptions at the architectural level while reproducing them in its evaluation criteria, thereby entrenching the constraint it seeks to overcome in a form the evaluation cannot detect.