Reward Hacking as Equilibrium under Finite Evaluation

TL;DR

This paper demonstrates that reward hacking is an inherent equilibrium in AI systems under certain axioms, predicting its severity and direction before deployment, and unifies various gaming phenomena under a single theoretical framework.

Contribution

It introduces a structural equilibrium perspective on reward hacking, providing a computable distortion index and an evaluation coverage analysis in AI alignment.

Findings

Reward hacking is a structural equilibrium, not a bug.

Evaluation coverage declines as tool count increases, amplifying hacking.

Unified explanation for sycophancy, length gaming, and specification gaming.

Abstract

We prove that under five minimal axioms -- multi-dimensional quality, finite evaluation, effective optimization, resource finiteness, and combinatorial interaction -- any optimized AI agent will systematically under-invest effort in quality dimensions not covered by its evaluation system. This result establishes reward hacking as a structural equilibrium, not a correctable bug, and holds regardless of the specific alignment method (RLHF, DPO, Constitutional AI, or others) or evaluation architecture employed. Our framework instantiates the multi-task principal-agent model of Holmstrom and Milgrom (1991) in the AI alignment setting, but exploits a structural feature unique to AI systems -- the known, differentiable architecture of reward models -- to derive a computable distortion index that predicts both the direction and severity of hacking on each quality dimension prior to deployment. We further prove that the transition from closed reasoning to agentic systems causes evaluation coverage to decline toward zero as tool count grows -- because quality dimensions expand combinatorially while evaluation costs grow at most linearly per tool -- so that hacking severity increases structurally and without bound. Our results unify the explanation of sycophancy, length gaming, and specification gaming under a single theoretical structure and yield an actionable vulnerability assessment procedure. We further conjecture -- with partial formal analysis -- the existence of a capability threshold beyond which agents transition from gaming within the evaluation system (Goodhart regime) to actively degrading the evaluation system itself (Campbell regime), providing the first economic formalization of Bostrom's (2014) "treacherous turn."