Epistemic Traps: Rational Misalignment Driven by Model Misspecification

TL;DR

This paper introduces a theoretical framework explaining persistent AI safety failures as rational behaviors caused by model misspecification, emphasizing the importance of designing agents' internal beliefs for robust alignment.

Contribution

It adapts Berk-Nash Rationalizability to AI, providing a formal model for understanding and predicting safety failures as stable or oscillatory equilibria.

Findings

Safety failures are structural, not transient.

Behavioral phase diagrams map safe and unsafe regions.

Safety depends on the agent's epistemic priors, not just rewards.

Abstract

The rapid deployment of Large Language Models and AI agents across critical societal and technical domains is hindered by persistent behavioral pathologies including sycophancy, hallucination, and strategic deception that resist mitigation via reinforcement learning. Current safety paradigms treat these failures as transient training artifacts, lacking a unified theoretical framework to explain their emergence and stability. Here we show that these misalignments are not errors, but mathematically rationalizable behaviors arising from model misspecification. By adapting Berk-Nash Rationalizability from theoretical economics to artificial intelligence, we derive a rigorous framework that models the agent as optimizing against a flawed subjective world model. We demonstrate that widely observed failures are structural necessities: unsafe behaviors emerge as either a stable misaligned equilibrium or oscillatory cycles depending on reward scheme, while strategic deception persists as a "locked-in" equilibrium or through epistemic indeterminacy robust to objective risks. We validate these theoretical predictions through behavioral experiments on six state-of-the-art model families, generating phase diagrams that precisely map the topological boundaries of safe behavior. Our findings reveal that safety is a discrete phase determined by the agent's epistemic priors rather than a continuous function of reward magnitude. This establishes Subjective Model Engineering, defined as the design of an agent's internal belief structure, as a necessary condition for robust alignment, marking a paradigm shift from manipulating environmental rewards to shaping the agent's interpretation of reality.