What Capable Agents Must Know: Selection Theorems for Robust Decision-Making under Uncertainty

TL;DR

This paper establishes that robust decision-making under uncertainty inherently requires internal representations like world models and belief states, supported by quantitative selection theorems.

Contribution

It provides formal selection theorems demonstrating the necessity of world models and belief-like memory for strong task performance in uncertain environments.

Findings

Low average-case regret enforces the use of world models and belief-like memory.

Regret bounds limit probability on suboptimal bets, ensuring predictive distinctions.

In fully observed settings, approximate recovery of transition kernels is possible.

Abstract

As artificial agents become increasingly capable, what internal structure is *necessary* for an agent to act competently under uncertainty? Classical results show that optimal control can be *implemented* using belief states or world models, but not that such representations are required. We prove quantitative "selection theorems" showing that strong task performance (low *average-case regret*) forces world models, belief-like memory and -- under task mixtures -- persistent variables resembling core primitives associated with emotion, along with informational modularity under block-structured tasks. Our results cover stochastic policies, partial observability, and evaluation under task distributions, without assuming optimality, determinism, or access to an explicit model. Technically, we reduce predictive modeling to binary "betting" decisions and show that regret bounds limit probability mass on suboptimal bets, enforcing the predictive distinctions needed to separate high-margin outcomes. In fully observed settings, this yields approximate recovery of the interventional transition kernel; under partial observability, it implies necessity of predictive state and belief-like memory, addressing an open question in prior world-model recovery work.