A Structural Threshold in Decision Capacity Governs Collapse in Self-Play Reinforcement Learning

TL;DR

This paper identifies a decision capacity threshold that determines whether self-play reinforcement learning agents collapse under perturbations, with the collapse being reversible and dependent on the preservation of certain decision points.

Contribution

It reveals a sharp threshold in decision capacity that governs collapse in self-play RL, supported by experiments across multiple games and algorithms.

Findings

Eliminating positive-reach decisions causes agents to collapse into a deterministic fixed point.

Preserving even one positive-reach decision prevents collapse.

Collapse severity scales with reach-weighted capacity.

Abstract

We show that a threshold in decision capacity determines whether self-play reinforcement learning agents collapse under asymmetric rule perturbations. Across poker variants, matrix games, a dice game, and multiple learning algorithms, eliminating all positive-reach contingent decisions causes rapid convergence to a deterministic exploitation attractor, a fixed point at near-maximal loss. Preserving even a single positive-reach contingent decision point prevents this collapse. A frozen baseline and fixed-opponent control confirm that the mechanism is co-adaptation under constraint, not the perturbation itself. The phenomenon is timing-invariant, fully reversible upon action restoration, and intensifies under function approximation. These results establish a sharp threshold at zero reach-weighted contingent action capacity, with severity scaling continuously via reach-weighted capacity in the tested domains.