The Alignment Game: A Theory of Long-Horizon Alignment Through Recursive Curation

TL;DR

This paper develops a formal theory of long-term alignment in recursive generative models, analyzing how iterative curation influences model preferences and diversity over time.

Contribution

It introduces a formal framework for understanding recursive alignment, revealing structural regimes and fundamental limitations of BT-based curation mechanisms.

Findings

Identifies three convergence regimes: consensus collapse, shared optima, asymmetric refinement.

Proves an impossibility theorem: no BT-based mechanism can preserve diversity, symmetry, and independence simultaneously.

Shows alignment as an evolving social choice influenced by power and history.

Abstract

In self-consuming generative models that train on their own outputs, alignment with user preferences becomes a recursive rather than one-time process. We provide the first formal foundation for analyzing the long-term effects of such recursive retraining on alignment. Under a two-stage curation mechanism based on the Bradley-Terry (BT) model, we model alignment as an interaction between two factions: the Model Owner, who filters which outputs should be learned by the model, and the Public User, who determines which outputs are ultimately shared and retained through interactions with the model. Our analysis reveals three structural convergence regimes depending on the degree of preference alignment: consensus collapse, compromise on shared optima, and asymmetric refinement. We prove a fundamental impossibility theorem: no recursive BT-based curation mechanism can simultaneously preserve diversity, ensure symmetric influence, and eliminate dependence on initialization. Framing the process as dynamic social choice, we show that alignment is not a static goal but an evolving equilibrium, shaped both by power asymmetries and path dependence.