Curated Synthetic Data Doesn't Have to Collapse: A Theoretical Study of Generative Retraining with Pluralistic Preferences

TL;DR

This paper provides a theoretical analysis of how using multiple reward functions in recursive generative model retraining can prevent collapse and maintain output diversity.

Contribution

It formalizes the dynamics of multi-reward curation in generative retraining and proves convergence to a stable, diverse distribution satisfying a Nash bargaining solution.

Findings

Multi-reward curation mitigates collapse in recursive training.

Models converge to a stable distribution over diverse high-reward outputs.

The limiting distribution aligns with a weighted Nash bargaining solution.

Abstract

Recursive retraining of generative models poses a critical representation challenge: when synthetic outputs are curated based on a fixed reward signal, the model tends to collapse onto a narrow set of outputs that over-optimize that objective. Prior work suggests that such collapse is unavoidable without adding real data into the mix. We revisit this conclusion from an alignment perspective and show that collapse can be mitigated through curation based on multiple reward functions. We formalize the dynamics of recursive training under heterogeneous preferences and prove that, under certain conditions, the model converges to a stable distribution that allocates probability mass across competing high-reward regions. The limiting distribution preserves diversity and provably satisfies a weighted Nash bargaining solution, offering a formal interpretation of value aggregation in synthetic retraining loops.