Language Generation with Replay: A Learning-Theoretic View of Model Collapse

TL;DR

This paper provides a theoretical analysis of how replay of generated content can lead to model collapse in large language models, revealing conditions under which it limits or does not limit model performance.

Contribution

It introduces a learning-theoretic framework with a replay adversary to analyze the impact of reusing generated data on language model training, highlighting when replay causes fundamental limitations.

Findings

Replay limits model generation under weaker notions but not under uniform generation.

Theoretical results align with practical heuristics like data cleaning and watermarking.

Identifies scenarios where output filtering fails to prevent model collapse.

Abstract

As scaling laws push the training of frontier large language models (LLMs) toward ever-growing data requirements, training pipelines are approaching a regime where much of the publicly available online text may be consumed. At the same time, widespread LLM usage increases the volume of machine-generated content on the web; together, these trends raise the likelihood of generated text re-entering future training corpora, increasing the associated risk of performance degradation often called model collapse. In practice, model developers address this concern through data cleaning, watermarking, synthetic-data policies, or, in some cases, blissful ignorance. However, the problem of model collapse in generative models has not been examined from a learning-theoretic perspective: we study it through the theoretical lens of the language generation in the limit framework, introducing a replay adversary that augments the example stream with the generator's own past outputs. Our main contribution is a fine-grained learning-theoretic characterization of when replay fundamentally limits generation: while replay is benign for the strongest notion of uniform generation, it provably creates separations for the weaker notions of non-uniform generation and generation in the limit. Interestingly, our positive results mirror heuristics widely used in practice, such as data cleaning, watermarking, and output filtering, while our separations show when these ideas can fail.