A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops

TL;DR

This paper provides a theoretical analysis of Self-consuming Training Loops (STLs) in generative models, explaining when they succeed or fail based on model architecture and data proportions, and offers insights into preventing model collapse.

Contribution

It introduces the concept of recursive stability and offers the first theoretical generalization analysis of STLs, linking success to architecture and data ratios.

Findings

Model architecture influences STL stability.

A constant proportion of real data guarantees convergence.

Insights into optimal synthetic data sizing for training.

Abstract

High-quality data is essential for training large generative models, yet the vast reservoir of real data available online has become nearly depleted. Consequently, models increasingly generate their own data for further training, forming Self-consuming Training Loops (STLs). However, the empirical results have been strikingly inconsistent: some models degrade or even collapse, while others successfully avoid these failures, leaving a significant gap in theoretical understanding to explain this discrepancy. This paper introduces the intriguing notion of recursive stability and presents the first theoretical generalization analysis, revealing how both model architecture and the proportion between real and synthetic data influence the success of STLs. We further extend this analysis to transformers in in-context learning, showing that even a constant-sized proportion of real data ensures convergence, while also providing insights into optimal synthetic data sizing.