When Models Don't Collapse: On the Consistency of Iterative MLE

TL;DR

This paper provides a theoretical analysis of model collapse in iterative generative modeling using maximum likelihood estimation, showing conditions under which collapse can be avoided or occurs rapidly.

Contribution

It offers the first rigorous theoretical bounds on model collapse in iterative MLE, clarifying when it can be prevented or is inevitable.

Findings

Collapse can be avoided with standard assumptions even as real data diminishes.

Certain assumptions are necessary; without them, collapse can happen quickly.

First rigorous examples of rapid collapse in iterative generative models.

Abstract

The widespread use of generative models has created a feedback loop, in which each generation of models is trained on data partially produced by its predecessors. This process has raised concerns about model collapse: A critical degradation in performance caused by repeated training on synthetic data. However, different analyses in the literature have reached different conclusions as to the severity of model collapse. As such, it remains unclear how concerning this phenomenon is, and under which assumptions it can be avoided. To address this, we theoretically study model collapse for maximum likelihood estimation (MLE), in a natural setting where synthetic data is gradually added to the original data set. Under standard assumptions (similar to those long used for proving asymptotic consistency and normality of MLE), we establish non-asymptotic bounds showing that collapse can be avoided even as the fraction of real data vanishes. On the other hand, we prove that some assumptions (beyond MLE consistency) are indeed necessary: Without them, model collapse can occur arbitrarily quickly, even when the original data is still present in the training set. To the best of our knowledge, these are the first rigorous examples of iterative generative modeling with accumulating data that rapidly leads to model collapse.