Universality of the $\pi^2/6$ Pathway in Avoiding Model Collapse

TL;DR

This paper investigates the universal behavior of the $rac{}{6}$ risk bound in avoiding model collapse across various statistical models, explaining why augment workflows prevent collapse while discard workflows do not.

Contribution

It demonstrates the universality of the $rac{}{6}$ risk bound in augment workflows across many models, extending previous specific results to a broad class of statistical models.

Findings

The $rac{}{6}$ risk bound applies broadly to many models in augment workflows.

Discard workflows tend to lead to model collapse, while augment workflows prevent it.

A flexible framework is provided to compare different workflows via Gaussian process simulations.

Abstract

Researchers in empirical machine learning recently spotlighted their fears of so-called Model Collapse. They imagined a discard workflow, where an initial generative model is trained with real data, after which the real data are discarded, and subsequently, the model generates synthetic data on which a new model is trained. They came to the conclusion that models degenerate as model-fitting generations proceed. However, other researchers considered an augment workflow, where the original real data continue to be used in each generation of training, augmented by synthetic data from models fit in all earlier generations. Empirical results on canonical datasets and learning procedures confirmed the occurrence of model collapse under the discard workflow and avoidance of model collapse under the augment workflow. Under the augment workflow, theoretical evidence also confirmed avoidance in particular instances; specifically, Gerstgrasser et al. (2024) found that for classical Linear Regression, test risk at any later generation is bounded by a moderate multiple, viz. pi-squared-over-6 of the test risk of training with the original real data alone. Some commentators questioned the generality of theoretical conclusions based on the generative model assumed in Gerstgrasser et al. (2024): could similar conclusions be reached for other task/model pairings? In this work, we demonstrate the universality of the pi-squared-over-6 augment risk bound across a large family of canonical statistical models, offering key insights into exactly why collapse happens under the discard workflow and is avoided under the augment workflow. In the process, we provide a framework that is able to accommodate a large variety of workflows (beyond discard and augment), thereby enabling an experimenter to judge the comparative merits of multiple different workflows by simulating a simple Gaussian process.