Model Collapse Demystified: The Case of Regression

TL;DR

This paper investigates the phenomenon of model collapse in high-dimensional regression, providing analytical insights, modified scaling laws, and proposing an adaptive regularization strategy to mitigate collapse, validated through experiments.

Contribution

It offers the first analytical framework for understanding model collapse in regression and introduces a mitigation strategy based on adaptive regularization.

Findings

Analytic formulae describing model collapse in regression

Modified scaling laws with crossover phenomena

Effective adaptive regularization strategy

Abstract

In the era of proliferation of large language and image generation models, the phenomenon of "model collapse" refers to the situation whereby as a model is trained recursively on data generated from previous generations of itself over time, its performance degrades until the model eventually becomes completely useless, i.e the model collapses. In this work, we study this phenomenon in the setting of high-dimensional regression and obtain analytic formulae which quantitatively outline this phenomenon in a broad range of regimes. In the special case of polynomial decaying spectral and source conditions, we obtain modified scaling laws which exhibit new crossover phenomena from fast to slow rates. We also propose a simple strategy based on adaptive regularization to mitigate model collapse. Our theoretical results are validated with experiments.