Strong Model Collapse

TL;DR

This paper demonstrates that even minimal synthetic data can cause severe model collapse in large neural networks, and that increasing model size can both worsen and, beyond a threshold, mitigate this phenomenon.

Contribution

It establishes the existence of a strong form of model collapse due to synthetic data and analyzes how model size influences this effect through theoretical and empirical methods.

Findings

Small synthetic data fractions cause model collapse.

Larger models can amplify collapse in simplified regimes.

Beyond the interpolation threshold, larger models may reduce collapse.

Abstract

Within the scaling laws paradigm, which underpins the training of large neural networks like ChatGPT and Llama, we consider a supervised regression setting and establish the existance of a strong form of the model collapse phenomenon, a critical performance degradation due to synthetic data in the training corpus. Our results show that even the smallest fraction of synthetic data (e.g., as little as 1\% of the total training dataset) can still lead to model collapse: larger and larger training sets do not enhance performance. We further investigate whether increasing model size, an approach aligned with current trends in training large language models, exacerbates or mitigates model collapse. In a simplified regime where neural networks are approximated via random projections of tunable size, we both theoretically and empirically show that larger models can amplify model collapse. Interestingly, our theory also indicates that, beyond the interpolation threshold (which can be extremely high for very large datasets), larger models may mitigate the collapse, although they do not entirely prevent it. Our theoretical findings are empirically verified through experiments on language models and feed-forward neural networks for images.