SIGMA: Scalable Spectral Insights for LLM Model Collapse

TL;DR

This paper introduces SIGMA, a spectral analysis framework that quantifies and predicts model collapse in large language models during recursive training, providing both theoretical insights and scalable monitoring tools.

Contribution

SIGMA offers a novel spectral-based method with bounds to detect and analyze model collapse, scalable to large models and recursive training scenarios.

Findings

SIGMA accurately detects the onset of model collapse.

The spectral bounds correlate with representational contraction.

Scalable estimation enables monitoring large models in practice.

Abstract

The rapid adoption of synthetic data for training Large Language Models (LLMs) has introduced the technical challenge of "model collapse"-a degenerative process where recursive training on model-generated content leads to a contraction of distributional variance and representational quality. While the phenomenology of collapse is increasingly evident, rigorous methods to quantify and predict its onset in high-dimensional spaces remain elusive. In this paper, we introduce SIGMA (Spectral Inequalities for Gram Matrix Analysis), a unified framework that benchmarks model collapse through the spectral lens of the embedding Gram matrix. By deriving and utilizing deterministic and stochastic bounds on the matrix's spectrum, SIGMA provides a mathematically grounded metric to track the contraction of the representation space. Crucially, our stochastic formulation enables scalable estimation of these bounds, making the framework applicable to large-scale foundation models where full eigendecomposition is intractable. We demonstrate that SIGMA effectively captures the transition towards degenerate states, offering both theoretical insights into the mechanics of collapse and a practical, scalable tool for monitoring the health of recursive training pipelines.