Stabilizing Self-Consuming Diffusion Models with Latent Space Filtering

TL;DR

This paper introduces Latent Space Filtering, a method to stabilize self-consuming diffusion models by filtering synthetic data in latent space, preventing model collapse without extra costs or human input.

Contribution

The paper presents a novel latent space filtering technique that mitigates model collapse in self-consuming diffusion models, supported by theoretical analysis and empirical validation.

Findings

LSF effectively prevents model collapse across multiple datasets.

It outperforms existing methods without additional training costs.

Latent space degradation correlates with model instability.

Abstract

As synthetic data proliferates across the Internet, it is often reused to train successive generations of generative models. This creates a ``self-consuming loop" that can lead to training instability or \textit{model collapse}. Common strategies to address the issue -- such as accumulating historical training data or injecting fresh real data -- either increase computational cost or require expensive human annotation. In this paper, we empirically analyze the latent space dynamics of self-consuming diffusion models and observe that the low-dimensional structure of latent representations extracted from synthetic data degrade over generations. Based on this insight, we propose \textit{Latent Space Filtering} (LSF), a novel approach that mitigates model collapse by filtering out less realistic synthetic data from mixed datasets. Theoretically, we present a framework that connects latent space degradation to empirical observations. Experimentally, we show that LSF consistently outperforms existing baselines across multiple real-world datasets, effectively mitigating model collapse without increasing training cost or relying on human annotation.