Model Collapse in the Self-Consuming Chain of Diffusion Finetuning: A Novel Perspective from Quantitative Trait Modeling

TL;DR

This paper investigates model collapse in self-finetuned diffusion models, introduces a genetic-inspired theoretical framework, and proposes ReDiFine, a robust finetuning strategy that prevents quality degradation.

Contribution

It offers a novel quantitative trait modeling approach to analyze model collapse and introduces ReDiFine, a hyperparameter-free finetuning method inspired by genetic mutations.

Findings

Model collapse is universal in Chain of Diffusion finetuning.

CFG scale significantly impacts model degradation.

ReDiFine effectively prevents collapse without hyperparameter tuning.

Abstract

Model collapse, the severe degradation of generative models when iteratively trained on their own outputs, has gained significant attention in recent years. This paper examines Chain of Diffusion, where a pretrained text-to-image diffusion model is finetuned on its own generated images. We demonstrate that severe image quality degradation was universal and identify CFG scale as the key factor impacting this model collapse. Drawing on an analogy between the Chain of Diffusion and biological evolution, we then introduce a novel theoretical analysis based on quantitative trait modeling from statistical genetics. Our theoretical analysis aligns with empirical observations of the generated images in the Chain of Diffusion. Finally, we propose Reusable Diffusion Finetuning (ReDiFine), a simple yet effective strategy inspired by genetic mutations. It operates robustly across various scenarios without requiring any hyperparameter tuning, making it a plug-and-play solution for reusable image generation.