Revisiting Diffusion Models: From Generative Pre-training to One-Step Generation

TL;DR

This paper reveals that diffusion training acts as a form of generative pre-training, and demonstrates that a standalone GAN objective can convert diffusion models into efficient one-step generators with minimal fine-tuning.

Contribution

It identifies the limitations of diffusion distillation, proposes using GAN objectives for one-step generation, and shows how diffusion training can be leveraged as a pre-training method for efficient models.

Findings

GAN objective overcomes distillation limitations

One-step models achieve near-SOTA results with minimal data

Diffusion training serves as effective generative pre-training

Abstract

Diffusion distillation is a widely used technique to reduce the sampling cost of diffusion models, yet it often requires extensive training, and the student performance tends to be degraded. Recent studies show that incorporating a GAN objective may alleviate these issues, yet the underlying mechanism remains unclear. In this work, we first identify a key limitation of distillation: mismatched step sizes and parameter numbers between the teacher and the student model lead them to converge to different local minima, rendering direct imitation suboptimal. We further demonstrate that a standalone GAN objective, without relying a distillation loss, overcomes this limitation and is sufficient to convert diffusion models into efficient one-step generators. Based on this finding, we propose that diffusion training may be viewed as a form of generative pre-training, equipping models with capabilities that can be unlocked through lightweight GAN fine-tuning. Supporting this view, we create a one-step generation model by fine-tuning a pre-trained model with 85% of parameters frozen, achieving strong performance with only 0.2M images and near-SOTA results with 5M images. We further present a frequency-domain analysis that may explain the one-step generative capability gained in diffusion training. Overall, our work provides a new perspective for diffusion training, highlighting its role as a powerful generative pre-training process, which can be the basis for building efficient one-step generation models.