A Survey on Pre-Trained Diffusion Model Distillations

TL;DR

This survey reviews various distillation techniques for pre-trained diffusion models, focusing on output, trajectory, and adversarial methods, to enhance efficiency and reduce computational costs in generative AI.

Contribution

It provides a systematic overview of distillation methods for diffusion models from a methodological perspective, highlighting challenges and future directions.

Findings

Distillation methods improve model efficiency and reduce generation steps.

Three main distillation approaches are identified: output loss, trajectory, and adversarial.

The survey outlines current challenges and potential research avenues.

Abstract

Diffusion Models~(DMs) have emerged as the dominant approach in Generative Artificial Intelligence (GenAI), owing to their remarkable performance in tasks such as text-to-image synthesis. However, practical DMs, such as stable diffusion, are typically trained on massive datasets and thus usually require large storage. At the same time, many steps may be required, i.e., recursively evaluating the trained neural network, to generate a high-quality image, which results in significant computational costs during sample generation. As a result, distillation methods on pre-trained DM have become widely adopted practices to develop smaller, more efficient models capable of rapid, few-step generation in low-resource environment. When these distillation methods are developed from different perspectives, there is an urgent need for a systematic survey, particularly from a methodological perspective. In this survey, we review distillation methods through three aspects: output loss distillation, trajectory distillation and adversarial distillation. We also discuss current challenges and outline future research directions in the conclusion.