Simple and Fast Distillation of Diffusion Models

TL;DR

This paper introduces Simple and Fast Distillation (SFD), a method that significantly reduces fine-tuning time for diffusion models, enabling efficient high-quality image synthesis with variable NFEs from a single model.

Contribution

SFD simplifies existing distillation methods, reduces fine-tuning time up to 1000 times, and allows variable NFE sampling with a single distilled model.

Findings

Achieves 4.53 FID with NFE=2 on CIFAR-10

Reduces fine-tuning time to 0.64 hours on a single GPU

Balances sample quality and fine-tuning costs effectively

Abstract

Diffusion-based generative models have demonstrated their powerful performance across various tasks, but this comes at a cost of the slow sampling speed. To achieve both efficient and high-quality synthesis, various distillation-based accelerated sampling methods have been developed recently. However, they generally require time-consuming fine tuning with elaborate designs to achieve satisfactory performance in a specific number of function evaluation (NFE), making them difficult to employ in practice. To address this issue, we propose Simple and Fast Distillation (SFD) of diffusion models, which simplifies the paradigm used in existing methods and largely shortens their fine-tuning time up to 1000$\times$. We begin with a vanilla distillation-based sampling method and boost its performance to state of the art by identifying and addressing several small yet vital factors affecting the synthesis efficiency and quality. Our method can also achieve sampling with variable NFEs using a single distilled model. Extensive experiments demonstrate that SFD strikes a good balance between the sample quality and fine-tuning costs in few-step image generation task. For example, SFD achieves 4.53 FID (NFE=2) on CIFAR-10 with only 0.64 hours of fine-tuning on a single NVIDIA A100 GPU. Our code is available at https://github.com/zju-pi/diff-sampler.