Accelerating Diffusion Sampling with Classifier-based Feature Distillation

TL;DR

This paper introduces Classifier-based Feature Distillation (CFD), a method to accelerate diffusion sampling by distilling important features using a classifier, significantly improving speed and quality on CIFAR-10.

Contribution

The paper proposes CFD, a novel feature distillation approach that enhances diffusion model sampling speed and quality, especially for few-step samplers, surpassing traditional progressive distillation.

Findings

CFD achieves higher quality images with fewer sampling steps.

Experimental results on CIFAR-10 demonstrate superior performance over existing methods.

The method effectively distills important features, leading to faster and better sampling.

Abstract

Although diffusion model has shown great potential for generating higher quality images than GANs, slow sampling speed hinders its wide application in practice. Progressive distillation is thus proposed for fast sampling by progressively aligning output images of $N$-step teacher sampler with $N/2$-step student sampler. In this paper, we argue that this distillation-based accelerating method can be further improved, especially for few-step samplers, with our proposed \textbf{C}lassifier-based \textbf{F}eature \textbf{D}istillation (CFD). Instead of aligning output images, we distill teacher's sharpened feature distribution into the student with a dataset-independent classifier, making the student focus on those important features to improve performance. We also introduce a dataset-oriented loss to further optimize the model. Experiments on CIFAR-10 show the superiority of our method in achieving high quality and fast sampling. Code is provided at \url{https://github.com/zju-SWJ/RCFD}.