Accelerating Diffusion Models with One-to-Many Knowledge Distillation

TL;DR

This paper introduces one-to-many knowledge distillation (O2MKD), a novel method to accelerate diffusion models by training multiple student models on different timestep subsets, significantly reducing computational costs for real-time image generation.

Contribution

The paper proposes a new distillation approach that trains multiple student diffusion models on different timestep subsets, enhancing speed without sacrificing quality.

Findings

O2MKD achieves significant acceleration on multiple datasets.

The method improves efficiency of existing knowledge distillation and sampling techniques.

Experimental results demonstrate effective speedup with maintained image quality.

Abstract

Significant advancements in image generation have been made with diffusion models. Nevertheless, when contrasted with previous generative models, diffusion models face substantial computational overhead, leading to failure in real-time generation. Recent approaches have aimed to accelerate diffusion models by reducing the number of sampling steps through improved sampling techniques or step distillation. However, the methods to diminish the computational cost for each timestep remain a relatively unexplored area. Observing the fact that diffusion models exhibit varying input distributions and feature distributions at different timesteps, we introduce one-to-many knowledge distillation (O2MKD), which distills a single teacher diffusion model into multiple student diffusion models, where each student diffusion model is trained to learn the teacher's knowledge for a subset of continuous timesteps. Experiments on CIFAR10, LSUN Church, CelebA-HQ with DDPM and COCO30K with Stable Diffusion show that O2MKD can be applied to previous knowledge distillation and fast sampling methods to achieve significant acceleration. Codes will be released in Github.