Dual-Rate Diffusion: Accelerating diffusion models with an interleaved heavy-light network

TL;DR

Dual-Rate Diffusion accelerates diffusion model sampling by interleaving a heavy context encoder with a lightweight denoising network, reducing computation while maintaining quality.

Contribution

It introduces a novel interleaved heavy-light network approach to significantly speed up diffusion sampling without quality loss.

Findings

Reduces computational cost by 2-4 times on ImageNet benchmarks.

Maintains sample quality comparable to standard diffusion models.

Compatible with distillation techniques for further efficiency.

Abstract

Diffusion models achieve state-of-the-art generative performance but suffer from high computational costs during inference due to the repeated evaluation of a heavy neural network. In this work, we propose Dual-Rate Diffusion, a method to accelerate sampling by interleaving the execution of a heavy high-capacity context encoder and a light efficient denoising model. The context encoder is evaluated sparsely to extract high-dimensional features, which are effectively reused by the light denoising model at every step to refine the sample efficiently. This approach significantly accelerates inference without compromising sample quality. On ImageNet benchmarks, Dual-Rate Diffusion matches the performance of standard baselines while reducing computational cost by a factor of $2$-$4$. Furthermore, we demonstrate that our method is compatible with distillation techniques, such as Moment Matching Distillation, enabling further efficiency gains in few-step generation.