AdaDiff: Adaptive Step Selection for Fast Diffusion Models

TL;DR

AdaDiff is a novel adaptive framework that learns instance-specific denoising step policies for diffusion models, significantly reducing inference time while maintaining high visual quality across image and video generation tasks.

Contribution

We propose AdaDiff, a lightweight, policy-gradient optimized method for adaptive step selection in diffusion models, improving efficiency without sacrificing output quality.

Findings

Reduces inference time by 33-40% while maintaining quality

Allocates more steps to complex prompts and fewer to simple ones

Can be combined with other acceleration techniques

Abstract

Diffusion models, as a type of generative model, have achieved impressive results in generating images and videos conditioned on textual conditions. However, the generation process of diffusion models involves denoising dozens of steps to produce photorealistic images/videos, which is computationally expensive. Unlike previous methods that design ``one-size-fits-all'' approaches for speed up, we argue denoising steps should be sample-specific conditioned on the richness of input texts. To this end, we introduce AdaDiff, a lightweight framework designed to learn instance-specific step usage policies, which are then used by the diffusion model for generation. AdaDiff is optimized using a policy gradient method to maximize a carefully designed reward function, balancing inference time and generation quality. We conduct experiments on three image generation and two video generation benchmarks and demonstrate that our approach achieves similar visual quality compared to the baseline using a fixed 50 denoising steps while reducing inference time by at least 33%, going as high as 40%. Furthermore, our method can be used on top of other acceleration methods to provide further speed benefits. Lastly, qualitative analysis shows that AdaDiff allocates more steps to more informative prompts and fewer steps to simpler prompts.