AdaDiff: Accelerating Diffusion Models through Step-Wise Adaptive Computation

TL;DR

AdaDiff introduces a dynamic adaptive computation framework for diffusion models, significantly speeding up image generation by evaluating uncertainty at each step to allocate resources efficiently and decide when to terminate inference.

Contribution

The paper presents a novel adaptive framework with a timestep-aware uncertainty estimation module and an uncertainty-aware loss to enhance diffusion model efficiency.

Findings

Achieves faster image generation with minimal quality loss.

Effectively reduces unnecessary computation during sampling.

Maintains high fidelity in generated images despite adaptive inference.

Abstract

Diffusion models achieve great success in generating diverse and high-fidelity images, yet their widespread application, especially in real-time scenarios, is hampered by their inherently slow generation speed. The slow generation stems from the necessity of multi-step network inference. While some certain predictions benefit from the full computation of the model in each sampling iteration, not every iteration requires the same amount of computation, potentially leading to inefficient computation. Unlike typical adaptive computation challenges that deal with single-step generation problems, diffusion processes with a multi-step generation need to dynamically adjust their computational resource allocation based on the ongoing assessment of each step's importance to the final image output, presenting a unique set of challenges. In this work, we propose AdaDiff, an adaptive framework that dynamically allocates computation resources in each sampling step to improve the generation efficiency of diffusion models. To assess the effects of changes in computational effort on image quality, we present a timestep-aware uncertainty estimation module (UEM). Integrated at each intermediate layer, the UEM evaluates the predictive uncertainty. This uncertainty measurement serves as an indicator for determining whether to terminate the inference process. Additionally, we introduce an uncertainty-aware layer-wise loss aimed at bridging the performance gap between full models and their adaptive counterparts.