Training-Free Adaptive Diffusion with Bounded Difference Approximation Strategy

TL;DR

This paper introduces AdaptiveDiffusion, a method that adaptively skips noise prediction steps in diffusion models, significantly speeding up image and video synthesis without sacrificing quality.

Contribution

The paper presents a novel adaptive skipping strategy guided by latent differences, enabling faster diffusion without loss of output quality.

Findings

Achieves 2-5x speedup in denoising process

Maintains identical results to full-step diffusion

Applicable to both image and video diffusion models

Abstract

Diffusion models have recently achieved great success in the synthesis of high-quality images and videos. However, the existing denoising techniques in diffusion models are commonly based on step-by-step noise predictions, which suffers from high computation cost, resulting in a prohibitive latency for interactive applications. In this paper, we propose AdaptiveDiffusion to relieve this bottleneck by adaptively reducing the noise prediction steps during the denoising process. Our method considers the potential of skipping as many noise prediction steps as possible while keeping the final denoised results identical to the original full-step ones. Specifically, the skipping strategy is guided by the third-order latent difference that indicates the stability between timesteps during the denoising process, which benefits the reusing of previous noise prediction results. Extensive experiments on image and video diffusion models demonstrate that our method can significantly speed up the denoising process while generating identical results to the original process, achieving up to an average 2~5x speedup without quality degradation.