Selective Guidance: Are All the Denoising Steps of Guided Diffusion Important?

TL;DR

This paper investigates the importance of each denoising step in guided diffusion models and proposes selective optimization techniques that significantly reduce inference time with minimal impact on image quality.

Contribution

It introduces a method to optimize only certain denoising steps in Stable Diffusion, reducing complexity and inference time without degrading image quality.

Findings

Optimizing the last 20% of denoising steps reduces inference time by 8.2%.

Extending optimization to 50% of steps reduces inference time by 20.3%.

Minimal perceptual difference observed in optimized outputs.

Abstract

This study examines the impact of optimizing the Stable Diffusion (SD) guided inference pipeline. We propose optimizing certain denoising steps by limiting the noise computation to conditional noise and eliminating unconditional noise computation, thereby reducing the complexity of the target iterations by 50%. Additionally, we demonstrate that later iterations of the SD are less sensitive to optimization, making them ideal candidates for applying the suggested optimization. Our experiments show that optimizing the last 20% of the denoising loop iterations results in an 8.2% reduction in inference time with almost no perceivable changes to the human eye. Furthermore, we found that by extending the optimization to 50% of the last iterations, we can reduce inference time by approximately 20.3%, while still generating visually pleasing images.