Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference

TL;DR

This paper proposes a novel method to accelerate diffusion models by analyzing encoder features and reusing them across time-steps, achieving significant speedups without distillation.

Contribution

It introduces a new approach that omits certain encoder computations and reuses features, enabling faster inference in diffusion models without additional training.

Findings

Achieves 41% speedup on Stable Diffusion

Maintains high-quality image generation

Applicable to multiple diffusion-based tasks

Abstract

One of the main drawback of diffusion models is the slow inference time for image generation. Among the most successful approaches to addressing this problem are distillation methods. However, these methods require considerable computational resources. In this paper, we take another approach to diffusion model acceleration. We conduct a comprehensive study of the UNet encoder and empirically analyze the encoder features. This provides insights regarding their changes during the inference process. In particular, we find that encoder features change minimally, whereas the decoder features exhibit substantial variations across different time-steps. This insight motivates us to omit encoder computation at certain adjacent time-steps and reuse encoder features of previous time-steps as input to the decoder in multiple time-steps. Importantly, this allows us to perform decoder computation in parallel, further accelerating the denoising process. Additionally, we introduce a prior noise injection method to improve the texture details in the generated image. Besides the standard text-to-image task, we also validate our approach on other tasks: text-to-video, personalized generation and reference-guided generation. Without utilizing any knowledge distillation technique, our approach accelerates both the Stable Diffusion (SD) and DeepFloyd-IF model sampling by 41$\%$ and 24$\%$ respectively, and DiT model sampling by 34$\%$, while maintaining high-quality generation performance.