Accelerating Diffusion Transformer via Increment-Calibrated Caching with Channel-Aware Singular Value Decomposition

TL;DR

This paper introduces an increment-calibrated caching method with channel-aware SVD to accelerate diffusion transformer models, reducing computation while maintaining high image quality without additional training.

Contribution

The proposed training-free acceleration method uses low-rank approximation and channel-aware SVD to improve caching effectiveness in diffusion transformers.

Findings

Achieves over 45% reduction in computation compared to 35-step DDIM.

Improves Inception Score (IS) by 12 points.

Maintains low FID increase of less than 0.06 while accelerating inference.

Abstract

Diffusion transformer (DiT) models have achieved remarkable success in image generation, thanks for their exceptional generative capabilities and scalability. Nonetheless, the iterative nature of diffusion models (DMs) results in high computation complexity, posing challenges for deployment. Although existing cache-based acceleration methods try to utilize the inherent temporal similarity to skip redundant computations of DiT, the lack of correction may induce potential quality degradation. In this paper, we propose increment-calibrated caching, a training-free method for DiT acceleration, where the calibration parameters are generated from the pre-trained model itself with low-rank approximation. To deal with the possible correction failure arising from outlier activations, we introduce channel-aware Singular Value Decomposition (SVD), which further strengthens the calibration effect. Experimental results show that our method always achieve better performance than existing naive caching methods with a similar computation resource budget. When compared with 35-step DDIM, our method eliminates more than 45% computation and improves IS by 12 at the cost of less than 0.06 FID increase. Code is available at https://github.com/ccccczzy/icc.