SODA: Sensitivity-Oriented Dynamic Acceleration for Diffusion Transformer

TL;DR

SODA is a novel method that adaptively accelerates diffusion transformers by dynamically balancing caching and pruning based on fine-grained sensitivity analysis, significantly improving inference efficiency without sacrificing quality.

Contribution

SODA introduces a sensitivity-oriented framework that adaptively configures caching and pruning, overcoming limitations of fixed strategies and enhancing diffusion transformer acceleration.

Findings

Achieves state-of-the-art fidelity at high acceleration ratios.

Effectively balances acceleration and quality through sensitivity-aware optimization.

Demonstrates superior performance on multiple diffusion models.

Abstract

Diffusion Transformers have become a dominant paradigm in visual generation, yet their low inference efficiency remains a key bottleneck hindering further advancement. Among common training-free techniques, caching offers high acceleration efficiency but often compromises fidelity, whereas pruning shows the opposite trade-off. Integrating caching with pruning achieves a balance between acceleration and generation quality. However, existing methods typically employ fixed and heuristic schemes to configure caching and pruning strategies. While they roughly follow the overall sensitivity trend of generation models to acceleration, they fail to capture fine-grained and complex variations, inevitably skipping highly sensitive computations and leading to quality degradation. Furthermore, such manually designed strategies exhibit poor generalization. To address these issues, we propose SODA, a Sensitivity-Oriented Dynamic Acceleration method that adaptively performs caching and pruning based on fine-grained sensitivity. SODA builds an offline sensitivity error modeling framework across timesteps, layers, and modules to capture the sensitivity to different acceleration operations. The cache intervals are optimized via dynamic programming with sensitivity error as the cost function, minimizing the impact of caching on model sensitivity. During pruning and cache reuse, SODA adaptively determines the pruning timing and rate to preserve computations of highly sensitive tokens, significantly enhancing generation fidelity. Extensive experiments on DiT-XL/2, PixArt-$\alpha$, and OpenSora demonstrate that SODA achieves state-of-the-art generation fidelity under controllable acceleration ratios. Our code is released publicly at: https://github.com/leaves162/SODA.