SADA: Stability-guided Adaptive Diffusion Acceleration

TL;DR

SADA introduces a stability-guided adaptive approach to accelerate diffusion model sampling, achieving over 1.8x speedups with minimal quality loss by leveraging a unified stability criterion for adaptive sparsity and principled ODE-based approximations.

Contribution

It proposes a novel stability-guided adaptive acceleration framework for ODE-based diffusion models, unifying step-wise and token-wise sparsity decisions for improved speed and fidelity.

Findings

Achieves ≥1.8× speedup with minimal fidelity loss.

Outperforms prior acceleration methods.

Seamlessly adapts to different pipelines and modalities.

Abstract

Diffusion models have achieved remarkable success in generative tasks but suffer from high computational costs due to their iterative sampling process and quadratic attention costs. Existing training-free acceleration strategies that reduce per-step computation cost, while effectively reducing sampling time, demonstrate low faithfulness compared to the original baseline. We hypothesize that this fidelity gap arises because (a) different prompts correspond to varying denoising trajectory, and (b) such methods do not consider the underlying ODE formulation and its numerical solution. In this paper, we propose Stability-guided Adaptive Diffusion Acceleration (SADA), a novel paradigm that unifies step-wise and token-wise sparsity decisions via a single stability criterion to accelerate sampling of ODE-based generative models (Diffusion and Flow-matching). For (a), SADA adaptively allocates sparsity based on the sampling trajectory. For (b), SADA introduces principled approximation schemes that leverage the precise gradient information from the numerical ODE solver. Comprehensive evaluations on SD-2, SDXL, and Flux using both EDM and DPM++ solvers reveal consistent $\ge 1.8\times$ speedups with minimal fidelity degradation (LPIPS $\leq 0.10$ and FID $\leq 4.5$) compared to unmodified baselines, significantly outperforming prior methods. Moreover, SADA adapts seamlessly to other pipelines and modalities: It accelerates ControlNet without any modifications and speeds up MusicLDM by $1.8\times$ with $\sim 0.01$ spectrogram LPIPS.