Sortblock: Similarity-Aware Feature Reuse for Diffusion Model

TL;DR

Sortblock is a training-free method that accelerates diffusion models by adaptively caching and skipping redundant features based on their similarity, doubling inference speed with minimal quality loss.

Contribution

It introduces a novel similarity-aware caching framework that dynamically determines which features to reuse or recompute during diffusion model inference.

Findings

Over 2x inference speedup achieved.

Minimal degradation in generated output quality.

Applicable across various diffusion transformer architectures.

Abstract

Diffusion Transformers (DiTs) have demonstrated remarkable generative capabilities, particularly benefiting from Transformer architectures that enhance visual and artistic fidelity. However, their inherently sequential denoising process results in high inference latency, limiting their deployment in real-time scenarios. Existing training-free acceleration approaches typically reuse intermediate features at fixed timesteps or layers, overlooking the evolving semantic focus across denoising stages and Transformer blocks.To address this, we propose Sortblock, a training-free inference acceleration framework that dynamically caches block-wise features based on their similarity across adjacent timesteps. By ranking the evolution of residuals, Sortblock adaptively determines a recomputation ratio, selectively skipping redundant computations while preserving generation quality. Furthermore, we incorporate a lightweight linear prediction mechanism to reduce accumulated errors in skipped blocks.Extensive experiments across various tasks and DiT architectures demonstrate that Sortblock achieves over 2$\times$ inference speedup with minimal degradation in output quality, offering an effective and generalizable solution for accelerating diffusion-based generative models.