FREE: Uncertainty-Aware Autoregression for Parallel Diffusion Transformers

TL;DR

This paper introduces FREE, a framework for accelerating diffusion transformers using feature-level autoregression and uncertainty-guided relaxation, achieving up to 2.25x speedup without sacrificing quality.

Contribution

We propose a novel uncertainty-aware autoregressive method for diffusion transformers that enables lossless parallel sampling and dynamic acceptance adjustment.

Findings

FREE achieves up to 1.86x acceleration on ImageNet-512^2.

FREE (relax) reaches 2.25x speedup with maintained quality.

The method guarantees lossless acceleration with theoretical and empirical validation.

Abstract

Diffusion Transformers (DiTs) achieve state-of-the-art generation quality but require long sequential denoising trajectories, leading to high inference latency. Recent speculative inference methods enable lossless parallel sampling in U-Net-based diffusion models via a drafter-verifier scheme, but their acceleration is limited on DiTs due to insufficient draft accuracy during verification. To address this limitation, we analyze the DiTs' feature dynamics and find the features of the final transformer layer (top-block) exhibit strong temporal consistency and rich semantic abstraction. Based on this insight, we propose FREE, a novel framework that employs a lightweight drafter to perform feature-level autoregression with parallel verification, guaranteeing lossless acceleration with theoretical and empirical support. Meanwhile, prediction variance (uncertainty) of DiTs naturally increases in later denoising steps, reducing acceptance rates under speculative sampling. To mitigate this effect, we further introduce an uncertainty-guided relaxation strategy, forming FREE (relax), which dynamically adjusts the acceptance probability in response to uncertainty levels. Experiments on ImageNet-$512^2$ show that FREE achieves up to $1.86 \times$ acceleration, and FREE (relax) further reaches $2.25 \times$ speedup while maintaining high perceptual and quantitative fidelity in generation quality.