Fast-ARDiff: An Entropy-informed Acceleration Framework for Continuous Space Autoregressive Generation

TL;DR

Fast-ARDiff introduces an entropy-informed, unified framework that accelerates autoregressive diffusion hybrid models by optimizing both components jointly, significantly reducing latency while maintaining high-quality synthesis.

Contribution

It proposes a novel joint optimization framework with entropy-informed strategies and dynamic scheduling to accelerate AR-diffusion models for image and text generation.

Findings

Achieves 4.3× speedup on ImageNet 256×256 with TransDiff.

Attains 3× acceleration on text-conditioned generation with NextStep-1.

Demonstrates stable training and high-quality synthesis with few diffusion steps.

Abstract

Autoregressive(AR)-diffusion hybrid paradigms combine AR's structured modeling with diffusion's photorealistic synthesis, yet suffer from high latency due to sequential AR generation and iterative denoising. In this work, we tackle this bottleneck and propose a unified AR-diffusion framework Fast-ARDiff that jointly optimizes both components, accelerating AR speculative decoding while simultaneously facilitating faster diffusion decoding. Specifically: (1) The entropy-informed speculative strategy encourages draft model to produce higher-entropy representations aligned with target model's entropy characteristics, mitigating entropy mismatch and high rejection rates caused by draft overconfidence. (2) For diffusion decoding, rather than treating it as an independent module, we integrate it into the same end-to-end framework using a dynamic scheduler that prioritizes AR optimization to guide the diffusion part in further steps. The diffusion part is optimized through a joint distillation framework combining trajectory and distribution matching, ensuring stable training and high-quality synthesis with extremely few steps. During inference, shallow feature entropy from AR module is used to pre-filter low-entropy drafts, avoiding redundant computation and improving latency. Fast-ARDiff achieves state-of-the-art acceleration across diverse models: on ImageNet 256$\times$256, TransDiff attains 4.3$\times$ lossless speedup, and NextStep-1 achieves 3$\times$ acceleration on text-conditioned generation. Code will be available at https://github.com/aSleepyTree/Fast-ARDiff.