Speculative Coupled Decoding for Training-Free Lossless Acceleration of Autoregressive Visual Generation

TL;DR

The paper introduces Speculative Coupled Decoding, a training-free method that significantly accelerates autoregressive visual generation without quality loss, by stabilizing draft token sampling.

Contribution

It proposes a novel, training-free, lossless decoding framework that enhances AR generation speed by coupling draft tokens, with minimal algorithm modification and no additional training.

Findings

Achieves up to 4.2x speedup in image generation

Achieves up to 13.6x speedup in video generation

Maintains generation quality without additional training

Abstract

Autoregressive (AR) modeling has recently emerged as a promising new paradigm in visual generation, but its practical adoption is severely constrained by the slow inference speed of per-token generation, which often requires thousands of steps to produce a single sample. While several Speculative Decoding (SD)-based methods have been proposed to solve this problem by generating multiple tokens in a single forward step, they suffer from limited speedup, degraded quality, or require the training of a draft model. To solve these problems, we propose a new training-free, lossless SD framework, Speculative Coupled Decoding (SCD), by extending the recently proposed Speculative Jacobi Decoding (SJD). While SJD shows strong potential for accelerating AR generation by combining Jacobi iteration and SD, we found that its acceptance rate is still significantly limited due to the instability arising from the independent sampling process used during draft token generation. To overcome this, we introduce an information-theoretic approach, Coupling, which stabilizes the drafting trajectory of SJD by maximizing the probability of sampling identical draft tokens across consecutive iterations, significantly enhancing the acceptance rate while preserving its lossless property. Remarkably, this method requires only a single-line modification to the existing algorithm with almost zero overhead, yet achieves substantial performance gains, delivering up to a 4.2x speedup in image generation and 13.6x speedup in video generation compared to standard AR decoding, without any degradation or the need for additional training. The source code is available at https://github.com/junhyukso/SCD