SJD++: Improved Speculative Jacobi Decoding for Training-free Acceleration of Discrete Auto-regressive Text-to-Image Generation

TL;DR

SJD++ is a training-free decoding algorithm that accelerates autoregressive text-to-image generation by performing multi-token predictions per step, reducing inference time significantly without quality loss.

Contribution

It introduces a novel speculative Jacobi decoding method that combines multi-token prediction and probabilistic verification, enabling faster inference in text-to-image models.

Findings

Achieves 2x to 3x reduction in inference latency.

Reduces number of generation steps by 2x to 7x.

Maintains visual quality comparable to baseline models.

Abstract

Large autoregressive models can generate high-quality, high-resolution images but suffer from slow generation speed, because these models require hundreds to thousands of sequential forward passes for next-token prediction during inference. To accelerate autoregressive text-to-image generation, we propose Speculative Jacobi Decoding++ (SJD++), a training-free probabilistic parallel decoding algorithm. Unlike traditional next-token prediction, SJD++ performs multi-token prediction in each forward pass, drastically reducing generation steps. Specifically, it integrates the iterative multi-token prediction mechanism from Jacobi decoding, with the probabilistic drafting-and-verification mechanism from speculative sampling. More importantly, for further acceleration, SJD++ reuses high-confidence draft tokens after each verification phase instead of resampling them all. We conduct extensive experiments on several representative autoregressive text-to-image generation models and demonstrate that SJD++ achieves $2\times$ to $3\times$ inference latency reduction and $2\times$ to $7\times$ step compression, while preserving visual quality with no observable degradation.