Multi-Scale Local Speculative Decoding for Image Generation

TL;DR

This paper introduces MuLo-SD, a multi-scale speculative decoding framework that accelerates autoregressive image generation by combining low-resolution drafting with spatially aware verification, achieving up to 1.7x speedup while maintaining quality.

Contribution

The paper presents a novel multi-scale speculative decoding method with local rejection and resampling, improving speed and accuracy in autoregressive image synthesis.

Findings

Achieves up to 1.7x speedup over baselines.

Maintains semantic and perceptual quality comparable to existing methods.

Validated on MS-COCO dataset using multiple evaluation metrics.

Abstract

Autoregressive (AR) models have achieved remarkable success in image synthesis, yet their sequential nature imposes significant latency constraints. Speculative Decoding offers a promising avenue for acceleration, but existing approaches are limited by token-level ambiguity and lack of spatial awareness. In this work, we introduce Multi-Scale Local Speculative Decoding (MuLo-SD), a novel framework that combines multi-resolution drafting with spatially informed verification to accelerate AR image generation. Our method leverages a low-resolution drafter paired with learned up-samplers to propose candidate image tokens, which are then verified in parallel by a high-resolution target model. Crucially, we incorporate a local rejection and resampling mechanism, enabling efficient correction of draft errors by focusing on spatial neighborhoods rather than raster-scan resampling after the first rejection. We demonstrate that MuLo-SD achieves substantial speedups - up to $\mathbf{1.7\times}$ - outperforming strong speculative decoding baselines such as EAGLE-2 and LANTERN in terms of acceleration, while maintaining comparable semantic alignment and perceptual quality. These results are validated using GenEval, DPG-Bench, and FID/HPSv2 on the MS-COCO 5k validation split. Extensive ablations highlight the impact of up-sampling design, probability pooling, and local rejection and resampling with neighborhood expansion. Our approach sets a new state-of-the-art in speculative decoding for image synthesis, bridging the gap between efficiency and fidelity.