Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation

TL;DR

This paper introduces Locality-aware Parallel Decoding (LPD), a novel method that significantly accelerates autoregressive image generation by enabling flexible, high-parallelism generation orders while maintaining quality.

Contribution

It proposes a new architecture and scheduling technique that allow arbitrary parallel generation with minimized dependencies, reducing steps and latency without quality loss.

Findings

Reduced generation steps from 256 to 20 for 256x256 images.

Achieved at least 3.4x lower latency compared to previous methods.

Maintained generation quality on ImageNet benchmarks.

Abstract

We present Locality-aware Parallel Decoding (LPD) to accelerate autoregressive image generation. Traditional autoregressive image generation relies on next-patch prediction, a memory-bound process that leads to high latency. Existing works have tried to parallelize next-patch prediction by shifting to multi-patch prediction to accelerate the process, but only achieved limited parallelization. To achieve high parallelization while maintaining generation quality, we introduce two key techniques: (1) Flexible Parallelized Autoregressive Modeling, a novel architecture that enables arbitrary generation ordering and degrees of parallelization. It uses learnable position query tokens to guide generation at target positions while ensuring mutual visibility among concurrently generated tokens for consistent parallel decoding. (2) Locality-aware Generation Ordering, a novel schedule that forms groups to minimize intra-group dependencies and maximize contextual support, enhancing generation quality. With these designs, we reduce the generation steps from 256 to 20 (256$\times$256 res.) and 1024 to 48 (512$\times$512 res.) without compromising quality on the ImageNet class-conditional generation, and achieving at least 3.4$\times$ lower latency than previous parallelized autoregressive models.