Hawk: Leveraging Spatial Context for Faster Autoregressive Text-to-Image Generation

TL;DR

Hawk is a novel method that leverages the spatial structure of images to accelerate autoregressive text-to-image generation, achieving significant speedups while maintaining quality.

Contribution

Hawk introduces a spatially-aware speculative decoding approach that improves the efficiency of autoregressive image generation models.

Findings

Achieves 1.71x speedup over standard AR models

Maintains high image fidelity and diversity

Effectively utilizes spatial structure for faster generation

Abstract

Autoregressive (AR) image generation models are capable of producing high-fidelity images but often suffer from slow inference due to their inherently sequential, token-by-token decoding process. Speculative decoding, which employs a lightweight draft model to approximate the output of a larger AR model, has shown promise in accelerating text generation without compromising quality. However, its application to image generation remains largely underexplored. The challenges stem from a significantly larger sampling space, which complicates the alignment between the draft and target model outputs, coupled with the inadequate use of the two-dimensional spatial structure inherent in images, thereby limiting the modeling of local dependencies. To overcome these challenges, we introduce Hawk, a new approach that harnesses the spatial structure of images to guide the speculative model toward more accurate and efficient predictions. Experimental results on multiple text-to-image benchmarks demonstrate a 1.71x speedup over standard AR models, while preserving both image fidelity and diversity.