HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation

TL;DR

HMAR introduces a hierarchical masked auto-regressive approach for image generation that improves quality, speed, and flexibility over existing autoregressive and diffusion models by using next-scale prediction and masked token generation.

Contribution

The paper proposes HMAR, a novel hierarchical autoregressive model that enhances image generation by reducing sequence length, enabling flexible sampling schedules, and improving efficiency and quality.

Findings

HMAR matches or outperforms existing models on ImageNet benchmarks.

HMAR achieves over 2.5x faster training and 1.75x faster inference.

HMAR reduces inference memory footprint by over 3x.

Abstract

Visual Auto-Regressive modeling (VAR) has shown promise in bridging the speed and quality gap between autoregressive image models and diffusion models. VAR reformulates autoregressive modeling by decomposing an image into successive resolution scales. During inference, an image is generated by predicting all the tokens in the next (higher-resolution) scale, conditioned on all tokens in all previous (lower-resolution) scales. However, this formulation suffers from reduced image quality due to the parallel generation of all tokens in a resolution scale; has sequence lengths scaling superlinearly in image resolution; and requires retraining to change the sampling schedule. We introduce Hierarchical Masked Auto-Regressive modeling (HMAR), a new image generation algorithm that alleviates these issues using next-scale prediction and masked prediction to generate high-quality images with fast sampling. HMAR reformulates next-scale prediction as a Markovian process, wherein the prediction of each resolution scale is conditioned only on tokens in its immediate predecessor instead of the tokens in all predecessor resolutions. When predicting a resolution scale, HMAR uses a controllable multi-step masked generation procedure to generate a subset of the tokens in each step. On ImageNet 256x256 and 512x512 benchmarks, HMAR models match or outperform parameter-matched VAR, diffusion, and autoregressive baselines. We develop efficient IO-aware block-sparse attention kernels that allow HMAR to achieve faster training and inference times over VAR by over 2.5x and 1.75x respectively, as well as over 3x lower inference memory footprint. Finally, HMAR yields additional flexibility over VAR; its sampling schedule can be changed without further training, and it can be applied to image editing tasks in a zero-shot manner.