E-CAR: Efficient Continuous Autoregressive Image Generation via Multistage Modeling

TL;DR

ECAR introduces a multistage, hierarchical approach for continuous autoregressive image generation that significantly improves efficiency while maintaining high image quality, enabling faster and more resource-effective image synthesis.

Contribution

The paper proposes a novel multistage modeling framework that reduces computational complexity and enables parallel token processing in continuous autoregressive image generation.

Findings

Achieves 10× FLOPs reduction compared to previous methods.

Speeds up image generation by 5× for 256×256 images.

Maintains comparable image quality to state-of-the-art models.

Abstract

Recent advances in autoregressive (AR) models with continuous tokens for image generation show promising results by eliminating the need for discrete tokenization. However, these models face efficiency challenges due to their sequential token generation nature and reliance on computationally intensive diffusion-based sampling. We present ECAR (Efficient Continuous Auto-Regressive Image Generation via Multistage Modeling), an approach that addresses these limitations through two intertwined innovations: (1) a stage-wise continuous token generation strategy that reduces computational complexity and provides progressively refined token maps as hierarchical conditions, and (2) a multistage flow-based distribution modeling method that transforms only partial-denoised distributions at each stage comparing to complete denoising in normal diffusion models. Holistically, ECAR operates by generating tokens at increasing resolutions while simultaneously denoising the image at each stage. This design not only reduces token-to-image transformation cost by a factor of the stage number but also enables parallel processing at the token level. Our approach not only enhances computational efficiency but also aligns naturally with image generation principles by operating in continuous token space and following a hierarchical generation process from coarse to fine details. Experimental results demonstrate that ECAR achieves comparable image quality to DiT Peebles & Xie [2023] while requiring 10$\times$ FLOPs reduction and 5$\times$ speedup to generate a 256$\times$256 image.