STAR: Scale-wise Text-conditioned AutoRegressive image generation

TL;DR

STAR is a novel auto-regressive text-to-image model that enables high-resolution image generation up to 1024x1024 by introducing scale-wise generation, a pre-trained text encoder, and a stable sampling method, surpassing previous models in quality.

Contribution

The paper presents a scale-wise auto-regressive framework with new design strategies, including a pre-trained text encoder, normalized 2D Rotary Positional Encoding, and a stable sampling method, enabling high-resolution and stable image synthesis.

Findings

STAR achieves high-fidelity 1024x1024 images in 2.21 seconds.

It surpasses existing diffusion and auto-regressive models in quality and consistency.

The method stabilizes high-resolution generation through novel sampling techniques.

Abstract

We introduce STAR, a text-to-image model that employs a scale-wise auto-regressive paradigm. Unlike VAR, which is constrained to class-conditioned synthesis for images up to 256$\times$256, STAR enables text-driven image generation up to 1024$\times$1024 through three key designs. First, we introduce a pre-trained text encoder to extract and adopt representations for textual constraints, enhancing details and generalizability. Second, given the inherent structural correlation across different scales, we leverage 2D Rotary Positional Encoding (RoPE) and tweak it into a normalized version, ensuring consistent interpretation of relative positions across token maps and stabilizing the training process. Third, we observe that simultaneously sampling all tokens within a single scale can disrupt inter-token relationships, leading to structural instability, particularly in high-resolution generation. To address this, we propose a novel stable sampling method that incorporates causal relationships into the sampling process, ensuring both rich details and stable structures. Compared to previous diffusion models and auto-regressive models, STAR surpasses existing benchmarks in fidelity, text-image consistency, and aesthetic quality, requiring just 2.21s for 1024$\times$1024 images on A100. This highlights the potential of auto-regressive methods in high-quality image synthesis, offering new directions for the text-to-image generation.