Context-Aware Autoregressive Models for Multi-Conditional Image Generation

TL;DR

ContextAR introduces a flexible autoregressive framework that effectively incorporates multiple conditions into image generation, achieving high controllability and competitive performance without fine-tuning.

Contribution

It proposes a novel method to embed diverse conditions into token sequences and introduces hybrid positional encodings and conditional attention for improved multi-conditional image generation.

Findings

Supports arbitrary condition combinations during inference.

Achieves competitive performance with diffusion models.

Demonstrates high controllability and versatility.

Abstract

Autoregressive transformers have recently shown impressive image generation quality and efficiency on par with state-of-the-art diffusion models. Unlike diffusion architectures, autoregressive models can naturally incorporate arbitrary modalities into a single, unified token sequence--offering a concise solution for multi-conditional image generation tasks. In this work, we propose $\textbf{ContextAR}$, a flexible and effective framework for multi-conditional image generation. ContextAR embeds diverse conditions (e.g., canny edges, depth maps, poses) directly into the token sequence, preserving modality-specific semantics. To maintain spatial alignment while enhancing discrimination among different condition types, we introduce hybrid positional encodings that fuse Rotary Position Embedding with Learnable Positional Embedding. We design Conditional Context-aware Attention to reduces computational complexity while preserving effective intra-condition perception. Without any fine-tuning, ContextAR supports arbitrary combinations of conditions during inference time. Experimental results demonstrate the powerful controllability and versatility of our approach, and show that the competitive perpormance than diffusion-based multi-conditional control approaches the existing autoregressive baseline across diverse multi-condition driven scenarios. Project page: $\href{https://context-ar.github.io/}{https://context-ar.github.io/.}$