Semantic Context Matters: Improving Conditioning for Autoregressive Models

TL;DR

SCAR enhances autoregressive image models by incorporating semantic context through a compact prefix and alignment guidance, significantly improving instruction adherence and visual quality in image editing tasks.

Contribution

Introduces SCAR, a novel semantic-context-driven method that improves conditioning in autoregressive models for image editing, overcoming previous semantic limitations and high computational costs.

Findings

Outperforms prior AR-based methods in visual fidelity and semantic alignment

Achieves superior results on instruction editing and controllable generation benchmarks

Maintains flexibility across different autoregressive paradigms

Abstract

Recently, autoregressive (AR) models have shown strong potential in image generation, offering better scalability and easier integration with unified multi-modal systems compared to diffusion-based methods. However, extending AR models to general image editing remains challenging due to weak and inefficient conditioning, often leading to poor instruction adherence and visual artifacts. To address this, we propose SCAR, a Semantic-Context-driven method for Autoregressive models. SCAR introduces two key components: Compressed Semantic Prefilling, which encodes high-level semantics into a compact and efficient prefix, and Semantic Alignment Guidance, which aligns the last visual hidden states with target semantics during autoregressive decoding to enhance instruction fidelity. Unlike decoding-stage injection methods, SCAR builds upon the flexibility and generality of vector-quantized-based prefilling while overcoming its semantic limitations and high cost. It generalizes across both next-token and next-set AR paradigms with minimal architectural changes. SCAR achieves superior visual fidelity and semantic alignment on both instruction editing and controllable generation benchmarks, outperforming prior AR-based methods while maintaining controllability. Code will be released at https://github.com/AMAP-ML/SCAR.