Semantic-Aware Prefix Learning for Token-Efficient Image Generation

TL;DR

This paper introduces SMAP, a semantic-aware prefix tokenizer for image generation that enforces semantic grounding in latent representations, leading to improved reconstruction and generation quality with fewer tokens.

Contribution

SMAP is a novel tokenization method that incorporates class-level semantics into the tokenization process, making semantics essential for representation learning.

Findings

SMAP improves reconstruction quality across tokenization settings.

Semantically grounded latent space enhances downstream generation performance.

SMAP achieves strong results with compact token budgets.

Abstract

Visual tokenizers play a central role in latent image generation by bridging high-dimensional images and tractable generative modeling. However, most existing tokenizers are still trained with reconstruction-dominated objectives, which often yield latent representations that are only weakly grounded in high-level semantics. Recent approaches improve semantic alignment, but typically treat semantic signals as auxiliary regularization rather than making them functionally necessary for representation learning. We propose SMAP, a SeMantic-Aware Prefix tokenizer that injects class-level semantic conditions into a query-based 1D tokenization framework. To make semantics indispensable during training, SMAP introduces a tail token dropping strategy, which forces semantic conditions and early latent prefixes to bear increasing responsibility under progressively reduced token budgets. To verify that the resulting latent space is useful for generation rather than reconstruction alone, we further introduce CARD, a hybrid Causal AutoRegressive--Diffusion generator. Extensive experiments on ImageNet show that SMAP consistently improves reconstruction quality across discrete and continuous tokenization settings, and that its semantically grounded latent space yields strong downstream generation performance under compact token budgets.