Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation

TL;DR

SemTok introduces a novel semantic 1D image tokenizer that compresses images into high-level semantic tokens, achieving state-of-the-art reconstruction fidelity and improving downstream image generation tasks.

Contribution

The paper presents SemTok, a new 1D semantic tokenizer with a 2D-to-1D scheme, semantic alignment, and a two-stage training strategy, advancing image tokenization methods.

Findings

Achieves superior image reconstruction fidelity.

Enhances downstream image generation performance.

Outperforms existing visual tokenizers in experiments.

Abstract

Visual generative models based on latent space have achieved great success, underscoring the significance of visual tokenization. Mapping images to latents boosts efficiency and enables multimodal alignment for scaling up in downstream tasks. Existing visual tokenizers primarily map images into fixed 2D spatial grids and focus on pixel-level restoration, which hinders the capture of representations with compact global semantics. To address these issues, we propose \textbf{SemTok}, a semantic one-dimensional tokenizer that compresses 2D images into 1D discrete tokens with high-level semantics. SemTok sets a new state-of-the-art in image reconstruction, achieving superior fidelity with a remarkably compact token representation. This is achieved via a synergistic framework with three key innovations: a 2D-to-1D tokenization scheme, a semantic alignment constraint, and a two-stage generative training strategy. Building on SemTok, we construct a masked autoregressive generation framework, which yields notable improvements in downstream image generation tasks. Experiments confirm the effectiveness of our semantic 1D tokenization. Our code will be open-sourced.