UniCode$^2$: Cascaded Large-scale Codebooks for Unified Multimodal Understanding and Generation

TL;DR

UniCode$^2$ introduces a cascaded large-scale visual codebook system that enhances multimodal understanding and generation by improving token semantics, stability, and alignment with text, enabling better visual synthesis and comprehension.

Contribution

The paper presents a novel cascaded codebook framework with 500K entries that improves semantic alignment, stability, and scalability in visual tokenization for multimodal models.

Findings

Achieves high performance across multiple benchmarks.

Enables high-quality visual synthesis with minimal adaptation.

Maintains stability and semantic alignment at large scale.

Abstract

Unified multimodal large language models (MLLMs) have shown promise in jointly advancing multimodal understanding and generation, with visual codebooks discretizing images into tokens for autoregressive modeling. Existing codebook-based methods either rely on small vocabularies (~16K entries) that lack fine-grained semantics or naively scale up, resulting in low token utilization and unstable training. We propose UniCode$^2$, a cascaded codebook framework enabling large-scale, semantically aligned, and stable visual tokenization. By clustering millions of SigLIP sequence embeddings, we build a 500K-entry codebook that preserves vision-language alignment while expanding capacity. Stability is ensured via a cascaded design: a frozen codebook anchors the embedding space, and a trainable codebook refines task-specific semantics. This decoupling promotes high utilization and robust learning. Moreover, the alignment of our visual tokens with textual semantics enables seamless integration with pretrained diffusion decoders, supporting high-quality visual synthesis with minimal adaptation. UniCode^2 delivers strong performance across diverse benchmarks, demonstrating the viability of scaling visual token spaces without sacrificing stability, semantics, or modularity.