VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction

TL;DR

VQRAE introduces a unified representation model combining continuous semantic features and discrete tokens for multimodal understanding, generation, and reconstruction, leveraging a novel high-dimensional vector quantization approach.

Contribution

The paper pioneers a unified tokenizer that produces both continuous and discrete representations within a single autoencoder framework, enabling multimodal tasks.

Findings

Achieves 100% codebook utilization at high dimensions.

Demonstrates competitive performance on visual understanding benchmarks.

Shows promising scaling in autoregressive generation tasks.

Abstract

Unifying multimodal understanding, generation and reconstruction representation in a single tokenizer remains a key challenge in building unified models. Previous research predominantly attempts to address this in a dual encoder paradigm, e.g., utilizing the separate encoders for understanding and generation respectively or balancing semantic representations and low-level features with contrastive loss. In this paper, we propose VQRAE, a Vector Quantization version of Representation AutoEncoders, which pioneers the first exploration in unified representation to produce Continuous semantic features for image understanding and Discrete tokens for visual generation within a unified tokenizer. Specifically, we build upon pretrained vision foundation models with a symmetric ViT decoder and adopt a two-stage training strategy: first, it freezes the encoder and learns a high-dimensional semantic VQ codebook with pixel reconstruction objective; then jointly optimizes the encoder with self-distillation constraints. This design enables negligible semantic information for maintaining the ability of multimodal understanding, discrete tokens that are compatible for generation and fine-grained reconstruction. Besides, we identify the intriguing property in quantizing semantic encoders that rely on high-dimensional codebook in contrast to the previous common practice of low-dimensional codebook in image reconstruction. The semantic VQ codebook can achieve a 100% utilization ratio at a dimension of 1536. VQRAE presents competitive performance on several benchmarks of visual understanding, generation and reconstruction with promising scaling property in the autoregressive paradigm for its discrete merits.