VAEVQ: Enhancing Discrete Visual Tokenization through Variational Modeling

TL;DR

VAEVQ introduces a variational approach to discrete visual tokenization, improving codebook utilization and stability, leading to better image reconstruction and generation performance.

Contribution

The paper proposes VAEVQ, a novel framework combining variational latent quantization, adaptive coherence strategies, and distribution regularization for enhanced discrete visual representations.

Findings

Outperforms state-of-the-art methods on benchmark datasets

Achieves more stable and coherent codebook learning

Improves reconstruction and generation quality

Abstract

Vector quantization (VQ) transforms continuous image features into discrete representations, providing compressed, tokenized inputs for generative models. However, VQ-based frameworks suffer from several issues, such as non-smooth latent spaces, weak alignment between representations before and after quantization, and poor coherence between the continuous and discrete domains. These issues lead to unstable codeword learning and underutilized codebooks, ultimately degrading the performance of both reconstruction and downstream generation tasks. To this end, we propose VAEVQ, which comprises three key components: (1) Variational Latent Quantization (VLQ), replacing the AE with a VAE for quantization to leverage its structured and smooth latent space, thereby facilitating more effective codeword activation; (2) Representation Coherence Strategy (RCS), adaptively modulating the alignment strength between pre- and post-quantization features to enhance consistency and prevent overfitting to noise; and (3) Distribution Consistency Regularization (DCR), aligning the entire codebook distribution with the continuous latent distribution to improve utilization. Extensive experiments on two benchmark datasets demonstrate that VAEVQ outperforms state-of-the-art methods.