Mitigating Premature Discretization with Progressive Quantization for Robust Vector Tokenization

TL;DR

ProVQ introduces a progressive, curriculum-based approach to vector quantization that mitigates premature discretization, leading to improved performance in multimodal and biological sequence modeling.

Contribution

This paper proposes ProVQ, a novel progressive quantization method that dynamically adjusts the hardness of quantization during training, addressing the issue of premature discretization in VQ.

Findings

Enhanced reconstruction and generation on ImageNet benchmarks.

Significant improvements in protein structure tokenization.

ProVQ outperforms existing VQ methods across multiple modalities.

Abstract

Vector Quantization (VQ) has become the cornerstone of tokenization for many multimodal Large Language Models and diffusion synthesis. However, existing VQ paradigms suffer from a fundamental conflict: they enforce discretization before the encoder has captured the underlying data manifold. We term this phenomenon Premature Discretization. To resolve this, we propose Progressive Quantization (ProVQ), which incorporates the dynamics of quantization hardness as a fundamental yet previously overlooked axis in VQ training. By treating quantization as a curriculum that smoothly anneals from a continuous latent space to a discrete one, ProVQ effectively guides the codebook toward the well-expanded manifolds. Extensive experimental results demonstrate the broad effectiveness of ProVQ across diverse modalities. We report improved reconstruction and generative performance on the ImageNet-1K and ImageNet-100 benchmarks, highlighting the ProVQ's boost for generative modeling. Furthermore, ProVQ proves highly effective for modeling complex biological sequences, establishing a new performance ceiling for protein structure tokenization on the StrutTokenBench leaderboard.