Masked Vector Quantization

TL;DR

The paper introduces Masked Vector Quantization (MVQ), a novel framework that enhances discrete latent representations in generative models, significantly improving efficiency and quality on ImageNet with fewer tokens and codebook entries.

Contribution

MVQ increases code vector capacity through mask learning with MH-Dropout, reducing sampling time and improving generative quality in vector quantization architectures.

Findings

Up to 68% FID reduction on ImageNet 64x64

7-45x faster token sampling during inference

Smaller latent spaces enable transferable visual representations

Abstract

Generative models with discrete latent representations have recently demonstrated an impressive ability to learn complex high-dimensional data distributions. However, their performance relies on a long sequence of tokens per instance and a large number of codebook entries, resulting in long sampling times and considerable computation to fit the categorical posterior. To address these issues, we propose the Masked Vector Quantization (MVQ) framework which increases the representational capacity of each code vector by learning mask configurations via a stochastic winner-takes-all training regime called Multiple Hypothese Dropout (MH-Dropout). On ImageNet 64$\times$64, MVQ reduces FID in existing vector quantization architectures by up to $68\%$ at 2 tokens per instance and $57\%$ at 5 tokens. These improvements widen as codebook entries is reduced and allows for $7\textit{--}45\times$ speed-up in token sampling during inference. As an additional benefit, we find that smaller latent spaces lead to MVQ identifying transferable visual representations where multiple can be smoothly combined.