Online Clustered Codebook
Chuanxia Zheng, Andrea Vedaldi

TL;DR
This paper introduces CVQ-VAE, an online clustering method for vector quantization that addresses codebook collapse by dynamically updating dead codevectors using encoded features as anchors, improving the capacity of VQ models.
Contribution
The paper proposes a simple, effective online clustering approach for VQ-VAE that mitigates codebook collapse and enhances codebook utilization in complex vision tasks.
Findings
Improved codebook utilization across various datasets.
Enhanced performance in reconstruction and generation tasks.
Easy integration into existing VQ models.
Abstract
Vector Quantisation (VQ) is experiencing a comeback in machine learning, where it is increasingly used in representation learning. However, optimizing the codevectors in existing VQ-VAE is not entirely trivial. A problem is codebook collapse, where only a small subset of codevectors receive gradients useful for their optimisation, whereas a majority of them simply ``dies off'' and is never updated or used. This limits the effectiveness of VQ for learning larger codebooks in complex computer vision tasks that require high-capacity representations. In this paper, we present a simple alternative method for online codebook learning, Clustering VQ-VAE (CVQ-VAE). Our approach selects encoded features as anchors to update the ``dead'' codevectors, while optimising the codebooks which are alive via the original loss. This strategy brings unused codevectors closer in distribution to the encoded…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Online Clustered Codebook· youtube
Taxonomy
TopicsAdvanced Image and Video Retrieval Techniques · Remote-Sensing Image Classification · Sparse and Compressive Sensing Techniques
MethodsVQ-VAE
