Rate-Adaptive Quantization: A Multi-Rate Codebook Adaptation for Vector Quantization-based Generative Models

TL;DR

This paper introduces Rate-Adaptive Quantization (RAQ), a framework that creates variable-rate codebooks from a single VQ model, allowing flexible compression and reconstruction without retraining.

Contribution

The paper proposes RAQ, a data-driven multi-rate codebook adaptation method for VQ models, including a clustering-based approach for pre-trained models, enhancing flexibility and efficiency.

Findings

RAQ outperforms fixed-rate VQ baselines across multiple rates.

RAQ enables seamless handling of diverse bitrate requirements.

The clustering-based procedure offers an alternative when retraining is infeasible.

Abstract

Learning discrete representations with vector quantization (VQ) has emerged as a powerful approach in various generative models. However, most VQ-based models rely on a single, fixed-rate codebook, requiring extensive retraining for new bitrates or efficiency requirements. We introduce Rate-Adaptive Quantization (RAQ), a multi-rate codebook adaptation framework for VQ-based generative models. RAQ applies a data-driven approach to generate variable-rate codebooks from a single baseline VQ model, enabling flexible tradeoffs between compression and reconstruction fidelity. Additionally, we provide a simple clustering-based procedure for pre-trained VQ models, offering an alternative when retraining is infeasible. Our experiments show that RAQ performs effectively across multiple rates, often outperforming conventional fixed-rate VQ baselines. By enabling a single system to seamlessly handle diverse bitrate requirements, RAQ extends the adaptability of VQ-based generative models and broadens their applicability to data compression, reconstruction, and generation tasks.