NDVQ: Robust Neural Audio Codec with Normal Distribution-Based Vector Quantization

TL;DR

NDVQ introduces a novel vector quantization method using normal distributions to improve robustness and perceptual quality in neural audio codecs, especially under extremely low bandwidth conditions.

Contribution

The paper proposes NDVQ, a distribution-based vector quantization technique that enhances audio codec robustness by explicitly modeling codebook variance, leading to better quality and lower distortion.

Findings

NDVQ outperforms existing codecs like EnCodec in audio quality.

NDVQ achieves superior zero-shot TTS performance in low bandwidth.

The method effectively reduces signal distortion in noisy VQ codebooks.

Abstract

Built upon vector quantization (VQ), discrete audio codec models have achieved great success in audio compression and auto-regressive audio generation. However, existing models face substantial challenges in perceptual quality and signal distortion, especially when operating in extremely low bandwidth, rooted in the sensitivity of the VQ codebook to noise. This degradation poses significant challenges for several downstream tasks, such as codec-based speech synthesis. To address this issue, we propose a novel VQ method, Normal Distribution-based Vector Quantization (NDVQ), by introducing an explicit margin between the VQ codes via learning a variance. Specifically, our approach involves mapping the waveform to a latent space and quantizing it by selecting the most likely normal distribution, with each codebook entry representing a unique normal distribution defined by its mean and variance. Using these distribution-based VQ codec codes, a decoder reconstructs the input waveform. NDVQ is trained with additional distribution-related losses, alongside reconstruction and discrimination losses. Experiments demonstrate that NDVQ outperforms existing audio compression baselines, such as EnCodec, in terms of audio quality and zero-shot TTS, particularly in very low bandwidth scenarios.