PURE Codec: Progressive Unfolding of Residual Entropy for Speech Codec Learning

TL;DR

PURE Codec introduces a progressive quantization framework guided by a speech enhancement model, improving stability and performance in low-bitrate neural speech codecs, especially under noisy conditions.

Contribution

It presents a novel multi-stage quantization method that enhances training stability and reconstruction quality in neural speech codecs using residual entropy unfolding.

Findings

Outperforms conventional RVQ codecs in reconstruction quality

Improves stability of training neural speech codecs

Enhances downstream TTS performance under noisy training conditions

Abstract

Neural speech codecs have achieved strong performance in low-bitrate compression, but residual vector quantization (RVQ) often suffers from unstable training and ineffective decomposition, limiting reconstruction quality and efficiency. We propose PURE Codec (Progressive Unfolding of Residual Entropy), a novel framework that guides multi-stage quantization using a pre-trained speech enhancement model. The first quantization stage reconstructs low-entropy, denoised speech embeddings, while subsequent stages encode residual high-entropy components. This design improves training stability significantly. Experiments demonstrate that PURE consistently outperforms conventional RVQ-based codecs in reconstruction and downstream speech language model-based text-to-speech, particularly under noisy training conditions.