HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling

TL;DR

HH-Codec is a neural speech codec that achieves high compression and fidelity for 24 kHz audio using a single quantizer, enabling efficient speech reconstruction at ultra-low bandwidths.

Contribution

The paper introduces HH-Codec, a novel neural codec with a specialized vector quantization space and an asymmetric architecture for efficient, high-fidelity speech compression at low bandwidths.

Findings

Achieves 24 tokens/sec compression at 0.3 kbps bandwidth.

Outperforms existing speech codecs in reconstruction quality.

Demonstrates effective codebook utilization and adaptability.

Abstract

Discrete speech tokenization is a fundamental component in speech codecs. However, in large-scale speech-to-speech systems, the complexity of parallel streams from multiple quantizers and the computational cost of high-time-dimensional codecs pose significant challenges. In this paper, we introduce HH-Codec, a neural codec that achieves extreme compression at 24 tokens per second for 24 kHz audio while relying on single-quantizer inference. Our approach involves a carefully designed Vector Quantization space for Spoken Language Modeling, optimizing compression efficiency while minimizing information loss. Building on this, we propose an asymmetric encoder-decoder architecture (Audio-VQ-Mel-Audio) that leverages dual supervision and progressive training to enhance reconstruction stability and fidelity. HH-Codec achieves state-of-the-art performance in speech reconstruction with an ultra-low bandwidth of 0.3 kbps. We further evaluate its effectiveness in codebook utilization and generative model adaptation, with extensive ablations validating the necessity of each module. HH-Codec is available at https://github.com/opendilab/HH-Codec.