Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation

TL;DR

Single-Codec introduces a single-codebook speech codec that improves efficiency and quality in speech generation by disentangling speech representations and enhancing encoding with contextual and resampling modules.

Contribution

It proposes a novel single-codebook, single-sequence speech codec using disentangled VQ-VAE and advanced encoder modules, outperforming multi-codebook codecs in quality and bandwidth.

Findings

Higher reconstruction quality than multi-codebook codecs

Lower bandwidth of only 304bps

Improved naturalness and intelligibility in LLM-TTS experiments

Abstract

The multi-codebook speech codec enables the application of large language models (LLM) in TTS but bottlenecks efficiency and robustness due to multi-sequence prediction. To avoid this obstacle, we propose Single-Codec, a single-codebook single-sequence codec, which employs a disentangled VQ-VAE to decouple speech into a time-invariant embedding and a phonetically-rich discrete sequence. Furthermore, the encoder is enhanced with 1) contextual modeling with a BLSTM module to exploit the temporal information, 2) a hybrid sampling module to alleviate distortion from upsampling and downsampling, and 3) a resampling module to encourage discrete units to carry more phonetic information. Compared with multi-codebook codecs, e.g., EnCodec and TiCodec, Single-Codec demonstrates higher reconstruction quality with a lower bandwidth of only 304bps. The effectiveness of Single-Code is further validated by LLM-TTS experiments, showing improved naturalness and intelligibility.