CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech

TL;DR

CLaM-TTS introduces a probabilistic residual vector quantization approach to improve neural codec-based zero-shot TTS, reducing token sequence length and enabling efficient multi-token generation, achieving competitive naturalness and speed.

Contribution

It presents a novel probabilistic residual vector quantization method that enhances compression and multi-token generation in neural codec-based TTS models.

Findings

Outperforms or matches state-of-the-art in naturalness and intelligibility.

Reduces inference time compared to existing models.

Shows impact of pretraining and tokenization strategies on performance.

Abstract

With the emergence of neural audio codecs, which encode multiple streams of discrete tokens from audio, large language models have recently gained attention as a promising approach for zero-shot Text-to-Speech (TTS) synthesis. Despite the ongoing rush towards scaling paradigms, audio tokenization ironically amplifies the scalability challenge, stemming from its long sequence length and the complexity of modelling the multiple sequences. To mitigate these issues, we present CLaM-TTS that employs a probabilistic residual vector quantization to (1) achieve superior compression in the token length, and (2) allow a language model to generate multiple tokens at once, thereby eliminating the need for cascaded modeling to handle the number of token streams. Our experimental results demonstrate that CLaM-TTS is better than or comparable to state-of-the-art neural codec-based TTS models regarding naturalness, intelligibility, speaker similarity, and inference speed. In addition, we examine the impact of the pretraining extent of the language models and their text tokenization strategies on performances.