Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance

TL;DR

Koel-TTS is a novel speech synthesis model that improves controllability, speaker similarity, and naturalness by integrating preference alignment and classifier-free guidance, outperforming existing models on key metrics.

Contribution

The paper introduces Koel-TTS, a Transformer-based TTS model that incorporates preference alignment and classifier-free guidance to enhance speech quality and controllability.

Findings

Significantly improves speaker similarity and naturalness.

Outperforms state-of-the-art TTS models on key metrics.

Achieves high-quality synthesis with less training data.

Abstract

While autoregressive speech token generation models produce speech with remarkable variety and naturalness, their inherent lack of controllability often results in issues such as hallucinations and undesired vocalizations that do not conform to conditioning inputs. We introduce Koel-TTS, a suite of enhanced encoder-decoder Transformer TTS models that address these challenges by incorporating preference alignment techniques guided by automatic speech recognition and speaker verification models. Additionally, we incorporate classifier-free guidance to further improve synthesis adherence to the transcript and reference speaker audio. Our experiments demonstrate that these optimizations significantly enhance target speaker similarity, intelligibility, and naturalness of synthesized speech. Notably, Koel-TTS directly maps text and context audio to acoustic tokens, and on the aforementioned metrics, outperforms state-of-the-art TTS models, despite being trained on a significantly smaller dataset. Audio samples and demos are available on our website.