FlexSpeech: Towards Stable, Controllable and Expressive Text-to-Speech

TL;DR

FlexSpeech introduces a novel TTS model that combines autoregressive duration prediction with non-autoregressive acoustic modeling to achieve state-of-the-art stability, naturalness, and rapid style transfer in speech synthesis.

Contribution

The paper presents a hybrid TTS framework that explicitly models phonetic durations for stability and employs a lightweight, optimized duration predictor for style transfer, enhancing naturalness and controllability.

Findings

Achieves state-of-the-art stability and naturalness in zero-shot TTS.

Enables rapid style transfer with minimal data (~100 samples).

Maintains stability and quality without retraining the acoustic model.

Abstract

Current speech generation research can be categorized into two primary classes: non-autoregressive and autoregressive. The fundamental distinction between these approaches lies in the duration prediction strategy employed for predictable-length sequences. The NAR methods ensure stability in speech generation by explicitly and independently modeling the duration of each phonetic unit. Conversely, AR methods employ an autoregressive paradigm to predict the compressed speech token by implicitly modeling duration with Markov properties. Although this approach improves prosody, it does not provide the structural guarantees necessary for stability. To simultaneously address the issues of stability and naturalness in speech generation, we propose FlexSpeech, a stable, controllable, and expressive TTS model. The motivation behind FlexSpeech is to incorporate Markov dependencies and preference optimization directly on the duration predictor to boost its naturalness while maintaining explicit modeling of the phonetic units to ensure stability. Specifically, we decompose the speech generation task into two components: an AR duration predictor and a NAR acoustic model. The acoustic model is trained on a substantial amount of data to learn to render audio more stably, given reference audio prosody and phone durations. The duration predictor is optimized in a lightweight manner for different stylistic variations, thereby enabling rapid style transfer while maintaining a decoupled relationship with the specified speaker timbre. Experimental results demonstrate that our approach achieves SOTA stability and naturalness in zero-shot TTS. More importantly, when transferring to a specific stylistic domain, we can accomplish lightweight optimization of the duration module solely with about 100 data samples, without the need to adjust the acoustic model, thereby enabling rapid and stable style transfer.