Investigating Stochastic Methods for Prosody Modeling in Speech Synthesis

TL;DR

This paper explores stochastic modeling techniques like Normalizing Flows and Rectified Flows for prosody generation in speech synthesis, showing they can produce natural, controllable prosody comparable to human speech.

Contribution

It introduces the application of stochastic methods to prosody modeling in TTS, demonstrating their effectiveness and controllability advantages over traditional deterministic approaches.

Findings

Stochastic methods achieve natural prosody comparable to humans.

They enable tunable controllability via sampling temperature.

Extensive evaluations validate the effectiveness of these methods.

Abstract

While generative methods have progressed rapidly in recent years, generating expressive prosody for an utterance remains a challenging task in text-to-speech synthesis. This is particularly true for systems that model prosody explicitly through parameters such as pitch, energy, and duration, which is commonly done for the sake of interpretability and controllability. In this work, we investigate the effectiveness of stochastic methods for this task, including Normalizing Flows, Conditional Flow Matching, and Rectified Flows. We compare these methods to a traditional deterministic baseline, as well as to real human realizations. Our extensive subjective and objective evaluations demonstrate that stochastic methods produce natural prosody on par with human speakers by capturing the variability inherent in human speech. Further, they open up additional controllability options by allowing the sampling temperature to be tuned.