ProMode: A Speech Prosody Model Conditioned on Acoustic and Textual Inputs

TL;DR

ProMode is a novel speech prosody model that encodes acoustic and textual inputs to generate detailed prosodic features, improving TTS systems' naturalness and expressiveness.

Contribution

The paper introduces ProMode, a new model that maps text and acoustic features to prosodic embeddings, enhancing prosody prediction and TTS performance.

Findings

ProMode outperforms state-of-the-art style encoders in prosody prediction.

Integrating ProMode features improves TTS prosody naturalness.

Perceptual tests favor ProMode-enhanced TTS over baselines.

Abstract

Prosody conveys rich emotional and semantic information of the speech signal as well as individual idiosyncrasies. We propose a stand-alone model that maps text-to-prosodic features such as F0 and energy and can be used in downstream tasks such as TTS. The ProMode encoder takes as input acoustic features and time-aligned textual content, both are partially masked, and obtains a fixed-length latent prosodic embedding. The decoder predicts acoustics in the masked region using both the encoded prosody input and unmasked textual content. Trained on the GigaSpeech dataset, we compare our method with state-of-the-art style encoders. For F0 and energy predictions, we show consistent improvements for our model at different levels of granularity. We also integrate these predicted prosodic features into a TTS system and conduct perceptual tests, which show higher prosody preference compared to the baselines, demonstrating the model's potential in tasks where prosody modeling is important.