Learning utterance-level representations through token-level acoustic latents prediction for Expressive Speech Synthesis

TL;DR

This paper introduces a novel expressive speech synthesis model that leverages token-level latent prosodic variables to effectively capture and control both fine-grained and utterance-level speech attributes, improving diversity and disentanglement.

Contribution

It proposes a method to use token-level latent spaces combined with a prior network to better capture and control utterance-level prosody in speech synthesis.

Findings

The model effectively captures diverse prosodic features.

It improves control over utterance-level speech attributes.

Qualitative and quantitative evaluations validate the approach.

Abstract

This paper proposes an Expressive Speech Synthesis model that utilizes token-level latent prosodic variables in order to capture and control utterance-level attributes, such as character acting voice and speaking style. Current works aim to explicitly factorize such fine-grained and utterance-level speech attributes into different representations extracted by modules that operate in the corresponding level. We show that the fine-grained latent space also captures coarse-grained information, which is more evident as the dimension of latent space increases in order to capture diverse prosodic representations. Therefore, a trade-off arises between the diversity of the token-level and utterance-level representations and their disentanglement. We alleviate this issue by first capturing rich speech attributes into a token-level latent space and then, separately train a prior network that given the input text, learns utterance-level representations in order to predict the phoneme-level, posterior latents extracted during the previous step. Both qualitative and quantitative evaluations are used to demonstrate the effectiveness of the proposed approach. Audio samples are available in our demo page.