Hierarchical Emotion Prediction and Control in Text-to-Speech Synthesis

TL;DR

This paper introduces a hierarchical emotion distribution framework for text-to-speech synthesis, enabling nuanced emotion control at multiple linguistic levels, validated through comprehensive evaluations.

Contribution

It proposes a novel hierarchical emotion distribution approach that captures emotion intensity variations across phonemes, words, and utterances for improved emotion control in TTS.

Findings

Effective emotion prediction demonstrated by both objective and subjective metrics.

Enhanced emotional expressiveness in synthesized speech with controllable granularity.

Hierarchical ED outperforms previous global prosody-based methods.

Abstract

It remains a challenge to effectively control the emotion rendering in text-to-speech (TTS) synthesis. Prior studies have primarily focused on learning a global prosodic representation at the utterance level, which strongly correlates with linguistic prosody. Our goal is to construct a hierarchical emotion distribution (ED) that effectively encapsulates intensity variations of emotions at various levels of granularity, encompassing phonemes, words, and utterances. During TTS training, the hierarchical ED is extracted from the ground-truth audio and guides the predictor to establish a connection between emotional and linguistic prosody. At run-time inference, the TTS model generates emotional speech and, at the same time, provides quantitative control of emotion over the speech constituents. Both objective and subjective evaluations validate the effectiveness of the proposed framework in terms of emotion prediction and control.