Improving Emotional Speech Synthesis by Using SUS-Constrained VAE and Text Encoder Aggregation

TL;DR

This paper introduces SUS-constrained VAE and text encoder aggregation techniques to improve emotion embedding extraction and integration in speech synthesis, resulting in more expressive emotional speech.

Contribution

It proposes a novel constraint for VAE to enhance emotion embedding clustering and a method to aggregate encoder layer representations for better emotion expression.

Findings

Enhanced emotion embedding quality with better cluster cohesion

Improved emotional expressiveness in synthesized speech

Effective integration of syntactic and semantic information

Abstract

Learning emotion embedding from reference audio is a straightforward approach for multi-emotion speech synthesis in encoder-decoder systems. But how to get better emotion embedding and how to inject it into TTS acoustic model more effectively are still under investigation. In this paper, we propose an innovative constraint to help VAE extract emotion embedding with better cluster cohesion. Besides, the obtained emotion embedding is used as query to aggregate latent representations of all encoder layers via attention. Moreover, the queries from encoder layers themselves are also helpful. Experiments prove the proposed methods can enhance the encoding of comprehensive syntactic and semantic information and produce more expressive emotional speech.