Exploiting Deep Sentential Context for Expressive End-to-End Speech Synthesis

TL;DR

This paper introduces a context extractor built on self-attention networks to better utilize sentential context in expressive speech synthesis, significantly improving prosody and naturalness in generated speech.

Contribution

It proposes a novel context extraction method that aggregates multi-layer SAN outputs to enhance prosodic modeling in end-to-end TTS systems, especially for expressive corpora.

Findings

Enhanced prosody and expressiveness in synthesized speech.

Weighted aggregation outperforms direct aggregation in modeling expressivity.

Approach improves naturalness on expressive speech corpora.

Abstract

Attention-based seq2seq text-to-speech systems, especially those use self-attention networks (SAN), have achieved state-of-art performance. But an expressive corpus with rich prosody is still challenging to model as 1) prosodic aspects, which span across different sentential granularities and mainly determine acoustic expressiveness, are difficult to quantize and label and 2) the current seq2seq framework extracts prosodic information solely from a text encoder, which is easily collapsed to an averaged expression for expressive contents. In this paper, we propose a context extractor, which is built upon SAN-based text encoder, to sufficiently exploit the sentential context over an expressive corpus for seq2seq-based TTS. Our context extractor first collects prosodic-related sentential context information from different SAN layers and then aggregates them to learn a comprehensive sentence representation to enhance the expressiveness of the final generated speech. Specifically, we investigate two methods of context aggregation: 1) direct aggregation which directly concatenates the outputs of different SAN layers, and 2) weighted aggregation which uses multi-head attention to automatically learn contributions for different SAN layers. Experiments on two expressive corpora show that our approach can produce more natural speech with much richer prosodic variations, and weighted aggregation is more superior in modeling expressivity.