Prosodic Representation Learning and Contextual Sampling for Neural Text-to-Speech

TL;DR

This paper presents Kathaka, a neural TTS model that learns prosody distribution at the sentence level and uses contextual information from text to improve naturalness, achieving significant quality improvements.

Contribution

Introduces a novel two-stage training process for neural TTS that incorporates contextual prosody sampling using BERT and graph-attention networks.

Findings

13.2% relative improvement in naturalness over baseline

Effective prosody modeling at sentence level

Robust sampling method with consistent improvements

Abstract

In this paper, we introduce Kathaka, a model trained with a novel two-stage training process for neural speech synthesis with contextually appropriate prosody. In Stage I, we learn a prosodic distribution at the sentence level from mel-spectrograms available during training. In Stage II, we propose a novel method to sample from this learnt prosodic distribution using the contextual information available in text. To do this, we use BERT on text, and graph-attention networks on parse trees extracted from text. We show a statistically significant relative improvement of $13.2\%$ in naturalness over a strong baseline when compared to recordings. We also conduct an ablation study on variations of our sampling technique, and show a statistically significant improvement over the baseline in each case.