Improving multi-speaker TTS prosody variance with a residual encoder and normalizing flows

TL;DR

This paper introduces a novel neural TTS model that enhances prosody variability and speaker distinctiveness by using flow-normalized speaker embeddings and a learned latent distribution, eliminating reference encoder dependency.

Contribution

The proposed model improves prosody variance and speaker separation in TTS by integrating flow-normalized embeddings and a new latent distribution, advancing naturalness and expressiveness.

Findings

Higher prosody variance compared to baseline

Increased speaker distinctiveness without reducing intelligibility

Enhanced speaker interpolation capabilities

Abstract

Text-to-speech systems recently achieved almost indistinguishable quality from human speech. However, the prosody of those systems is generally flatter than natural speech, producing samples with low expressiveness. Disentanglement of speaker id and prosody is crucial in text-to-speech systems to improve on naturalness and produce more variable syntheses. This paper proposes a new neural text-to-speech model that approaches the disentanglement problem by conditioning a Tacotron2-like architecture on flow-normalized speaker embeddings, and by substituting the reference encoder with a new learned latent distribution responsible for modeling the intra-sentence variability due to the prosody. By removing the reference encoder dependency, the speaker-leakage problem typically happening in this kind of systems disappears, producing more distinctive syntheses at inference time. The new model achieves significantly higher prosody variance than the baseline in a set of quantitative prosody features, as well as higher speaker distinctiveness, without decreasing the speaker intelligibility. Finally, we observe that the normalized speaker embeddings enable much richer speaker interpolations, substantially improving the distinctiveness of the new interpolated speakers.