Multi-Speaker End-to-End Speech Synthesis

TL;DR

This paper introduces a multi-speaker end-to-end speech synthesis model based on ClariNet, which uses trainable speaker embeddings to generate high-fidelity speech for multiple voices, outperforming existing systems.

Contribution

The paper extends ClariNet to multi-speaker synthesis by incorporating speaker embeddings and demonstrates improved naturalness over state-of-the-art models.

Findings

Outperforms existing systems in speech naturalness

Uses shared speaker embeddings across model components

Achieves high-fidelity multi-speaker speech synthesis

Abstract

In this work, we extend ClariNet (Ping et al., 2019), a fully end-to-end speech synthesis model (i.e., text-to-wave), to generate high-fidelity speech from multiple speakers. To model the unique characteristic of different voices, low dimensional trainable speaker embeddings are shared across each component of ClariNet and trained together with the rest of the model. We demonstrate that the multi-speaker ClariNet outperforms state-of-the-art systems in terms of naturalness, because the whole model is jointly optimized in an end-to-end manner.