End-to-End Adversarial Text-to-Speech

TL;DR

This paper introduces an end-to-end adversarial text-to-speech model that directly converts text or phonemes into raw speech audio, achieving high quality with a simple, efficient architecture.

Contribution

It presents a novel, fully end-to-end, feed-forward TTS model using adversarial training and differentiable alignment, eliminating the need for multi-stage pipelines.

Findings

Achieves mean opinion score over 4 out of 5, comparable to state-of-the-art.

Uses soft dynamic time warping for temporal variation in spectrogram prediction.

Operates efficiently with a single, differentiable model.

Abstract

Modern text-to-speech synthesis pipelines typically involve multiple processing stages, each of which is designed or learnt independently from the rest. In this work, we take on the challenging task of learning to synthesise speech from normalised text or phonemes in an end-to-end manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs. Our proposed generator is feed-forward and thus efficient for both training and inference, using a differentiable alignment scheme based on token length prediction. It learns to produce high fidelity audio through a combination of adversarial feedback and prediction losses constraining the generated audio to roughly match the ground truth in terms of its total duration and mel-spectrogram. To allow the model to capture temporal variation in the generated audio, we employ soft dynamic time warping in the spectrogram-based prediction loss. The resulting model achieves a mean opinion score exceeding 4 on a 5 point scale, which is comparable to the state-of-the-art models relying on multi-stage training and additional supervision.