DiscreTalk: Text-to-Speech as a Machine Translation Problem

TL;DR

DiscreTalk introduces a novel end-to-end text-to-speech system that models speech as a machine translation task using discrete symbols, enabling the application of NMT techniques and improving naturalness over traditional TTS models.

Contribution

The paper presents a new TTS approach combining VQ-VAE and Transformer NMT, eliminating the need for hyperparameter tuning and reducing over-smoothing issues.

Findings

Outperforms conventional Transformer-TTS in naturalness

Achieves performance comparable to VQ-VAE reconstruction

Utilizes NMT techniques like beam search and subword units

Abstract

This paper proposes a new end-to-end text-to-speech (E2E-TTS) model based on neural machine translation (NMT). The proposed model consists of two components; a non-autoregressive vector quantized variational autoencoder (VQ-VAE) model and an autoregressive Transformer-NMT model. The VQ-VAE model learns a mapping function from a speech waveform into a sequence of discrete symbols, and then the Transformer-NMT model is trained to estimate this discrete symbol sequence from a given input text. Since the VQ-VAE model can learn such a mapping in a fully-data-driven manner, we do not need to consider hyperparameters of the feature extraction required in the conventional E2E-TTS models. Thanks to the use of discrete symbols, we can use various techniques developed in NMT and automatic speech recognition (ASR) such as beam search, subword units, and fusions with a language model. Furthermore, we can avoid an over smoothing problem of predicted features, which is one of the common issues in TTS. The experimental evaluation with the JSUT corpus shows that the proposed method outperforms the conventional Transformer-TTS model with a non-autoregressive neural vocoder in naturalness, achieving the performance comparable to the reconstruction of the VQ-VAE model.