End-to-End Text-to-Speech using Latent Duration based on VQ-VAE

TL;DR

This paper introduces a novel end-to-end text-to-speech framework that models explicit duration as a discrete latent variable using VQ-VAE, enabling joint optimization and improved alignment in TTS systems.

Contribution

The paper proposes a new TTS approach incorporating duration as a discrete latent variable via conditional VQ-VAE, with a theoretical basis and joint optimization from scratch.

Findings

Achieved naturalness ratings between soft-attention and explicit duration methods.

Demonstrated effective explicit duration modeling with a variational autoencoder.

Validated the approach through listening tests comparing with existing TTS methods.

Abstract

Explicit duration modeling is a key to achieving robust and efficient alignment in text-to-speech synthesis (TTS). We propose a new TTS framework using explicit duration modeling that incorporates duration as a discrete latent variable to TTS and enables joint optimization of whole modules from scratch. We formulate our method based on conditional VQ-VAE to handle discrete duration in a variational autoencoder and provide a theoretical explanation to justify our method. In our framework, a connectionist temporal classification (CTC) -based force aligner acts as the approximate posterior, and text-to-duration works as the prior in the variational autoencoder. We evaluated our proposed method with a listening test and compared it with other TTS methods based on soft-attention or explicit duration modeling. The results showed that our systems rated between soft-attention-based methods (Transformer-TTS, Tacotron2) and explicit duration modeling-based methods (Fastspeech).