Text-to-speech synthesis based on latent variable conversion using diffusion probabilistic model and variational autoencoder

TL;DR

This paper introduces a novel text-to-speech synthesis approach combining diffusion probabilistic models with variational autoencoders to improve robustness and flexibility in speech generation from text.

Contribution

It proposes a new latent variable conversion framework integrating diffusion models with VAE for TTS, enhancing robustness to alignment errors and orthography issues.

Findings

Robustness to poor orthography and alignment errors

Effective latent variable modeling with diffusion and VAE

Flexible incorporation of various latent feature extractors

Abstract

Text-to-speech synthesis (TTS) is a task to convert texts into speech. Two of the factors that have been driving TTS are the advancements of probabilistic models and latent representation learning. We propose a TTS method based on latent variable conversion using a diffusion probabilistic model and the variational autoencoder (VAE). In our TTS method, we use a waveform model based on VAE, a diffusion model that predicts the distribution of latent variables in the waveform model from texts, and an alignment model that learns alignments between the text and speech latent sequences. Our method integrates diffusion with VAE by modeling both mean and variance parameters with diffusion, where the target distribution is determined by approximation from VAE. This latent variable conversion framework potentially enables us to flexibly incorporate various latent feature extractors. Our experiments show that our method is robust to linguistic labels with poor orthography and alignment errors.