Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis

TL;DR

This paper introduces a continuous autoregressive speech synthesis model combining VAE-based speech representations with stochastic monotonic alignment, outperforming state-of-the-art models with fewer parameters.

Contribution

It presents a novel autoregressive approach using continuous latent speech representations and stochastic monotonic alignment, simplifying training and improving performance.

Findings

Outperforms VALL-E in subjective and objective evaluations

Uses only 10.3% of VALL-E's parameters

Leverages continuous speech representations for efficiency

Abstract

We propose a novel autoregressive modeling approach for speech synthesis, combining a variational autoencoder (VAE) with a multi-modal latent space and an autoregressive model that uses Gaussian Mixture Models (GMM) as the conditional probability distribution. Unlike previous methods that rely on residual vector quantization, our model leverages continuous speech representations from the VAE's latent space, greatly simplifying the training and inference pipelines. We also introduce a stochastic monotonic alignment mechanism to enforce strict monotonic alignments. Our approach significantly outperforms the state-of-the-art autoregressive model VALL-E in both subjective and objective evaluations, achieving these results with only 10.3\% of VALL-E's parameters. This demonstrates the potential of continuous speech language models as a more efficient alternative to existing quantization-based speech language models. Sample audio can be found at https://tinyurl.com/gmm-lm-tts.