Sample-Efficient Diffusion for Text-To-Speech Synthesis

TL;DR

This paper presents SESD, a novel diffusion-based speech synthesis method that achieves high-quality results with significantly less training data by leveraging a new architecture and latent space modeling.

Contribution

It introduces the U-Audio Transformer architecture and a latent diffusion approach for efficient, data-light speech synthesis in low-resource settings.

Findings

Outperforms state-of-the-art auto-regressive models in speech intelligibility.

Uses less than 1,000 hours of training data, significantly reducing data requirements.

Synthesizes more intelligible speech than VALL-E with only 2% of the training data.

Abstract

This work introduces Sample-Efficient Speech Diffusion (SESD), an algorithm for effective speech synthesis in modest data regimes through latent diffusion. It is based on a novel diffusion architecture, that we call U-Audio Transformer (U-AT), that efficiently scales to long sequences and operates in the latent space of a pre-trained audio autoencoder. Conditioned on character-aware language model representations, SESD achieves impressive results despite training on less than 1k hours of speech - far less than current state-of-the-art systems. In fact, it synthesizes more intelligible speech than the state-of-the-art auto-regressive model, VALL-E, while using less than 2% the training data.