Autoregressive Diffusion Transformer for Text-to-Speech Synthesis

TL;DR

This paper introduces ARDiT, a diffusion transformer model that encodes audio as continuous vectors for high-quality, low-latency text-to-speech synthesis, surpassing state-of-the-art performance in zero-shot scenarios.

Contribution

The novel ARDiT model uses continuous audio encoding and diffusion-based autoregressive generation, enabling near-flawless speech reconstruction and efficient sampling with reduced latency.

Findings

ARDiT achieves state-of-the-art zero-shot TTS performance.

Integral Kullback-Leibler divergence improves sample quality.

Model generates 170 ms of 24 kHz speech per step with minimal degradation.

Abstract

Audio language models have recently emerged as a promising approach for various audio generation tasks, relying on audio tokenizers to encode waveforms into sequences of discrete symbols. Audio tokenization often poses a necessary compromise between code bitrate and reconstruction accuracy. When dealing with low-bitrate audio codes, language models are constrained to process only a subset of the information embedded in the audio, which in turn restricts their generative capabilities. To circumvent these issues, we propose encoding audio as vector sequences in continuous space $\mathbb R^d$ and autoregressively generating these sequences using a decoder-only diffusion transformer (ARDiT). Our findings indicate that ARDiT excels in zero-shot text-to-speech and exhibits performance that compares to or even surpasses that of state-of-the-art models. High-bitrate continuous speech representation enables almost flawless reconstruction, allowing our model to achieve nearly perfect speech editing. Our experiments reveal that employing Integral Kullback-Leibler (IKL) divergence for distillation at each autoregressive step significantly boosts the perceived quality of the samples. Simultaneously, it condenses the iterative sampling process of the diffusion model into a single step. Furthermore, ARDiT can be trained to predict several continuous vectors in one step, significantly reducing latency during sampling. Impressively, one of our models can generate $170$ ms of $24$ kHz speech per evaluation step with minimal degradation in performance. Audio samples are available at http://ardit-tts.github.io/ .