CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis

TL;DR

CLEAR introduces a novel continuous latent autoregressive framework for zero-shot speech synthesis, achieving high-quality, low-latency, and streaming capable TTS by modeling continuous audio representations directly.

Contribution

The paper proposes a unified zero-shot TTS framework that models continuous audio representations with an enhanced variational autoencoder and a lightweight flow head, reducing inference latency and improving synthesis quality.

Findings

CLEAR achieves state-of-the-art results on LibriSpeech with 1.88% WER.

It has a low real-time factor of 0.29, enabling fast synthesis.

Supports streaming synthesis with 96ms delay.

Abstract

Autoregressive (AR) language models have emerged as powerful solutions for zero-shot text-to-speech (TTS) synthesis, capable of generating natural speech from a few seconds of audio prompts. However, conventional AR-based TTS systems relying on discrete audio tokens face the challenge of lossy compression during tokenization, requiring longer discrete token sequences to capture the same information as continuous ones, which adds inference latency and complicates AR modeling. To address this challenge, this paper proposes the Continuous Latent Autoregressive model (CLEAR), a unified zero-shot TTS framework that directly models continuous audio representations. More specifically, CLEAR introduces an enhanced variational autoencoder with shortcut connections, which achieves a high compression ratio to map waveforms into compact continuous latents. A lightweight MLP-based rectified flow head that operates independently for each hidden state is presented to model the continuous latent probability distribution, and trained jointly with the AR model within a single-stage framework. Experiments show that the proposed zero-shot CLEAR TTS can synthesize high-quality speech with low latency. Compared to state-of-the-art (SOTA) TTS models, CLEAR delivers competitive performance in robustness, speaker similarity and naturalness, while offering a lower real-time factor (RTF). In particular, CLEAR achieves SOTA results on the LibriSpeech test-clean dataset, with a word error rate of 1.88\% and an RTF of 0.29. Moreover, CLEAR facilitates streaming speech synthesis with a first-frame delay of 96ms, while maintaining high-quality speech synthesis.