StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modeling

TL;DR

StreamMel introduces a novel single-stage streaming TTS system that enables real-time, high-quality, zero-shot speech synthesis by interleaving text and acoustic modeling, outperforming existing methods in quality and latency.

Contribution

It is the first to propose a single-stage, interleaved autoregressive TTS framework that models continuous mel-spectrograms for real-time zero-shot speech synthesis.

Findings

Outperforms existing streaming TTS baselines in quality and latency.

Achieves performance comparable to offline systems in real-time generation.

Supports integration with real-time speech large language models.

Abstract

Recent advances in zero-shot text-to-speech (TTS) synthesis have achieved high-quality speech generation for unseen speakers, but most systems remain unsuitable for real-time applications because of their offline design. Current streaming TTS paradigms often rely on multi-stage pipelines and discrete representations, leading to increased computational cost and suboptimal system performance. In this work, we propose StreamMel, a pioneering single-stage streaming TTS framework that models continuous mel-spectrograms. By interleaving text tokens with acoustic frames, StreamMel enables low-latency, autoregressive synthesis while preserving high speaker similarity and naturalness. Experiments on LibriSpeech demonstrate that StreamMel outperforms existing streaming TTS baselines in both quality and latency. It even achieves performance comparable to offline systems while supporting efficient real-time generation, showcasing broad prospects for integration with real-time speech large language models. Audio samples are available at: https://aka.ms/StreamMel.