Zero-Shot Text-to-Speech from Continuous Text Streams

TL;DR

This paper introduces LiveSpeech 2, a streaming-aware zero-shot TTS model capable of generating continuous, synchronized speech from ongoing text streams with seamless chunk transitions, outperforming prior sentence-based systems.

Contribution

The paper presents LiveSpeech 2, a novel stream-aware zero-shot TTS model that supports infinite text streams, real-time synchronization, and smooth chunk transitions, using innovative sequence modeling and attention techniques.

Findings

Competitive with state-of-the-art zero-shot TTS models

Supports infinitely long speech generation from continuous text streams

Enables seamless transitions between speech chunks

Abstract

Existing zero-shot text-to-speech (TTS) systems are typically designed to process complete sentences and are constrained by the maximum duration for which they have been trained. However, in many streaming applications, texts arrive continuously in short chunks, necessitating instant responses from the system. We identify the essential capabilities required for chunk-level streaming and introduce LiveSpeech 2, a stream-aware model that supports infinitely long speech generation, text-audio stream synchronization, and seamless transitions between short speech chunks. To achieve these, we propose (1) adopting Mamba, a class of sequence modeling distinguished by linear-time decoding, which is augmented by cross-attention mechanisms for conditioning, (2) utilizing rotary positional embeddings in the computation of cross-attention, enabling the model to process an infinite text stream by sliding a window, and (3) decoding with semantic guidance, a technique that aligns speech with the transcript during inference with minimal overhead. Experimental results demonstrate that our models are competitive with state-of-the-art language model-based zero-shot TTS models, while also providing flexibility to support a wide range of streaming scenarios.