Interleaved Speech-Text Language Models for Simple Streaming Text-to-Speech Synthesis

TL;DR

This paper presents IST-LM, a novel interleaved speech-text language model for zero-shot streaming TTS that simplifies training and maintains high performance with minimal latency.

Contribution

It introduces a new interleaved training method for streaming TTS that eliminates complex alignments and demonstrates effective performance through comprehensive analysis.

Findings

Optimal text-speech chunk ratio improves performance.

Key factors include token distance, future text access, and token precedence.

Streaming TTS performance closely matches non-streaming models.

Abstract

This paper introduces Interleaved Speech-Text Language Model (IST-LM) for zero-shot streaming Text-to-Speech (TTS). Unlike many previous approaches, IST-LM is directly trained on interleaved sequences of text and speech tokens with a fixed ratio, eliminating the need for additional efforts like forced alignment or complex designs. The ratio of text chunk size to speech chunk size is crucial for the performance of IST-LM. To explore this, we conducted a comprehensive series of statistical analyses on the training data and performed correlation analysis with the final performance, uncovering several key factors: 1) the distance between speech tokens and their corresponding text tokens, 2) the number of future text tokens accessible to each speech token, and 3) the frequency of speech tokens precedes their corresponding text tokens. Experimental results demonstrate how to achieve an optimal streaming TTS system with a limited performance gap compared to its non-streaming counterpart. IST-LM is conceptually simple and empirically powerful, enabling streaming TTS with minimal overhead while largely preserving performance, and offering broad potential for integration with real-time text streams from large language models.