Speak While You Think: Streaming Speech Synthesis During Text Generation

TL;DR

This paper introduces LLM2Speech, a streaming speech synthesis architecture that reduces latency during LLM text generation, enabling more natural voice conversations without sacrificing quality.

Contribution

It presents a novel streaming speech synthesis method that leverages LLM embeddings to mimic non-streaming models with significantly lower latency.

Findings

Maintains synthesis quality comparable to non-streaming models

Reduces latency to enable real-time voice conversations

Utilizes LLM hidden embeddings for effective streaming

Abstract

Large Language Models (LLMs) demonstrate impressive capabilities, yet interaction with these models is mostly facilitated through text. Using Text-To-Speech to synthesize LLM outputs typically results in notable latency, which is impractical for fluent voice conversations. We propose LLM2Speech, an architecture to synthesize speech while text is being generated by an LLM which yields significant latency reduction. LLM2Speech mimics the predictions of a non-streaming teacher model while limiting the exposure to future context in order to enable streaming. It exploits the hidden embeddings of the LLM, a by-product of the text generation that contains informative semantic context. Experimental results show that LLM2Speech maintains the teacher's quality while reducing the latency to enable natural conversations.