PredGen: Accelerated Inference of Large Language Models through Input-Time Speculation for Real-Time Speech Interaction

TL;DR

PredGen introduces a speculative decoding framework that predicts and generates candidate responses during user speech, significantly reducing latency in real-time speech interactions with large language models.

Contribution

This paper presents PredGen, a novel input-time speculative decoding method that accelerates LLM inference for real-time speech applications, reducing latency by around 2x.

Findings

Latency reduced by approximately 2x

Minimal additional computation cost incurred

Effective across various use cases

Abstract

Large Language Models (LLMs) are widely used in real-time voice chat applications, typically in combination with text-to-speech (TTS) systems to generate audio responses. However, their large size often leads to noticeable latency between the end of user input and the start of audio output, resulting in suboptimal user experiences. This latency is particularly evident when LLMs are deployed as single-user voice assistants on consumer-grade hardware with limited computing capacity. We discovered that this latency is primarily dominated by the time it takes for the LLMs to generate the first sentence, which is required as input by the TTS systems that synthesize audio responses on a sentence-by-sentence basis. To address this bottleneck, we propose Predictive Generation (PredGen), a novel framework that mitigates-or even eliminates-this delay through speculative decoding at input time. PredGen generates candidate responses while the user is still speaking, enabling the system to begin TTS processing with minimal delay. Simulated experiments on the Lmsys and MT-Bench datasets show that the proposed method can effectively reduce the latency by around 2x across a wide range of use cases, while incurring only minimal additional computation cost at input time-computation that would otherwise go unused.