RelayS2S: A Dual-Path Speculative Generation for Real-Time Dialogue

TL;DR

RelayS2S is a hybrid real-time dialogue system that combines a fast speculative response draft with a high-quality slow response, achieving low latency without sacrificing response quality.

Contribution

It introduces a dual-path architecture with a speculative fast path and a high-quality slow path, enabling real-time dialogue with minimal latency and high response quality.

Findings

Achieves P90 onset latency comparable to end-to-end models

Retains 99% of cascaded pipeline response quality

Scalable benefits as slow-path model size increases

Abstract

Real-time spoken dialogue systems face a fundamental tension between latency and response quality. End-to-end speech-to-speech (S2S) models respond immediately and naturally handle turn-taking, backchanneling, and interruption, but produce semantically weaker outputs. Cascaded pipelines (ASR -> LLM) deliver stronger responses at the cost of latency that grows with model size. We present RelayS2S, a hybrid architecture that runs two paths in parallel upon turn detection. The fast path -- a duplex S2S model -- speculatively drafts a short response prefix that is streamed immediately to TTS for low-latency audio onset, while continuing to monitor live audio events. The slow path -- a cascaded ASR -> LLM pipeline -- generates a higher-quality continuation conditioned on the committed prefix, producing a seamless utterance. A lightweight learned verifier gates the handoff, committing the prefix when appropriate or falling back gracefully to the slow path alone. Experiments show that RelayS2S achieves P90 onset latency comparable to the S2S model while retaining 99% cascaded response quality in average score, with benefits growing as the slow-path model scales. Because the prefix handoff requires no architectural modification to either component, RelayS2S serves as a lightweight, drop-in addition to existing cascaded pipelines. Our code and data are publicly available at: https://github.com/mailong25/relays2s