Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorial

TL;DR

This paper provides a comprehensive tutorial on building self-hosted enterprise-grade realtime voice agents, focusing on a cascaded pipeline approach due to current limitations of end-to-end models, and includes a complete codebase.

Contribution

It introduces a practical, step-by-step tutorial for constructing self-hosted realtime voice agents using existing components, filling a gap in accessible technical guidance.

Findings

Cascaded pipeline (STT → LLM → TTS) is the practical architecture for self-hosted realtime voice agents.

Achieved a time-to-first-audio of approximately 730-755ms with full function calling support.

Evaluated various configurations of Qwen3-Omni, highlighting current limitations for real-time self-hosted deployment.

Abstract

We present a technical tutorial for building enterprise-grade realtime voice agents from first principles. While end-to-end speech-to-speech models may ultimately provide the best latency for voice agents, fully self-hosted end-to-end solutions are not yet available. We evaluate the closest candidate, Qwen3-Omni, across three configurations: its cloud-only DashScope Realtime API achieves $\sim$702ms audio-to-audio latency with streaming, but is not self-hostable; its local vLLM deployment supports only the Thinker (text generation from audio, 516ms), not the Talker (audio synthesis); and its local Transformers deployment runs the full pipeline but at $\sim$146s -- far too slow for realtime. The cascaded streaming pipeline (STT $\rightarrow$ LLM $\rightarrow$ TTS) therefore remains the practical architecture for self-hosted realtime voice agents, and the focus of this tutorial. We build a complete voice agent using Deepgram (streaming STT), vLLM-served LLMs with function calling (streaming text generation), and ElevenLabs (streaming TTS), achieving a measured time-to-first-audio of 755ms (best case 729ms) with full function calling support. We release the full codebase as a 9-chapter progressive tutorial with working, tested code for every component.