VoiceAgentRAG: Solving the RAG Latency Bottleneck in Real-Time Voice Agents Using Dual-Agent Architectures

TL;DR

VoiceAgentRAG introduces a dual-agent architecture for real-time voice agents, significantly reducing latency by pre-fetching relevant data with a background agent and enabling rapid response generation from a semantic cache.

Contribution

The paper proposes a novel dual-agent memory router that decouples retrieval from response generation, improving latency in real-time voice agents.

Findings

Reduces response latency in voice agents

Pre-fetching improves retrieval efficiency

Achieves near-instantaneous responses on cache hits

Abstract

We present VoiceAgentRAG, an open-source dual-agent memory router that decouples retrieval from response generation. A background Slow Thinker agent continuously monitors the conversation stream, predicts likely follow-up topics using an LLM, and pre-fetches relevant document chunks into a FAISS-backed semantic cache. A foreground Fast Talker agent reads only from this sub-millisecond cache, bypassing the vector database entirely on cache hits.