WavRAG: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Models

TL;DR

WavRAG introduces an end-to-end audio-integrated retrieval augmented generation framework that processes raw audio directly, enabling more efficient and effective spoken dialogue models with hybrid knowledge retrieval.

Contribution

It is the first framework to support native audio processing in RAG, bypassing ASR, and integrating audio with text for improved spoken dialogue capabilities.

Findings

Achieves comparable retrieval performance to ASR-based methods

Provides 10x faster retrieval speed

Extends RAG capabilities to audio modality

Abstract

Retrieval Augmented Generation (RAG) has gained widespread adoption owing to its capacity to empower large language models (LLMs) to integrate external knowledge. However, existing RAG frameworks are primarily designed for text-based LLMs and rely on Automatic Speech Recognition to process speech input, which discards crucial audio information, risks transcription errors, and increases computational overhead. Therefore, we introduce WavRAG, the first retrieval augmented generation framework with native, end-to-end audio support. WavRAG offers two key features: 1) Bypassing ASR, WavRAG directly processes raw audio for both embedding and retrieval. 2) WavRAG integrates audio and text into a unified knowledge representation. Specifically, we propose the WavRetriever to facilitate the retrieval from a text-audio hybrid knowledge base, and further enhance the in-context capabilities of spoken dialogue models through the integration of chain-of-thought reasoning. In comparison to state-of-the-art ASR-Text RAG pipelines, WavRAG achieves comparable retrieval performance while delivering a 10x acceleration. Furthermore, WavRAG's unique text-audio hybrid retrieval capability extends the boundaries of RAG to the audio modality.