LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generation

TL;DR

LA-RAG introduces a retrieval-augmented approach that enhances LLM-based ASR accuracy by leveraging token-level speech datastores and speech-to-speech retrieval, especially effective for diverse accents.

Contribution

It proposes LA-RAG, a novel retrieval-augmented generation framework that improves speech recognition accuracy under varied acoustic conditions using token-level datastores and retrieval mechanisms.

Findings

Significant ASR accuracy improvements on Mandarin and Chinese dialect datasets.

Effective handling of accent variations through retrieval-augmented methods.

Validation of LA-RAG's effectiveness over existing approaches.

Abstract

Recent advancements in integrating speech information into large language models (LLMs) have significantly improved automatic speech recognition (ASR) accuracy. However, existing methods often constrained by the capabilities of the speech encoders under varied acoustic conditions, such as accents. To address this, we propose LA-RAG, a novel Retrieval-Augmented Generation (RAG) paradigm for LLM-based ASR. LA-RAG leverages fine-grained token-level speech datastores and a speech-to-speech retrieval mechanism to enhance ASR accuracy via LLM in-context learning (ICL) capabilities. Experiments on Mandarin and various Chinese dialect datasets demonstrate significant improvements in ASR accuracy compared to existing methods, validating the effectiveness of our approach, especially in handling accent variations.