SEAL: Speech Embedding Alignment Learning for Speech Large Language Model with Retrieval-Augmented Generation

TL;DR

This paper introduces SEAL, a unified speech embedding framework for speech large language models that reduces latency and improves retrieval accuracy without relying on intermediate text representations.

Contribution

The paper presents a novel unified embedding approach with shared scaling for speech and text, enhancing retrieval efficiency and accuracy in speech LLMs over traditional two-stage methods.

Findings

Reduces pipeline latency by 50%

Achieves higher retrieval accuracy

Robust across diverse acoustic conditions

Abstract

Embedding-based retrieval models have made significant strides in retrieval-augmented generation (RAG) techniques for text and multimodal large language models (LLMs) applications. However, when it comes to speech larage language models (SLLMs), these methods are limited to a two-stage process, where automatic speech recognition (ASR) is combined with text-based retrieval. This sequential architecture suffers from high latency and error propagation. To address these limitations, we propose a unified embedding framework that eliminates the need for intermediate text representations. Specifically, the framework includes separate speech and text encoders, followed by a shared scaling layer that maps both modalities into a common embedding space. Our model reduces pipeline latency by 50\% while achieving higher retrieval accuracy compared to traditional two-stage methods. We also provide a theoretical analysis of the challenges inherent in end-to-end speech retrieval and introduce architectural principles for effective speech-to-document matching. Extensive experiments demonstrate the robustness of our approach across diverse acoustic conditions and speaker variations, paving the way for a new paradigm in multimodal SLLMs retrieval systems.