ATIR: Towards Audio-Text Interleaved Contextual Retrieval

TL;DR

This paper introduces the ATIR task and benchmark for audio-text interleaved retrieval, evaluates models on it, and proposes a novel token compression method to improve multimodal retrieval performance.

Contribution

The work defines a new interleaved audio-text retrieval task, creates a comprehensive benchmark, and proposes a novel token compression technique for multimodal models.

Findings

ATIR model outperforms strong baselines in experiments.

The benchmark unifies four types of contextual retrieval tasks.

Token compression alleviates excessive audio token issues in MLLM models.

Abstract

Audio carries richer information than text, including emotion, speaker traits, and environmental context, while also enabling lower-latency processing compared to speech-to-text pipelines. However, recent multimodal information retrieval research has predominantly focused on images, largely overlooking audio, especially in the setting of interleaved audio-text contextual retrieval. In this work, we introduce the Audio-Text Interleaved contextual Retrieval (ATIR) task, where queries can alternate between audio and text modalities. We construct an ATIR benchmark by integrating several Automatic Speech Recognition (ASR), QA, and retrieval datasets, ultimately unifying four types of contextual retrieval tasks. This benchmark substantially addresses the limitations of existing audio retrieval datasets in semantic retrieval. To study this task, we evaluate several off-the-shelf retrievers and train our ATIR model based on a Multimodal Large Language Model (MLLM). We further introduce a novel token compression mechanism that is orthogonal to existing compression methods, thereby alleviating the issue of excessive audio tokens in MLLM-based ATIR models. Experimental results demonstrate that our ATIR model achieves substantial improvements over strong baselines.