Language-based Audio Retrieval with Co-Attention Networks

TL;DR

This paper introduces a co-attention network framework for language-based audio retrieval, effectively aligning text and audio representations to improve retrieval accuracy on public datasets.

Contribution

The work proposes a cascaded co-attention architecture that enhances cross-modal semantic alignment for audio retrieval using natural language queries.

Findings

Achieves 16.6% improvement in mean Average Precision on Clotho dataset.

Achieves 15.1% improvement on AudioCaps dataset.

Outperforms previous state-of-the-art methods in the task.

Abstract

In recent years, user-generated audio content has proliferated across various media platforms, creating a growing need for efficient retrieval methods that allow users to search for audio clips using natural language queries. This task, known as language-based audio retrieval, presents significant challenges due to the complexity of learning semantic representations from heterogeneous data across both text and audio modalities. In this work, we introduce a novel framework for the language-based audio retrieval task that leverages co-attention mechanismto jointly learn meaningful representations from both modalities. To enhance the model's ability to capture fine-grained cross-modal interactions, we propose a cascaded co-attention architecture, where co-attention modules are stacked or iterated to progressively refine the semantic alignment between text and audio. Experiments conducted on two public datasets show that the proposed method can achieve better performance than the state-of-the-art method. Specifically, our best performed co-attention model achieves a 16.6% improvement in mean Average Precision on Clotho dataset, and a 15.1% improvement on AudioCaps.