Estimated Audio-Caption Correspondences Improve Language-Based Audio Retrieval

TL;DR

This paper introduces a two-stage training method for audio-caption retrieval that estimates correspondences between audio and text, improving retrieval accuracy by leveraging predicted matches instead of random pairings.

Contribution

The authors propose a novel two-stage training approach that uses estimated audio-caption correspondences to enhance retrieval performance, outperforming existing methods.

Findings

Improved retrieval performance on ClothoV2 and AudioCaps benchmarks.

Outperforms state-of-the-art by 1.6 percentage points in mAP@10 on ClothoV2.

Effective even with a single model generating and learning from estimated correspondences.

Abstract

Dual-encoder-based audio retrieval systems are commonly optimized with contrastive learning on a set of matching and mismatching audio-caption pairs. This leads to a shared embedding space in which corresponding items from the two modalities end up close together. Since audio-caption datasets typically only contain matching pairs of recordings and descriptions, it has become common practice to create mismatching pairs by pairing the audio with a caption randomly drawn from the dataset. This is not ideal because the randomly sampled caption could, just by chance, partly or entirely describe the audio recording. However, correspondence information for all possible pairs is costly to annotate and thus typically unavailable; we, therefore, suggest substituting it with estimated correspondences. To this end, we propose a two-staged training procedure in which multiple retrieval models are first trained as usual, i.e., without estimated correspondences. In the second stage, the audio-caption correspondences predicted by these models then serve as prediction targets. We evaluate our method on the ClothoV2 and the AudioCaps benchmark and show that it improves retrieval performance, even in a restricting self-distillation setting where a single model generates and then learns from the estimated correspondences. We further show that our method outperforms the current state of the art by 1.6 pp. mAP@10 on the ClothoV2 benchmark.