Killing two birds with one stone: Can an audio captioning system also be used for audio-text retrieval?

TL;DR

This paper explores whether an audio captioning system can be repurposed for audio-text retrieval without additional training, showing promising results and potential for multi-task audio understanding.

Contribution

It demonstrates that an unmodified AAC system can effectively perform ATR tasks, achieving competitive results without fine-tuning or external data.

Findings

AAC system achieves high captioning scores on Clotho and AudioCaps.

The system attains a Text-to-Audio R@1 of 0.382 on AudioCaps, surpassing some state-of-the-art methods.

Normalizing loss values improves Audio-to-Text retrieval performance.

Abstract

Automated Audio Captioning (AAC) aims to develop systems capable of describing an audio recording using a textual sentence. In contrast, Audio-Text Retrieval (ATR) systems seek to find the best matching audio recording(s) for a given textual query (Text-to-Audio) or vice versa (Audio-to-Text). These tasks require different types of systems: AAC employs a sequence-to-sequence model, while ATR utilizes a ranking model that compares audio and text representations within a shared projection subspace. However, this work investigates the relationship between AAC and ATR by exploring the ATR capabilities of an unmodified AAC system, without fine-tuning for the new task. Our AAC system consists of an audio encoder (ConvNeXt-Tiny) trained on AudioSet for audio tagging, and a transformer decoder responsible for generating sentences. For AAC, it achieves a high SPIDEr-FL score of 0.298 on Clotho and 0.472 on AudioCaps on average. For ATR, we propose using the standard Cross-Entropy loss values obtained for any audio/caption pair. Experimental results on the Clotho and AudioCaps datasets demonstrate decent recall values using this simple approach. For instance, we obtained a Text-to-Audio R@1 value of 0.382 for Au-dioCaps, which is above the current state-of-the-art method without external data. Interestingly, we observe that normalizing the loss values was necessary for Audio-to-Text retrieval.