RECAP: Retrieval-Augmented Audio Captioning

TL;DR

RECAP is a retrieval-augmented audio captioning system that leverages retrieved similar captions to generate accurate descriptions, demonstrating strong performance across domains without additional fine-tuning and enabling novel audio event captioning.

Contribution

The paper introduces RECAP, a novel retrieval-augmented approach for audio captioning that operates without fine-tuning and can handle unseen audio events and complex compositions.

Findings

Achieves competitive in-domain captioning performance.

Significantly improves out-of-domain captioning accuracy.

Demonstrates ability to caption novel and compositional audio events.

Abstract

We present RECAP (REtrieval-Augmented Audio CAPtioning), a novel and effective audio captioning system that generates captions conditioned on an input audio and other captions similar to the audio retrieved from a datastore. Additionally, our proposed method can transfer to any domain without the need for any additional fine-tuning. To generate a caption for an audio sample, we leverage an audio-text model CLAP to retrieve captions similar to it from a replaceable datastore, which are then used to construct a prompt. Next, we feed this prompt to a GPT-2 decoder and introduce cross-attention layers between the CLAP encoder and GPT-2 to condition the audio for caption generation. Experiments on two benchmark datasets, Clotho and AudioCaps, show that RECAP achieves competitive performance in in-domain settings and significant improvements in out-of-domain settings. Additionally, due to its capability to exploit a large text-captions-only datastore in a training-free fashion, RECAP shows unique capabilities of captioning novel audio events never seen during training and compositional audios with multiple events. To promote research in this space, we also release 150,000+ new weakly labeled captions for AudioSet, AudioCaps, and Clotho.