Enhancing Retrieval-Augmented Audio Captioning with Generation-Assisted Multimodal Querying and Progressive Learning

TL;DR

This paper introduces a novel multimodal querying method and progressive learning strategy to enhance retrieval-augmented audio captioning, achieving state-of-the-art results on multiple benchmarks.

Contribution

It proposes Generation-Assisted Multimodal Querying and a progressive training approach to improve retrieval-augmented audio captioning performance.

Findings

Achieves state-of-the-art results on AudioCaps, Clotho, and Auto-ACD datasets.

Demonstrates the effectiveness of multimodal queries over unimodal queries.

Shows that progressive learning enhances model training and performance.

Abstract

Retrieval-augmented generation can improve audio captioning by incorporating relevant audio-text pairs from a knowledge base. Existing methods typically rely solely on the input audio as a unimodal retrieval query. In contrast, we propose Generation-Assisted Multimodal Querying, which generates a text description of the input audio to enable multimodal querying. This approach aligns the query modality with the audio-text structure of the knowledge base, leading to more effective retrieval. Furthermore, we introduce a novel progressive learning strategy that gradually increases the number of interleaved audio-text pairs to enhance the training process. Our experiments on AudioCaps, Clotho, and Auto-ACD demonstrate that our approach achieves state-of-the-art results across these benchmarks.