Synth-AC: Enhancing Audio Captioning with Synthetic Supervision

TL;DR

Synth-AC introduces a framework that uses synthetic audio generated from text to improve audio captioning models, addressing data scarcity by leveraging cross-domain data and generative models.

Contribution

The paper presents Synth-AC, a novel approach that creates synthetic text-audio pairs using audio generative models to enhance audio captioning performance.

Findings

Synth-AC improves captioning accuracy on benchmark datasets.

Synthetic data augmentation leads to significant performance gains.

The framework is adaptable to various existing models.

Abstract

Data-driven approaches hold promise for audio captioning. However, the development of audio captioning methods can be biased due to the limited availability and quality of text-audio data. This paper proposes a SynthAC framework, which leverages recent advances in audio generative models and commonly available text corpus to create synthetic text-audio pairs, thereby enhancing text-audio representation. Specifically, the text-to-audio generation model, i.e., AudioLDM, is used to generate synthetic audio signals with captions from an image captioning dataset. Our SynthAC expands the availability of well-annotated captions from the text-vision domain to audio captioning, thus enhancing text-audio representation by learning relations within synthetic text-audio pairs. Experiments demonstrate that our SynthAC framework can benefit audio captioning models by incorporating well-annotated text corpus from the text-vision domain, offering a promising solution to the challenge caused by data scarcity. Furthermore, SynthAC can be easily adapted to various state-of-the-art methods, leading to substantial performance improvements.