Improving Text-To-Audio Models with Synthetic Captions

TL;DR

This paper introduces an audio captioning pipeline using an audio language model to generate synthetic captions, enhancing training data for text-to-audio models and significantly improving audio generation quality.

Contribution

It presents a novel audio captioning pipeline with an audio language model, enabling large-scale synthetic caption generation for audio datasets.

Findings

Synthetic captions improve text-to-audio model performance

Achieved state-of-the-art results on AudioCaps and MusicCaps

Demonstrated scalability and diversity in caption synthesis

Abstract

It is an open challenge to obtain high quality training data, especially captions, for text-to-audio models. Although prior methods have leveraged \textit{text-only language models} to augment and improve captions, such methods have limitations related to scale and coherence between audio and captions. In this work, we propose an audio captioning pipeline that uses an \textit{audio language model} to synthesize accurate and diverse captions for audio at scale. We leverage this pipeline to produce a dataset of synthetic captions for AudioSet, named \texttt{AF-AudioSet}, and then evaluate the benefit of pre-training text-to-audio models on these synthetic captions. Through systematic evaluations on AudioCaps and MusicCaps, we find leveraging our pipeline and synthetic captions leads to significant improvements on audio generation quality, achieving a new \textit{state-of-the-art}.