WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research

TL;DR

WavCaps is a large-scale, weakly-labelled audio captioning dataset created using a three-stage filtering process with ChatGPT, significantly advancing audio-language multimodal research by enabling better model training and outperforming previous SOTA systems.

Contribution

The paper introduces WavCaps, the first large-scale weakly-labelled audio captioning dataset, and demonstrates how ChatGPT can be used to filter and generate high-quality captions from noisy web data.

Findings

Models trained on WavCaps outperform previous SOTA models.

The dataset enables improved performance on multiple audio-language tasks.

ChatGPT effectively filters and transforms raw descriptions for dataset creation.

Abstract

The advancement of audio-language (AL) multimodal learning tasks has been significant in recent years. However, researchers face challenges due to the costly and time-consuming collection process of existing audio-language datasets, which are limited in size. To address this data scarcity issue, we introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions. We sourced audio clips and their raw descriptions from web sources and a sound event detection dataset. However, the online-harvested raw descriptions are highly noisy and unsuitable for direct use in tasks such as automated audio captioning. To overcome this issue, we propose a three-stage processing pipeline for filtering noisy data and generating high-quality captions, where ChatGPT, a large language model, is leveraged to filter and transform raw descriptions automatically. We conduct a comprehensive analysis of the characteristics of WavCaps dataset and evaluate it on multiple downstream audio-language multimodal learning tasks. The systems trained on WavCaps outperform previous state-of-the-art (SOTA) models by a significant margin. Our aspiration is for the WavCaps dataset we have proposed to facilitate research in audio-language multimodal learning and demonstrate the potential of utilizing ChatGPT to enhance academic research. Our dataset and codes are available at https://github.com/XinhaoMei/WavCaps.