ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning

TL;DR

This paper introduces ACAV100M, a scalable method for automatically curating large-scale audio-visual datasets by maximizing mutual information, enabling effective self-supervised video representation learning without manual annotation.

Contribution

The authors propose a novel subset optimization approach for automatic dataset curation that significantly reduces manual effort and scales to 100 million videos with high audio-visual correspondence.

Findings

Self-supervised models trained on ACAV100M achieve competitive performance.

The approach effectively identifies videos with high audio-visual correspondence.

ACAV100M enables large-scale, automated dataset creation for video learning.

Abstract

The natural association between visual observations and their corresponding sound provides powerful self-supervisory signals for learning video representations, which makes the ever-growing amount of online videos an attractive source of training data. However, large portions of online videos contain irrelevant audio-visual signals because of edited/overdubbed audio, and models trained on such uncurated videos have shown to learn suboptimal representations. Therefore, existing approaches rely almost exclusively on datasets with predetermined taxonomies of semantic concepts, where there is a high chance of audio-visual correspondence. Unfortunately, constructing such datasets require labor intensive manual annotation and/or verification, which severely limits the utility of online videos for large-scale learning. In this work, we present an automatic dataset curation approach based on subset optimization where the objective is to maximize the mutual information between audio and visual channels in videos. We demonstrate that our approach finds videos with high audio-visual correspondence and show that self-supervised models trained on our data achieve competitive performances compared to models trained on existing manually curated datasets. The most significant benefit of our approach is scalability: We release ACAV100M that contains 100 million videos with high audio-visual correspondence, ideal for self-supervised video representation learning.