Cross-modal supervised learning for better acoustic representations

TL;DR

This paper leverages large-scale video data with machine-generated labels to improve acoustic representations, achieving significant performance gains on standard audio classification benchmarks.

Contribution

It introduces a method to utilize synchronized vision-audio data with machine labels for training improved acoustic models, surpassing state-of-the-art results.

Findings

Achieved significant performance improvements on external benchmarks.

Collected 15 million video samples with automatic annotations.

Enhanced VGGish model with better results.

Abstract

Obtaining large-scale human-labeled datasets to train acoustic representation models is a very challenging task. On the contrary, we can easily collect data with machine-generated labels. In this work, we propose to exploit machine-generated labels to learn better acoustic representations, based on the synchronization between vision and audio. Firstly, we collect a large-scale video dataset with 15 million samples, which totally last 16,320 hours. Each video is 3 to 5 seconds in length and annotated automatically by publicly available visual and audio classification models. Secondly, we train various classical convolutional neural networks (CNNs) including VGGish, ResNet 50 and Mobilenet v2. We also make several improvements to VGGish and achieve better results. Finally, we transfer our models on three external standard benchmarks for audio classification task, and achieve significant performance boost over the state-of-the-art results. Models and codes are available at: https://github.com/Deeperjia/vgg-like-audio-models.