Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization

TL;DR

This paper introduces a self-supervised approach to learn audio and video models by leveraging their natural synchronization, resulting in improved audio classification and action recognition without additional labels.

Contribution

It presents a novel self-supervised training scheme using synchronization cues, contrastive loss, and curriculum learning to enhance multi-sensory representations for audio and video tasks.

Findings

Audio features outperform state-of-the-art on DCASE2014 and ESC-50 benchmarks.

Self-supervised pretraining significantly improves video action recognition accuracy.

Contrastive learning with negative example selection is crucial for effective synchronization learning.

Abstract

There is a natural correlation between the visual and auditive elements of a video. In this work we leverage this connection to learn general and effective models for both audio and video analysis from self-supervised temporal synchronization. We demonstrate that a calibrated curriculum learning scheme, a careful choice of negative examples, and the use of a contrastive loss are critical ingredients to obtain powerful multi-sensory representations from models optimized to discern temporal synchronization of audio-video pairs. Without further finetuning, the resulting audio features achieve performance superior or comparable to the state-of-the-art on established audio classification benchmarks (DCASE2014 and ESC-50). At the same time, our visual subnet provides a very effective initialization to improve the accuracy of video-based action recognition models: compared to learning from scratch, our self-supervised pretraining yields a remarkable gain of +19.9% in action recognition accuracy on UCF101 and a boost of +17.7% on HMDB51.