On Negative Sampling for Audio-Visual Contrastive Learning from Movies

TL;DR

This paper investigates the impact of within-movie negative sampling in audio-visual contrastive learning from uncurated long-form videos, revealing that tailored sampling strategies improve representation quality for downstream tasks.

Contribution

It introduces a novel focus on within-movie negative sampling for contrastive learning from movies, highlighting the importance of semantic diversity and non-semantic consistency in long-form content.

Findings

Within-movie negative sampling enhances representation quality.

Training on uncurated movies transfers well to action recognition.

Modified sampling strategies outperform prior short-video methods.

Abstract

The abundance and ease of utilizing sound, along with the fact that auditory clues reveal a plethora of information about what happens in a scene, make the audio-visual space an intuitive choice for representation learning. In this paper, we explore the efficacy of audio-visual self-supervised learning from uncurated long-form content i.e movies. Studying its differences with conventional short-form content, we identify a non-i.i.d distribution of data, driven by the nature of movies. Specifically, we find long-form content to naturally contain a diverse set of semantic concepts (semantic diversity), where a large portion of them, such as main characters and environments often reappear frequently throughout the movie (reoccurring semantic concepts). In addition, movies often contain content-exclusive artistic artifacts, such as color palettes or thematic music, which are strong signals for uniquely distinguishing a movie (non-semantic consistency). Capitalizing on these observations, we comprehensively study the effect of emphasizing within-movie negative sampling in a contrastive learning setup. Our view is different from those of prior works who consider within-video positive sampling, inspired by the notion of semantic persistency over time, and operate in a short-video regime. Our empirical findings suggest that, with certain modifications, training on uncurated long-form videos yields representations which transfer competitively with the state-of-the-art to a variety of action recognition and audio classification tasks.