Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization

TL;DR

This paper introduces a self-supervised learning method that emphasizes local motion dynamics in videos by using synthetic tubelet trajectories, leading to data-efficient and generalizable video representations.

Contribution

It presents a novel approach that learns motion similarities through synthetic tubelet trajectories, improving data efficiency and generalization in video representation learning.

Findings

Maintains performance with only 25% of pretraining videos.

Demonstrates strong results across 10 diverse downstream tasks.

Shows improved generalization to new domains and fine-grained actions.

Abstract

We propose a self-supervised method for learning motion-focused video representations. Existing approaches minimize distances between temporally augmented videos, which maintain high spatial similarity. We instead propose to learn similarities between videos with identical local motion dynamics but an otherwise different appearance. We do so by adding synthetic motion trajectories to videos which we refer to as tubelets. By simulating different tubelet motions and applying transformations, such as scaling and rotation, we introduce motion patterns beyond what is present in the pretraining data. This allows us to learn a video representation that is remarkably data efficient: our approach maintains performance when using only 25\% of the pretraining videos. Experiments on 10 diverse downstream settings demonstrate our competitive performance and generalizability to new domains and fine-grained actions.