SCVRL: Shuffled Contrastive Video Representation Learning

TL;DR

SCVRL introduces a contrastive learning framework that effectively captures both semantic and motion patterns in videos, leveraging a transformer-based network to outperform existing methods on multiple benchmarks.

Contribution

It reformulates the shuffling pretext task within a contrastive learning paradigm and demonstrates the effectiveness of transformers in learning motion in self-supervised video representations.

Findings

Outperforms CVRL on four benchmarks

Capable of learning both semantic and motion patterns

Uses a transformer-based network for video representation

Abstract

We propose SCVRL, a novel contrastive-based framework for self-supervised learning for videos. Differently from previous contrast learning based methods that mostly focus on learning visual semantics (e.g., CVRL), SCVRL is capable of learning both semantic and motion patterns. For that, we reformulate the popular shuffling pretext task within a modern contrastive learning paradigm. We show that our transformer-based network has a natural capacity to learn motion in self-supervised settings and achieves strong performance, outperforming CVRL on four benchmarks.