Spatio-Temporal Crop Aggregation for Video Representation Learning

TL;DR

SCALE introduces a scalable, self-supervised method for video representation learning that efficiently constructs long-range features from sparse clip inputs, achieving state-of-the-art results in action classification.

Contribution

The paper presents a novel spatio-temporal crop aggregation approach that leverages sparse inputs and dimensionality reduction for efficient, high-performance video representation learning.

Findings

Achieves state-of-the-art performance on action classification datasets.

Efficient training due to sparse input and loss design.

Effective transfer learning with linear, non-linear, and KNN probing.

Abstract

We propose Spatio-temporal Crop Aggregation for video representation LEarning (SCALE), a novel method that enjoys high scalability at both training and inference time. Our model builds long-range video features by learning from sets of video clip-level features extracted with a pre-trained backbone. To train the model, we propose a self-supervised objective consisting of masked clip feature prediction. We apply sparsity to both the input, by extracting a random set of video clips, and to the loss function, by only reconstructing the sparse inputs. Moreover, we use dimensionality reduction by working in the latent space of a pre-trained backbone applied to single video clips. These techniques make our method not only extremely efficient to train but also highly effective in transfer learning. We demonstrate that our video representation yields state-of-the-art performance with linear, non-linear, and KNN probing on common action classification and video understanding datasets.