Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning

TL;DR

This paper introduces Space-Time Crop & Attend (STiCA), a novel method that enhances self-supervised video representation learning by efficiently applying spatial augmentations and transformer-based attention, achieving state-of-the-art results.

Contribution

The paper proposes Feature Crop for efficient spatial augmentation simulation and demonstrates the effectiveness of transformer-based attention in video representation learning.

Findings

Feature Crop enables scalable spatial augmentation in videos.

Transformer attention significantly improves video feature pooling.

STiCA achieves new state-of-the-art accuracy on benchmark datasets.

Abstract

The quality of the image representations obtained from self-supervised learning depends strongly on the type of data augmentations used in the learning formulation. Recent papers have ported these methods from still images to videos and found that leveraging both audio and video signals yields strong gains; however, they did not find that spatial augmentations such as cropping, which are very important for still images, work as well for videos. In this paper, we improve these formulations in two ways unique to the spatio-temporal aspect of videos. First, for space, we show that spatial augmentations such as cropping do work well for videos too, but that previous implementations, due to the high processing and memory cost, could not do this at a scale sufficient for it to work well. To address this issue, we first introduce Feature Crop, a method to simulate such augmentations much more efficiently directly in feature space. Second, we show that as opposed to naive average pooling, the use of transformer-based attention improves performance significantly, and is well suited for processing feature crops. Combining both of our discoveries into a new method, Space-Time Crop & Attend (STiCA) we achieve state-of-the-art performance across multiple video-representation learning benchmarks. In particular, we achieve new state-of-the-art accuracies of 67.0% on HMDB-51 and 93.1% on UCF-101 when pre-training on Kinetics-400.