Cross-modal Manifold Cutmix for Self-supervised Video Representation Learning

TL;DR

This paper introduces Cross-Modal Manifold Cutmix, a novel video augmentation method that combines videos across different modalities in feature space, improving self-supervised video representation learning with less data.

Contribution

The paper proposes a new cross-modal video mixing strategy, STC-mix, that enhances self-supervised learning by integrating videos across modalities in feature space.

Findings

STC-mix improves downstream task performance on UCF101 and HMDB51.

STC-mix achieves comparable results to existing methods with less training data.

Effective on datasets with limited domain knowledge, like NTU.

Abstract

Contrastive representation learning of videos highly relies on the availability of millions of unlabelled videos. This is practical for videos available on web but acquiring such large scale of videos for real-world applications is very expensive and laborious. Therefore, in this paper we focus on designing video augmentation for self-supervised learning, we first analyze the best strategy to mix videos to create a new augmented video sample. Then, the question remains, can we make use of the other modalities in videos for data mixing? To this end, we propose Cross-Modal Manifold Cutmix (CMMC) that inserts a video tesseract into another video tesseract in the feature space across two different modalities. We find that our video mixing strategy STC-mix, i.e. preliminary mixing of videos followed by CMMC across different modalities in a video, improves the quality of learned video representations. We conduct thorough experiments for two downstream tasks: action recognition and video retrieval on two small scale video datasets UCF101, and HMDB51. We also demonstrate the effectiveness of our STC-mix on NTU dataset where domain knowledge is limited. We show that the performance of our STC-mix on both the downstream tasks is on par with the other self-supervised approaches while requiring less training data.