ASCNet: Self-supervised Video Representation Learning with Appearance-Speed Consistency

TL;DR

ASCNet introduces a self-supervised video representation learning method focusing on appearance and speed consistency, outperforming supervised models on action recognition without using negative pairs or extra modalities.

Contribution

The paper proposes a novel self-supervised learning framework that leverages appearance-speed consistency tasks, eliminating the need for negative pairs and improving downstream video understanding tasks.

Findings

Achieves 90.8% accuracy on UCF-101 without extra modalities.

Outperforms ImageNet supervised pretrained models.

Enhances downstream tasks like action recognition and video retrieval.

Abstract

We study self-supervised video representation learning, which is a challenging task due to 1) lack of labels for explicit supervision; 2) unstructured and noisy visual information. Existing methods mainly use contrastive loss with video clips as the instances and learn visual representation by discriminating instances from each other, but they need a careful treatment of negative pairs by either relying on large batch sizes, memory banks, extra modalities or customized mining strategies, which inevitably includes noisy data. In this paper, we observe that the consistency between positive samples is the key to learn robust video representation. Specifically, we propose two tasks to learn the appearance and speed consistency, respectively. The appearance consistency task aims to maximize the similarity between two clips of the same video with different playback speeds. The speed consistency task aims to maximize the similarity between two clips with the same playback speed but different appearance information. We show that optimizing the two tasks jointly consistently improves the performance on downstream tasks, e.g., action recognition and video retrieval. Remarkably, for action recognition on the UCF-101 dataset, we achieve 90.8\% accuracy without using any extra modalities or negative pairs for unsupervised pretraining, which outperforms the ImageNet supervised pretrained model. Codes and models will be available.