Self-supervised Video Representation Learning with Cascade Positive Retrieval

TL;DR

This paper introduces Cascade Positive Retrieval (CPR), a novel self-supervised learning method that progressively mines positive video examples across multiple views and stages, significantly improving video retrieval and action recognition performance.

Contribution

The paper proposes CPR, a new multi-stage positive example mining approach for self-supervised video representation learning, enhancing retrieval accuracy and downstream task performance.

Findings

CPR achieves 83.3% class mining recall, outperforming previous methods.

CPR improves state-of-the-art R@1 in video retrieval to 56.7%.

CPR enhances action recognition accuracy on UCF101 and HMDB51 datasets.

Abstract

Self-supervised video representation learning has been shown to effectively improve downstream tasks such as video retrieval and action recognition. In this paper, we present the Cascade Positive Retrieval (CPR) that successively mines positive examples w.r.t. the query for contrastive learning in a cascade of stages. Specifically, CPR exploits multiple views of a query example in different modalities, where an alternative view may help find another positive example dissimilar in the query view. We explore the effects of possible CPR configurations in ablations including the number of mining stages, the top similar example selection ratio in each stage, and progressive training with an incremental number of the final Top-k selection. The overall mining quality is measured to reflect the recall across training set classes. CPR reaches a median class mining recall of 83.3%, outperforming previous work by 5.5%. Implementation-wise, CPR is complementary to pretext tasks and can be easily applied to previous work. In the evaluation of pretraining on UCF101, CPR consistently improves existing work and even achieves state-of-the-art R@1 of 56.7% and 24.4% in video retrieval as well as 83.8% and 54.8% in action recognition on UCF101 and HMDB51. The code is available at https://github.com/necla-ml/CPR.