Temporal Contrastive Learning with Curriculum

TL;DR

ConCur introduces a curriculum-based contrastive learning approach for videos, progressively increasing difficulty in positive sample selection to improve action recognition and retrieval performance.

Contribution

It proposes a novel curriculum learning strategy in contrastive video representation learning, including an auxiliary task for temporal distance prediction.

Findings

Achieves state-of-the-art results on UCF101 and HMDB51 datasets.

Effective across different encoder backbones and pre-training datasets.

Component ablation confirms the importance of curriculum and auxiliary tasks.

Abstract

We present ConCur, a contrastive video representation learning method that uses curriculum learning to impose a dynamic sampling strategy in contrastive training. More specifically, ConCur starts the contrastive training with easy positive samples (temporally close and semantically similar clips), and as the training progresses, it increases the temporal span effectively sampling hard positives (temporally away and semantically dissimilar). To learn better context-aware representations, we also propose an auxiliary task of predicting the temporal distance between a positive pair of clips. We conduct extensive experiments on two popular action recognition datasets, UCF101 and HMDB51, on which our proposed method achieves state-of-the-art performance on two benchmark tasks of video action recognition and video retrieval. We explore the impact of encoder backbones and pre-training strategies by using R(2+1)D and C3D encoders and pre-training on Kinetics-400 and Kinetics-200 datasets. Moreover, a detailed ablation study shows the effectiveness of each of the components of our proposed method.