TCLR: Temporal Contrastive Learning for Video Representation

TL;DR

This paper introduces a novel temporal contrastive learning framework for videos, employing two new loss functions to enhance temporal feature diversity and improve performance on various video understanding tasks.

Contribution

It proposes a new temporal contrastive learning framework with two innovative loss functions to explicitly encourage temporal feature distinction in self-supervised video representation learning.

Findings

Achieves 82.4% top-1 accuracy on UCF101 with 3D ResNet-18

Improves nearest neighbor video retrieval accuracy by 11.7% on UCF101

Significantly outperforms previous methods on multiple video understanding benchmarks

Abstract

Contrastive learning has nearly closed the gap between supervised and self-supervised learning of image representations, and has also been explored for videos. However, prior work on contrastive learning for video data has not explored the effect of explicitly encouraging the features to be distinct across the temporal dimension. We develop a new temporal contrastive learning framework consisting of two novel losses to improve upon existing contrastive self-supervised video representation learning methods. The local-local temporal contrastive loss adds the task of discriminating between non-overlapping clips from the same video, whereas the global-local temporal contrastive aims to discriminate between timesteps of the feature map of an input clip in order to increase the temporal diversity of the learned features. Our proposed temporal contrastive learning framework achieves significant improvement over the state-of-the-art results in various downstream video understanding tasks such as action recognition, limited-label action classification, and nearest-neighbor video retrieval on multiple video datasets and backbones. We also demonstrate significant improvement in fine-grained action classification for visually similar classes. With the commonly used 3D ResNet-18 architecture with UCF101 pretraining, we achieve 82.4\% (+5.1\% increase over the previous best) top-1 accuracy on UCF101 and 52.9\% (+5.4\% increase) on HMDB51 action classification, and 56.2\% (+11.7\% increase) Top-1 Recall on UCF101 nearest neighbor video retrieval. Code released at github.com/DAVEISHAN/TCLR.