Semi-Supervised Action Recognition with Temporal Contrastive Learning

TL;DR

This paper introduces a semi-supervised action recognition method using temporal contrastive learning that leverages unlabeled videos at different speeds to improve recognition accuracy and generalization.

Contribution

It proposes a novel two-pathway temporal contrastive model that exploits video speed variations as supervisory signals, outperforming existing semi-supervised methods.

Findings

Outperforms state-of-the-art semi-supervised methods across multiple datasets.

Benefits from out-of-domain unlabeled videos, showing robustness.

Effective across various network architectures.

Abstract

Learning to recognize actions from only a handful of labeled videos is a challenging problem due to the scarcity of tediously collected activity labels. We approach this problem by learning a two-pathway temporal contrastive model using unlabeled videos at two different speeds leveraging the fact that changing video speed does not change an action. Specifically, we propose to maximize the similarity between encoded representations of the same video at two different speeds as well as minimize the similarity between different videos played at different speeds. This way we use the rich supervisory information in terms of `time' that is present in otherwise unsupervised pool of videos. With this simple yet effective strategy of manipulating video playback rates, we considerably outperform video extensions of sophisticated state-of-the-art semi-supervised image recognition methods across multiple diverse benchmark datasets and network architectures. Interestingly, our proposed approach benefits from out-of-domain unlabeled videos showing generalization and robustness. We also perform rigorous ablations and analysis to validate our approach. Project page: https://cvir.github.io/TCL/.