Can Temporal Information Help with Contrastive Self-Supervised Learning?

TL;DR

This paper introduces TaCo, a novel framework that leverages carefully selected temporal transformations as both data augmentation and self-supervision signals to improve contrastive self-supervised learning for video understanding.

Contribution

The paper proposes a new paradigm, TaCo, that effectively integrates temporal information into video CSL by using temporal transformations for data augmentation and self-supervision, leading to significant performance gains.

Findings

TaCo outperforms previous state-of-the-art methods on UCF-101 and HMDB-51 datasets.

Temporal transformations as self-supervision improve video representation learning.

Direct application of temporal augmentations does not help, motivating the new approach.

Abstract

Leveraging temporal information has been regarded as essential for developing video understanding models. However, how to properly incorporate temporal information into the recent successful instance discrimination based contrastive self-supervised learning (CSL) framework remains unclear. As an intuitive solution, we find that directly applying temporal augmentations does not help, or even impair video CSL in general. This counter-intuitive observation motivates us to re-design existing video CSL frameworks, for better integration of temporal knowledge. To this end, we present Temporal-aware Contrastive self-supervised learningTaCo, as a general paradigm to enhance video CSL. Specifically, TaCo selects a set of temporal transformations not only as strong data augmentation but also to constitute extra self-supervision for video understanding. By jointly contrasting instances with enriched temporal transformations and learning these transformations as self-supervised signals, TaCo can significantly enhance unsupervised video representation learning. For instance, TaCo demonstrates consistent improvement in downstream classification tasks over a list of backbones and CSL approaches. Our best model achieves 85.1% (UCF-101) and 51.6% (HMDB-51) top-1 accuracy, which is a 3% and 2.4% relative improvement over the previous state-of-the-art.