Spatial-then-Temporal Self-Supervised Learning for Video Correspondence

TL;DR

This paper introduces a novel self-supervised learning approach that combines spatial and temporal cues to improve video correspondence representations, outperforming existing methods.

Contribution

It proposes a spatial-then-temporal self-supervised learning framework with correlation distillation losses, enhancing video correspondence tasks.

Findings

Outperforms state-of-the-art self-supervised methods

Effective in various correspondence-based video analysis tasks

Ablation studies confirm the benefits of the two-step design

Abstract

In low-level video analyses, effective representations are important to derive the correspondences between video frames. These representations have been learned in a self-supervised fashion from unlabeled images or videos, using carefully designed pretext tasks in some recent studies. However, the previous work concentrates on either spatial-discriminative features or temporal-repetitive features, with little attention to the synergy between spatial and temporal cues. To address this issue, we propose a spatial-then-temporal self-supervised learning method. Specifically, we firstly extract spatial features from unlabeled images via contrastive learning, and secondly enhance the features by exploiting the temporal cues in unlabeled videos via reconstructive learning. In the second step, we design a global correlation distillation loss to ensure the learning not to forget the spatial cues, and a local correlation distillation loss to combat the temporal discontinuity that harms the reconstruction. The proposed method outperforms the state-of-the-art self-supervised methods, as established by the experimental results on a series of correspondence-based video analysis tasks. Also, we performed ablation studies to verify the effectiveness of the two-step design as well as the distillation losses.