Modelling Neighbor Relation in Joint Space-Time Graph for Video Correspondence Learning

TL;DR

This paper introduces a self-supervised approach for learning visual correspondence in videos by modeling neighbor relations in a joint space-time graph, improving performance on various video understanding tasks without requiring labeled data.

Contribution

The method uniquely models neighbor relations in a joint space-time graph and leverages cycle-consistency for self-supervised learning, outperforming existing methods on multiple video tasks.

Findings

Outperforms state-of-the-art self-supervised methods on video tasks

Surpasses some fully supervised algorithms in accuracy

Effective in tasks like object propagation and pose tracking

Abstract

This paper presents a self-supervised method for learning reliable visual correspondence from unlabeled videos. We formulate the correspondence as finding paths in a joint space-time graph, where nodes are grid patches sampled from frames, and are linked by two types of edges: (i) neighbor relations that determine the aggregation strength from intra-frame neighbors in space, and (ii) similarity relations that indicate the transition probability of inter-frame paths across time. Leveraging the cycle-consistency in videos, our contrastive learning objective discriminates dynamic objects from both their neighboring views and temporal views. Compared with prior works, our approach actively explores the neighbor relations of central instances to learn a latent association between center-neighbor pairs (e.g., "hand -- arm") across time, thus improving the instance discrimination. Without fine-tuning, our learned representation outperforms the state-of-the-art self-supervised methods on a variety of visual tasks including video object propagation, part propagation, and pose keypoint tracking. Our self-supervised method also surpasses some fully supervised algorithms designed for the specific tasks.