Nearest-Neighbor Inter-Intra Contrastive Learning from Unlabeled Videos

TL;DR

This paper introduces Inter-Intra Video Contrastive Learning (IIVCL), a method that leverages nearest-neighbor videos as positives to enhance diversity and semantic scope in self-supervised video representation learning.

Contribution

The paper proposes a novel contrastive learning approach that incorporates nearest-neighbor videos as positives, extending beyond local clips and class boundaries for improved video representations.

Findings

Improved performance on various video tasks.

Enhanced positive key diversity.

More relaxed similarity notion extending beyond class boundaries.

Abstract

Contrastive learning has recently narrowed the gap between self-supervised and supervised methods in image and video domain. State-of-the-art video contrastive learning methods such as CVRL and $\rho$-MoCo spatiotemporally augment two clips from the same video as positives. By only sampling positive clips locally from a single video, these methods neglect other semantically related videos that can also be useful. To address this limitation, we leverage nearest-neighbor videos from the global space as additional positive pairs, thus improving positive key diversity and introducing a more relaxed notion of similarity that extends beyond video and even class boundaries. Our method, Inter-Intra Video Contrastive Learning (IIVCL), improves performance on a range of video tasks.