Probabilistic Representations for Video Contrastive Learning

TL;DR

This paper introduces a probabilistic approach to video contrastive learning, representing clips as distributions and combining them into a mixture model to improve self-supervised video representations for tasks like action recognition.

Contribution

It proposes a novel probabilistic embedding method that models video clips as distributions and uses a stochastic contrastive loss, advancing self-supervised video representation learning.

Findings

Achieves state-of-the-art results on UCF101 and HMDB51 benchmarks.

Effectively models video uncertainty with probabilistic embeddings.

Reduces reliance on data augmentation strategies.

Abstract

This paper presents Probabilistic Video Contrastive Learning, a self-supervised representation learning method that bridges contrastive learning with probabilistic representation. We hypothesize that the clips composing the video have different distributions in short-term duration, but can represent the complicated and sophisticated video distribution through combination in a common embedding space. Thus, the proposed method represents video clips as normal distributions and combines them into a Mixture of Gaussians to model the whole video distribution. By sampling embeddings from the whole video distribution, we can circumvent the careful sampling strategy or transformations to generate augmented views of the clips, unlike previous deterministic methods that have mainly focused on such sample generation strategies for contrastive learning. We further propose a stochastic contrastive loss to learn proper video distributions and handle the inherent uncertainty from the nature of the raw video. Experimental results verify that our probabilistic embedding stands as a state-of-the-art video representation learning for action recognition and video retrieval on the most popular benchmarks, including UCF101 and HMDB51.