Contrastive Video Representation Learning via Adversarial Perturbations

TL;DR

This paper introduces a novel contrastive learning method for video representations that uses adversarial perturbations to generate negative samples, resulting in more robust and discriminative video features.

Contribution

It proposes a new contrastive learning framework utilizing adversarial perturbations to create negative samples and a discriminative subspace pooling method for improved video representation.

Findings

Achieves state-of-the-art results on multiple video datasets.

Demonstrates robustness of learned representations against adversarial noise.

Introduces a subspace learning approach on the Stiefel manifold for video descriptors.

Abstract

Adversarial perturbations are noise-like patterns that can subtly change the data, while failing an otherwise accurate classifier. In this paper, we propose to use such perturbations within a novel contrastive learning setup to build negative samples, which are then used to produce improved video representations. To this end, given a well-trained deep model for per-frame video recognition, we first generate adversarial noise adapted to this model. Positive and negative bags are produced using the original data features from the full video sequence and their perturbed counterparts, respectively. Unlike the classic contrastive learning methods, we develop a binary classification problem that learns a set of discriminative hyperplanes -- as a subspace -- that will separate the two bags from each other. This subspace is then used as a descriptor for the video, dubbed \emph{discriminative subspace pooling}. As the perturbed features belong to data classes that are likely to be confused with the original features, the discriminative subspace will characterize parts of the feature space that are more representative of the original data, and thus may provide robust video representations. To learn such descriptors, we formulate a subspace learning objective on the Stiefel manifold and resort to Riemannian optimization methods for solving it efficiently. We provide experiments on several video datasets and demonstrate state-of-the-art results.