Representation Learning via Adversarially-Contrastive Optimal Transport

TL;DR

This paper introduces a novel contrastive representation learning method using optimal transport and adversarial distributions to learn low-dimensional, spatio-temporal representations for sequential data, demonstrated on human action recognition.

Contribution

It proposes a new objective combining optimal transport and adversarial distribution generation for contrastive learning, optimized on the Grassmann manifold.

Findings

Achieves competitive accuracy on human action recognition.

Introduces a Wasserstein GAN-based framework for negative sample generation.

Demonstrates effectiveness of the method on sequential data tasks.

Abstract

In this paper, we study the problem of learning compact (low-dimensional) representations for sequential data that captures its implicit spatio-temporal cues. To maximize extraction of such informative cues from the data, we set the problem within the context of contrastive representation learning and to that end propose a novel objective via optimal transport. Specifically, our formulation seeks a low-dimensional subspace representation of the data that jointly (i) maximizes the distance of the data (embedded in this subspace) from an adversarial data distribution under the optimal transport, a.k.a. the Wasserstein distance, (ii) captures the temporal order, and (iii) minimizes the data distortion. To generate the adversarial distribution, we propose a novel framework connecting Wasserstein GANs with a classifier, allowing a principled mechanism for producing good negative distributions for contrastive learning, which is currently a challenging problem. Our full objective is cast as a subspace learning problem on the Grassmann manifold and solved via Riemannian optimization. To empirically study our formulation, we provide experiments on the task of human action recognition in video sequences. Our results demonstrate competitive performance against challenging baselines.