Hard Negative Sampling via Regularized Optimal Transport for Contrastive Representation Learning

TL;DR

This paper introduces a novel regularized optimal transport framework for hard negative sampling in contrastive learning, improving the quality of negative samples and the resulting representations for downstream tasks.

Contribution

It develops a min-max contrastive learning framework with OT-based regularization, providing theoretical insights and practical methods for better negative sampling.

Findings

Negative samples are more similar to anchors than baseline samples.

Entropic regularization yields negative distributions similar to recent methods.

Using OT-based ground costs improves downstream task performance.

Abstract

We study the problem of designing hard negative sampling distributions for unsupervised contrastive representation learning. We propose and analyze a novel min-max framework that seeks a representation which minimizes the maximum (worst-case) generalized contrastive learning loss over all couplings (joint distributions between positive and negative samples subject to marginal constraints) and prove that the resulting min-max optimum representation will be degenerate. This provides the first theoretical justification for incorporating additional regularization constraints on the couplings. We re-interpret the min-max problem through the lens of Optimal Transport (OT) theory and utilize regularized transport couplings to control the degree of hardness of negative examples. Through experiments we demonstrate that the negative samples generated from our designed negative distribution are more similar to the anchor than those generated from the baseline negative distribution. We also demonstrate that entropic regularization yields negative sampling distributions with parametric form similar to that in a recent state-of-the-art negative sampling design and has similar performance in multiple datasets. Utilizing the uncovered connection with OT, we propose a new ground cost for designing the negative distribution and show improved performance of the learned representation on downstream tasks compared to the representation learned when using squared Euclidean cost.