On the Surrogate Gap between Contrastive and Supervised Losses

TL;DR

This paper establishes new theoretical bounds linking contrastive and supervised losses, showing that increasing negative samples reduces the surrogate gap and improves downstream classification, supported by experiments across multiple domains.

Contribution

It provides the first surrogate bounds valid for all negative sample sizes, clarifying the theoretical relationship between contrastive and supervised learning.

Findings

Larger negative sample sizes decrease the surrogate gap.

Contrastive loss acts as a surrogate for downstream classification loss.

Theoretical bounds align with empirical results across datasets.

Abstract

Contrastive representation learning encourages data representation to make semantically similar pairs closer than randomly drawn negative samples, which has been successful in various domains such as vision, language, and graphs. Recent theoretical studies have attempted to explain the benefit of the large negative sample size by upper-bounding the downstream classification loss with the contrastive loss. However, the previous surrogate bounds have two drawbacks: they are only legitimate for a limited range of negative sample sizes and prohibitively large even within that range. Due to these drawbacks, there still does not exist a consensus on how negative sample size theoretically correlates with downstream classification performance. Following the simplified setting where positive pairs are drawn from the true distribution (not generated by data augmentation; as supposed in previous studies), this study establishes surrogate upper and lower bounds for the downstream classification loss for all negative sample sizes that best explain the empirical observations on the negative sample size in the earlier studies. Our bounds suggest that the contrastive loss can be viewed as a surrogate objective of the downstream loss and larger negative sample sizes improve downstream classification because the surrogate gap between contrastive and supervised losses decays. We verify that our theory is consistent with experiments on synthetic, vision, and language datasets.