Bridging Mini-Batch and Asymptotic Analysis in Contrastive Learning: From InfoNCE to Kernel-Based Losses

TL;DR

This paper unifies various contrastive learning methods through theoretical analysis, introduces a new loss function DHEL, and demonstrates improved performance and robustness in vision tasks.

Contribution

It provides a theoretical connection between contrastive learning losses and hyperspherical energy minimization, and proposes DHEL, a novel loss with better practical properties.

Findings

DHEL simplifies contrastive learning while maintaining theoretical guarantees.

Kernel contrastive learning (KCL) achieves batch-size independent minimizers.

Empirical results show improved downstream performance and robustness.

Abstract

What do different contrastive learning (CL) losses actually optimize for? Although multiple CL methods have demonstrated remarkable representation learning capabilities, the differences in their inner workings remain largely opaque. In this work, we analyse several CL families and prove that, under certain conditions, they admit the same minimisers when optimizing either their batch-level objectives or their expectations asymptotically. In both cases, an intimate connection with the hyperspherical energy minimisation (HEM) problem resurfaces. Drawing inspiration from this, we introduce a novel CL objective, coined Decoupled Hyperspherical Energy Loss (DHEL). DHEL simplifies the problem by decoupling the target hyperspherical energy from the alignment of positive examples while preserving the same theoretical guarantees. Going one step further, we show the same results hold for another relevant CL family, namely kernel contrastive learning (KCL), with the additional advantage of the expected loss being independent of batch size, thus identifying the minimisers in the non-asymptotic regime. Empirical results demonstrate improved downstream performance and robustness across combinations of different batch sizes and hyperparameters and reduced dimensionality collapse, on several computer vision datasets.