InfoNCE Induces Gaussian Distribution

TL;DR

This paper demonstrates that the InfoNCE contrastive loss induces Gaussian distributions in learned representations, providing a theoretical explanation for the Gaussianity observed in contrastive learning models.

Contribution

It establishes that contrastive training with InfoNCE leads to Gaussian representations under certain conditions, supported by theoretical analysis and experiments.

Findings

Representations tend to Gaussian distribution under contrastive learning.

Adding regularization promotes Gaussianity in representations.

Experimental results confirm Gaussian behavior across datasets and architectures.

Abstract

Contrastive learning has become a cornerstone of modern representation learning, allowing training with massive unlabeled data for both task-specific and general (foundation) models. A prototypical loss in contrastive training is InfoNCE and its variants. In this work, we show that the InfoNCE objective induces Gaussian structure in representations that emerge from contrastive training. We establish this result in two complementary regimes. First, we show that under certain alignment and concentration assumptions, projections of the high-dimensional representation asymptotically approach a multivariate Gaussian distribution. Next, under less strict assumptions, we show that adding a small asymptotically vanishing regularization term that promotes low feature norm and high feature entropy leads to similar asymptotic results. We support our analysis with experiments on synthetic and CIFAR-10 datasets across multiple encoder architectures and sizes, demonstrating consistent Gaussian behavior. This perspective provides a principled explanation for commonly observed Gaussianity in contrastive representations. The resulting Gaussian model enables principled analytical treatment of learned representations and is expected to support a wide range of applications in contrastive learning.