Contrastive Learning Inverts the Data Generating Process

TL;DR

This paper proves that contrastive learning with InfoNCE objectives implicitly inverts the data's generative process, explaining its success in learning generalizable representations across tasks.

Contribution

It provides a theoretical proof linking contrastive learning to inverting generative models, enhancing understanding of its effectiveness and guiding future loss design.

Findings

Contrastive models implicitly invert the data generative process.

Empirical results hold even when assumptions are violated.

The theory connects contrastive learning with generative modeling and ICA.

Abstract

Contrastive learning has recently seen tremendous success in self-supervised learning. So far, however, it is largely unclear why the learned representations generalize so effectively to a large variety of downstream tasks. We here prove that feedforward models trained with objectives belonging to the commonly used InfoNCE family learn to implicitly invert the underlying generative model of the observed data. While the proofs make certain statistical assumptions about the generative model, we observe empirically that our findings hold even if these assumptions are severely violated. Our theory highlights a fundamental connection between contrastive learning, generative modeling, and nonlinear independent component analysis, thereby furthering our understanding of the learned representations as well as providing a theoretical foundation to derive more effective contrastive losses.