Probabilistic Contrastive Loss for Self-Supervised Learning

TL;DR

This paper introduces a probabilistic contrastive loss for self-supervised learning that incorporates uncertainty quantification, providing a more grounded and potentially more human-like approach to contrastive learning.

Contribution

It reinterprets the temperature hyperparameter as a confidence measure, deriving a new loss function that models uncertainty in contrastive learning.

Findings

Empirically demonstrates properties aligning with human predictions

Provides a mathematically grounded uncertainty measure

Enhances contrastive learning with probabilistic insights

Abstract

This paper proposes a probabilistic contrastive loss function for self-supervised learning. The well-known contrastive loss is deterministic and involves a temperature hyperparameter that scales the inner product between two normed feature embeddings. By reinterpreting the temperature hyperparameter as a quantity related to the radius of the hypersphere, we derive a new loss function that involves a confidence measure which quantifies uncertainty in a mathematically grounding manner. Some intriguing properties of the proposed loss function are empirically demonstrated, which agree with human-like predictions. We believe the present work brings up a new prospective to the area of contrastive learning.