Temperature-Free Loss Function for Contrastive Learning

TL;DR

This paper introduces a temperature-free variant of the InfoNCE loss for contrastive learning, eliminating the need for hyperparameter tuning and improving gradient properties, with validated performance gains across multiple benchmarks.

Contribution

The paper proposes a novel temperature-free InfoNCE loss using inverse hyperbolic tangent, addressing hyperparameter sensitivity and gradient issues in contrastive learning.

Findings

Achieves comparable or better performance without temperature tuning

Provides better gradient properties for stable training

Validated on five benchmark datasets

Abstract

As one of the most promising methods in self-supervised learning, contrastive learning has achieved a series of breakthroughs across numerous fields. A predominant approach to implementing contrastive learning is applying InfoNCE loss: By capturing the similarities between pairs, InfoNCE loss enables learning the representation of data. Albeit its success, adopting InfoNCE loss requires tuning a temperature, which is a core hyperparameter for calibrating similarity scores. Despite its significance and sensitivity to performance being emphasized by several studies, searching for a valid temperature requires extensive trial-and-error-based experiments, which increases the difficulty of adopting InfoNCE loss. To address this difficulty, we propose a novel method to deploy InfoNCE loss without temperature. Specifically, we replace temperature scaling with the inverse hyperbolic tangent function, resulting in a modified InfoNCE loss. In addition to hyperparameter-free deployment, we observed that the proposed method even yielded a performance gain in contrastive learning. Our detailed theoretical analysis discovers that the current practice of temperature scaling in InfoNCE loss causes serious problems in gradient descent, whereas our method provides desirable gradient properties. The proposed method was validated on five benchmarks on contrastive learning, yielding satisfactory results without temperature tuning.