Understanding the Behaviour of Contrastive Loss

TL;DR

This paper analyzes the behavior of unsupervised contrastive loss, revealing its role as a hardness-aware function controlled by temperature, and discusses the trade-off between uniformity and tolerance affecting feature learning and downstream tasks.

Contribution

It provides a theoretical understanding of contrastive loss, highlighting the uniformity-tolerance dilemma and the impact of temperature on feature quality and downstream performance.

Findings

Contrastive loss is hardness-aware and temperature controls penalty strength.

Uniformity promotes separable features but can harm semantic structure.

Optimal temperature balances uniformity and tolerance, improving downstream results.

Abstract

Unsupervised contrastive learning has achieved outstanding success, while the mechanism of contrastive loss has been less studied. In this paper, we concentrate on the understanding of the behaviours of unsupervised contrastive loss. We will show that the contrastive loss is a hardness-aware loss function, and the temperature {\tau} controls the strength of penalties on hard negative samples. The previous study has shown that uniformity is a key property of contrastive learning. We build relations between the uniformity and the temperature {\tau} . We will show that uniformity helps the contrastive learning to learn separable features, however excessive pursuit to the uniformity makes the contrastive loss not tolerant to semantically similar samples, which may break the underlying semantic structure and be harmful to the formation of features useful for downstream tasks. This is caused by the inherent defect of the instance discrimination objective. Specifically, instance discrimination objective tries to push all different instances apart, ignoring the underlying relations between samples. Pushing semantically consistent samples apart has no positive effect for acquiring a prior informative to general downstream tasks. A well-designed contrastive loss should have some extents of tolerance to the closeness of semantically similar samples. Therefore, we find that the contrastive loss meets a uniformity-tolerance dilemma, and a good choice of temperature can compromise these two properties properly to both learn separable features and tolerant to semantically similar samples, improving the feature qualities and the downstream performances.