Tuned Contrastive Learning

TL;DR

This paper introduces Tuned Contrastive Learning (TCL), a novel loss function that enhances contrastive learning by tuning gradient responses, improving performance in both supervised and self-supervised visual representation tasks.

Contribution

The paper proposes TCL, a new contrastive loss with tunable parameters, providing theoretical gradient analysis and demonstrating superior or comparable results to state-of-the-art methods.

Findings

TCL outperforms SupCon and cross-entropy losses in supervised classification.

TCL maintains stability across various hyper-parameter settings.

TCL achieves state-of-the-art performance in both supervised and self-supervised learning.

Abstract

In recent times, contrastive learning based loss functions have become increasingly popular for visual self-supervised representation learning owing to their state-of-the-art (SOTA) performance. Most of the modern contrastive learning methods generalize only to one positive and multiple negatives per anchor. A recent state-of-the-art, supervised contrastive (SupCon) loss, extends self-supervised contrastive learning to supervised setting by generalizing to multiple positives and negatives in a batch and improves upon the cross-entropy loss. In this paper, we propose a novel contrastive loss function -- Tuned Contrastive Learning (TCL) loss, that generalizes to multiple positives and negatives in a batch and offers parameters to tune and improve the gradient responses from hard positives and hard negatives. We provide theoretical analysis of our loss function's gradient response and show mathematically how it is better than that of SupCon loss. We empirically compare our loss function with SupCon loss and cross-entropy loss in supervised setting on multiple classification-task datasets to show its effectiveness. We also show the stability of our loss function to a range of hyper-parameter settings. Unlike SupCon loss which is only applied to supervised setting, we show how to extend TCL to self-supervised setting and empirically compare it with various SOTA self-supervised learning methods. Hence, we show that TCL loss achieves performance on par with SOTA methods in both supervised and self-supervised settings.