An Inclusive Theoretical Framework of Robust Supervised Contrastive Loss against Label Noise

TL;DR

This paper develops a unified theoretical framework for robust supervised contrastive loss functions, identifies non-robustness in popular losses like InfoNCE, and proposes a new robust version called SymNCE, validated on benchmark datasets.

Contribution

It introduces the first systematic theoretical foundation for robust contrastive losses, deriving a robustness criterion and designing a new robust loss function SymNCE.

Findings

SymNCE outperforms existing methods under label noise.

Theoretical analysis reveals InfoNCE is non-robust.

Framework explains prior robust contrastive techniques.

Abstract

Learning from noisy labels is a critical challenge in machine learning, with vast implications for numerous real-world scenarios. While supervised contrastive learning has recently emerged as a powerful tool for navigating label noise, many existing solutions remain heuristic, often devoid of a systematic theoretical foundation for crafting robust supervised contrastive losses. To address the gap, in this paper, we propose a unified theoretical framework for robust losses under the pairwise contrastive paradigm. In particular, we for the first time derive a general robust condition for arbitrary contrastive losses, which serves as a criterion to verify the theoretical robustness of a supervised contrastive loss against label noise. The theory indicates that the popular InfoNCE loss is in fact non-robust, and accordingly inspires us to develop a robust version of InfoNCE, termed Symmetric InfoNCE (SymNCE). Moreover, we highlight that our theory is an inclusive framework that provides explanations to prior robust techniques such as nearest-neighbor (NN) sample selection and robust contrastive loss. Validation experiments on benchmark datasets demonstrate the superiority of SymNCE against label noise.