Generalizing Supervised Contrastive learning: A Projection Perspective

TL;DR

This paper introduces ProjNCE, a generalized contrastive loss that unifies supervised and self-supervised learning, providing a theoretical MI bound and improving performance on image and audio tasks.

Contribution

It proposes ProjNCE, a novel MI-bounded contrastive loss with flexible projection strategies, and demonstrates its effectiveness over existing methods.

Findings

ProjNCE outperforms SupCon and cross-entropy in experiments.

Theoretical proof that ProjNCE bounds mutual information.

Flexible projection strategies improve class embedding quality.

Abstract

Self-supervised contrastive learning (SSCL) has emerged as a powerful paradigm for representation learning and has been studied from multiple perspectives, including mutual information and geometric viewpoints. However, supervised contrastive (SupCon) approaches have received comparatively little attention in this context: for instance, while InfoNCE used in SSCL is known to form a lower bound on mutual information (MI), the relationship between SupCon and MI remains unexplored. To address this gap, we introduce ProjNCE, a generalization of the InfoNCE loss that unifies supervised and self-supervised contrastive objectives by incorporating projection functions and an adjustment term for negative pairs. We prove that ProjNCE constitutes a valid MI bound and affords greater flexibility in selecting projection strategies for class embeddings. Building on this flexibility, we further explore the centroid-based class embeddings in SupCon by exploring a variety of projection methods. Extensive experiments on image and audio datasets demonstrate that ProjNCE consistently outperforms both SupCon and standard cross-entropy training. Our work thus refines SupCon along two complementary perspectives--information-theoretic and projection viewpoints--and offers broadly applicable improvements whenever SupCon serves as the foundational contrastive objective.