A Unified Contrastive Loss for Self-Training

TL;DR

This paper introduces a unified contrastive loss framework for self-training in semi-supervised learning, replacing traditional cross-entropy losses to improve performance, convergence, and stability across multiple datasets.

Contribution

It proposes a novel contrastive loss-based framework that generalizes and enhances existing self-training methods using class prototypes, with theoretical equivalence to cross-entropy loss.

Findings

Significant performance improvements on three datasets with limited labeled data

Faster convergence and better transfer ability

Enhanced hyperparameter stability

Abstract

Self-training methods have proven to be effective in exploiting abundant unlabeled data in semi-supervised learning, particularly when labeled data is scarce. While many of these approaches rely on a cross-entropy loss function (CE), recent advances have shown that the supervised contrastive loss function (SupCon) can be more effective. Additionally, unsupervised contrastive learning approaches have also been shown to capture high quality data representations in the unsupervised setting. To benefit from these advantages in a semi-supervised setting, we propose a general framework to enhance self-training methods, which replaces all instances of CE losses with a unique contrastive loss. By using class prototypes, which are a set of class-wise trainable parameters, we recover the probability distributions of the CE setting and show a theoretical equivalence with it. Our framework, when applied to popular self-training methods, results in significant performance improvements across three different datasets with a limited number of labeled data. Additionally, we demonstrate further improvements in convergence speed, transfer ability, and hyperparameter stability. The code is available at \url{https://github.com/AurelienGauffre/semisupcon/}.