Semi-Supervised Contrastive Learning with Orthonormal Prototypes

TL;DR

This paper introduces CLOP, a semi-supervised contrastive learning method that prevents dimensional collapse by encouraging orthogonal class embeddings, leading to improved stability and performance in image tasks.

Contribution

The paper identifies a critical learning-rate threshold causing collapse and proposes CLOP, a novel loss function promoting orthogonal embeddings to enhance semi-supervised contrastive learning.

Findings

CLOP outperforms standard contrastive methods in classification accuracy.

CLOP demonstrates greater stability across various learning rates and batch sizes.

Experiments confirm CLOP effectively prevents dimensional collapse.

Abstract

Contrastive learning has emerged as a powerful method in deep learning, excelling at learning effective representations through contrasting samples from different distributions. However, dimensional collapse, where embeddings converge into a lower-dimensional space, poses a significant challenge, especially in semi-supervised and self-supervised setups. In this paper, we first identify a critical learning-rate threshold, beyond which standard contrastive losses converge to collapsed solutions. Building on these insights, we propose CLOP, a novel semi-supervised loss function designed to prevent dimensional collapse by promoting the formation of orthogonal linear subspaces among class embeddings. Through extensive experiments on real and synthetic datasets, we demonstrate that CLOP improves performance in image classification and object detection tasks while also exhibiting greater stability across different learning rates and batch sizes.