Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasets

TL;DR

This paper characterizes the geometry of optimal representations in contrastive learning with imbalanced datasets, revealing phenomena like Neural Collapse and Minority Collapse depending on class proportions.

Contribution

It provides a theoretical framework for understanding optimal contrastive representations under class imbalance, including new phenomena and geometric characterizations.

Findings

Optimal representations exhibit class mean collapse with angular symmetry.

Class imbalance can cause Minority Collapse, where minority class samples collapse into a single vector.

The geometry of optimal representations can be determined by solving a convex optimization problem.

Abstract

In this paper, we provide a computable characterization of the geometry of optimal representations in Contrastive Learning (CL) when the classes are imbalanced. When classes are balanced and the representation dimension is greater than the number of classes, it is well-known that the optimal representations exhibit Neural Collapse (NC), i.e., representations from the same class collapse to their class means and the class means form an Equiangular Tight Frame (ETF). For imbalanced classes and a large, generalized family of CL losses, we prove that the optimal representations of all samples from the same class collapse to their class means and their geometry exhibits an angular symmetry structure that is determined by the relative class proportions. In general, we show that the geometry can be determined by solving a convex optimization problem. Exploiting this symmetry structure, we analytically investigate a special case where class imbalance is extreme and prove that CL exhibits a phenomenon called Minority Collapse (MC) where all samples from the minority classes (classes with small probabilities) collapse into a single vector, whenever the class imbalance exceeds a threshold, which in turn depends on the regularity properties of the CL loss used and on the number of negative samples. Numerical results are provided to illustrate these phenomena and corroborate the theoretical results. We conclude by identifying a number of open problems.