Contrasting the landscape of contrastive and non-contrastive learning

TL;DR

This paper compares contrastive and non-contrastive learning, revealing that non-contrastive methods often get stuck in poor minima and do not inherently avoid collapsed solutions, challenging previous assumptions.

Contribution

The study provides theoretical and experimental evidence that non-contrastive learning methods frequently encounter bad minima and do not naturally circumvent collapsed solutions.

Findings

Non-contrastive losses have many bad minima even in simple models.

Training does not inherently avoid collapsed solutions.

Contrasts with the folk belief that avoiding collapse ensures good representations.

Abstract

A lot of recent advances in unsupervised feature learning are based on designing features which are invariant under semantic data augmentations. A common way to do this is contrastive learning, which uses positive and negative samples. Some recent works however have shown promising results for non-contrastive learning, which does not require negative samples. However, the non-contrastive losses have obvious "collapsed" minima, in which the encoders output a constant feature embedding, independent of the input. A folk conjecture is that so long as these collapsed solutions are avoided, the produced feature representations should be good. In our paper, we cast doubt on this story: we show through theoretical results and controlled experiments that even on simple data models, non-contrastive losses have a preponderance of non-collapsed bad minima. Moreover, we show that the training process does not avoid these minima.