Collapse-Proof Non-Contrastive Self-Supervised Learning

TL;DR

This paper introduces a theoretically grounded, simplified approach to non-contrastive self-supervised learning that prevents collapse and improves generalization by leveraging hyperdimensional computing principles.

Contribution

It proposes a new projector and loss function design based on hyperdimensional computing that inherently avoids collapse modes without explicit regularization.

Findings

The method prevents representation collapse across multiple datasets.

It achieves strong generalization in clustering and classification tasks.

The approach outperforms existing methods in avoiding training failures.

Abstract

We present a principled and simplified design of the projector and loss function for non-contrastive self-supervised learning based on hyperdimensional computing. We theoretically demonstrate that this design introduces an inductive bias that encourages representations to be simultaneously decorrelated and clustered, without explicitly enforcing these properties. This bias provably enhances generalization and suffices to avoid known training failure modes, such as representation, dimensional, cluster, and intracluster collapses. We validate our theoretical findings on image datasets, including SVHN, CIFAR-10, CIFAR-100, and ImageNet-100. Our approach effectively combines the strengths of feature decorrelation and cluster-based self-supervised learning methods, overcoming training failure modes while achieving strong generalization in clustering and linear classification tasks.