Hypersolid: Emergent Vision Representations via Short-Range Repulsion

TL;DR

Hypersolid introduces a novel self-supervised learning approach that uses short-range repulsion to prevent representation collapse, maintaining diversity and improving performance on fine-grained and low-resolution classification tasks.

Contribution

It reinterprets representation learning as a packing problem and operationalizes this with short-range repulsion to enhance diversity and prevent collapse.

Findings

Outperforms existing methods on fine-grained classification tasks

Preserves augmentation diversity effectively

Prevents local collisions in representation space

Abstract

A recurring challenge in self-supervised learning is preventing representation collapse. Existing solutions typically rely on global regularization, such as maximizing distances, decorrelating dimensions or enforcing certain distributions. We instead reinterpret representation learning as a discrete packing problem, where preserving information simplifies to maintaining injectivity. We operationalize this in Hypersolid, a method using short-range hard-ball repulsion to prevent local collisions. This constraint results in a high-separation geometric regime that preserves augmentation diversity, excelling on fine-grained and low-resolution classification tasks.