Hyperspherically Regularized Networks for Self-Supervision

TL;DR

This paper improves self-supervised learning by introducing hyperspherical regularization to enhance the uniformity and diversity of image representations in BYOL, leading to better downstream task performance.

Contribution

It demonstrates that hyperspherical energy regularization enhances representation uniformity and diversity in BYOL, improving its effectiveness without contrastive learning.

Findings

Regularization improves representation uniformity.

Enhanced uniformity leads to better downstream performance.

Hyperspherical energy minimization benefits self-supervised learning.

Abstract

Bootstrap Your Own Latent (BYOL) introduced an approach to self-supervised learning avoiding the contrastive paradigm and subsequently removing the computational burden of negative sampling associated with such methods. However, we empirically find that the image representations produced under the BYOL's self-distillation paradigm are poorly distributed in representation space compared to contrastive methods. This work empirically demonstrates that feature diversity enforced by contrastive losses is beneficial to image representation uniformity when employed in BYOL, and as such, provides greater inter-class representation separability. Additionally, we explore and advocate the use of regularization methods, specifically the layer-wise minimization of hyperspherical energy (i.e. maximization of entropy) of network weights to encourage representation uniformity. We show that directly optimizing a measure of uniformity alongside the standard loss, or regularizing the networks of the BYOL architecture to minimize the hyperspherical energy of neurons can produce more uniformly distributed and therefore better performing representations for downstream tasks.