Implicit variance regularization in non-contrastive SSL

TL;DR

This paper provides a theoretical analysis of how predictor networks in non-contrastive self-supervised learning methods like BYOL and SimSiam prevent collapse through implicit variance regularization, and introduces IsoLoss to improve learning dynamics.

Contribution

It offers a theoretical understanding of variance regularization in non-contrastive SSL and proposes IsoLoss to enhance convergence and robustness.

Findings

Both Euclidean and cosine similarity avoid collapse via implicit variance regularization.

Eigenvalues act as learning rate multipliers, influencing convergence.

IsoLoss accelerates initial learning and improves robustness.

Abstract

Non-contrastive SSL methods like BYOL and SimSiam rely on asymmetric predictor networks to avoid representational collapse without negative samples. Yet, how predictor networks facilitate stable learning is not fully understood. While previous theoretical analyses assumed Euclidean losses, most practical implementations rely on cosine similarity. To gain further theoretical insight into non-contrastive SSL, we analytically study learning dynamics in conjunction with Euclidean and cosine similarity in the eigenspace of closed-form linear predictor networks. We show that both avoid collapse through implicit variance regularization albeit through different dynamical mechanisms. Moreover, we find that the eigenvalues act as effective learning rate multipliers and propose a family of isotropic loss functions (IsoLoss) that equalize convergence rates across eigenmodes. Empirically, IsoLoss speeds up the initial learning dynamics and increases robustness, thereby allowing us to dispense with the EMA target network typically used with non-contrastive methods. Our analysis sheds light on the variance regularization mechanisms of non-contrastive SSL and lays the theoretical grounds for crafting novel loss functions that shape the learning dynamics of the predictor's spectrum.