Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods

TL;DR

This paper unifies contrastive and non-contrastive self-supervised learning methods within a spectral manifold learning framework, providing theoretical insights, closed-form solutions, and guidelines for practitioners based on pairwise relations and downstream tasks.

Contribution

It establishes a spectral unification of SSL methods, deriving closed-form solutions and analyzing the influence of pairwise relations on performance and optimal parameters.

Findings

VICReg, SimCLR, BarlowTwins correspond to spectral methods like Laplacian Eigenmaps and MDS.

Theoretical formulas for optimal representations and network parameters are derived.

Guidelines for choosing SSL methods based on pairwise relation alignment with downstream tasks.

Abstract

Self-Supervised Learning (SSL) surmises that inputs and pairwise positive relationships are enough to learn meaningful representations. Although SSL has recently reached a milestone: outperforming supervised methods in many modalities\dots the theoretical foundations are limited, method-specific, and fail to provide principled design guidelines to practitioners. In this paper, we propose a unifying framework under the helm of spectral manifold learning to address those limitations. Through the course of this study, we will rigorously demonstrate that VICReg, SimCLR, BarlowTwins et al. correspond to eponymous spectral methods such as Laplacian Eigenmaps, Multidimensional Scaling et al. This unification will then allow us to obtain (i) the closed-form optimal representation for each method, (ii) the closed-form optimal network parameters in the linear regime for each method, (iii) the impact of the pairwise relations used during training on each of those quantities and on downstream task performances, and most importantly, (iv) the first theoretical bridge between contrastive and non-contrastive methods towards global and local spectral embedding methods respectively, hinting at the benefits and limitations of each. For example, (i) if the pairwise relation is aligned with the downstream task, any SSL method can be employed successfully and will recover the supervised method, but in the low data regime, VICReg's invariance hyper-parameter should be high; (ii) if the pairwise relation is misaligned with the downstream task, VICReg with small invariance hyper-parameter should be preferred over SimCLR or BarlowTwins.