Infinite Width Limits of Self Supervised Neural Networks

TL;DR

This paper proves that the NTK of two-layer neural networks trained with Barlow Twins loss converges to a constant as width increases, providing a theoretical foundation for using kernel methods to analyze self-supervised learning.

Contribution

It establishes the first rigorous connection between NTK and self-supervised learning with Barlow Twins, showing NTK convergence in the infinite width limit.

Findings

NTK of Barlow Twins becomes constant at infinite width

Provides generalization bounds for kernelized Barlow Twins

Connects kernel theory with finite-width neural networks

Abstract

The NTK is a widely used tool in the theoretical analysis of deep learning, allowing us to look at supervised deep neural networks through the lenses of kernel regression. Recently, several works have investigated kernel models for self-supervised learning, hypothesizing that these also shed light on the behavior of wide neural networks by virtue of the NTK. However, it remains an open question to what extent this connection is mathematically sound -- it is a commonly encountered misbelief that the kernel behavior of wide neural networks emerges irrespective of the loss function it is trained on. In this paper, we bridge the gap between the NTK and self-supervised learning, focusing on two-layer neural networks trained under the Barlow Twins loss. We prove that the NTK of Barlow Twins indeed becomes constant as the width of the network approaches infinity. Our analysis technique is a bit different from previous works on the NTK and may be of independent interest. Overall, our work provides a first justification for the use of classic kernel theory to understand self-supervised learning of wide neural networks. Building on this result, we derive generalization error bounds for kernelized Barlow Twins and connect them to neural networks of finite width.