Wilsonian Renormalization of Neural Network Gaussian Processes

TL;DR

This paper applies Wilsonian renormalization group techniques from theoretical physics to Gaussian Process regression, providing a new analytical framework to understand feature learning and universality in neural networks.

Contribution

It introduces a practical method for performing RG analysis on Gaussian Processes, linking RG flow to learnable and unlearnable modes in neural network models.

Findings

RG flow of the GP kernel can be analytically derived

Universal flow of the ridge parameter in simple cases

Input-dependent flow when non-Gaussianities are included

Abstract

Separating relevant and irrelevant information is key to any modeling process or scientific inquiry. Theoretical physics offers a powerful tool for achieving this in the form of the renormalization group (RG). Here we demonstrate a practical approach to performing Wilsonian RG in the context of Gaussian Process (GP) Regression. We systematically integrate out the unlearnable modes of the GP kernel, thereby obtaining an RG flow of the GP in which the data sets the IR scale. In simple cases, this results in a universal flow of the ridge parameter, which becomes input-dependent in the richer scenario in which non-Gaussianities are included. In addition to being analytically tractable, this approach goes beyond structural analogies between RG and neural networks by providing a natural connection between RG flow and learnable vs. unlearnable modes. Studying such flows may improve our understanding of feature learning in deep neural networks, and enable us to identify potential universality classes in these models.