Renormalization group for deep neural networks: Universality of learning and scaling laws

TL;DR

This paper applies the renormalization group framework to analyze self-similarity, universality, and scaling laws in deep neural networks trained on power-law distributed data, revealing new insights into their learning behavior.

Contribution

It develops an RG-based approach tailored for neural networks that accounts for spectrum discreteness and lack of translation invariance, extending traditional RG analysis to deep learning.

Findings

Identification of scaling intervals replacing scaling dimensions.

Classification of perturbations as relevant or irrelevant.

Discovery of a universality class governed by a Gaussian Process-like UV fixed point.

Abstract

Self-similarity, where observables at different length scales exhibit similar behavior, is ubiquitous in natural systems. Such systems are typically characterized by power-law correlations and universality, and are studied using the powerful framework of the renormalization group (RG). Intriguingly, power laws and weak forms of universality also pervade real-world datasets and deep learning models, motivating the application of RG ideas to the analysis of deep learning. In this work, we develop an RG framework to analyze self-similarity and its breakdown in learning curves for a class of weakly non-linear (non-lazy) neural networks trained on power-law distributed data. Features often neglected in standard treatments -- such as spectrum discreteness and lack of translation invariance -- lead to both quantitative and qualitative departures from conventional perturbative RG. In particular, we find that the concept of scaling intervals naturally replaces that of scaling dimensions. Despite these differences, the framework retains key RG features: it enables the classification of perturbations as relevant or irrelevant, and reveals a form of universality at large data limits, governed by a Gaussian Process-like UV fixed point.