Neural Scaling Laws From Large-N Field Theory: Solvable Model Beyond the Ridgeless Limit

TL;DR

This paper uses large-N field theory to analytically derive and extend neural scaling laws in a simplified model, revealing a duality that explains the symmetry between model and data set sizes.

Contribution

It extends previous work by solving a model with nonzero ridge parameter, providing more precise scaling laws and uncovering a duality transformation at the diagram level.

Findings

Derived new scaling laws for neural networks with regularization.

Uncovered a duality explaining symmetry between model and data sizes.

Extended theoretical understanding of neural scaling beyond the ridgeless limit.

Abstract

Many machine learning models based on neural networks exhibit scaling laws: their performance scales as power laws with respect to the sizes of the model and training data set. We use large-N field theory methods to solve a model recently proposed by Maloney, Roberts and Sully which provides a simplified setting to study neural scaling laws. Our solution extends the result in this latter paper to general nonzero values of the ridge parameter, which are essential to regularize the behavior of the model. In addition to obtaining new and more precise scaling laws, we also uncover a duality transformation at the diagrams level which explains the symmetry between model and training data set sizes. The same duality underlies recent efforts to design neural networks to simulate quantum field theories.