Infinite-width limit of deep linear neural networks

TL;DR

This paper analyzes the behavior of deep linear neural networks as their width approaches infinity, revealing convergence properties of training dynamics and the predictors to deterministic limits with exponential convergence rates.

Contribution

It provides a rigorous analysis of the infinite-width limit of deep linear neural networks, including convergence of training dynamics and the law of weights during training.

Findings

Training dynamics converge to a deterministic limit as width increases.

The linear predictor converges exponentially to the minimal $\

The law of weights along training can be precisely characterized.

Abstract

This paper studies the infinite-width limit of deep linear neural networks initialized with random parameters. We obtain that, when the number of neurons diverges, the training dynamics converge (in a precise sense) to the dynamics obtained from a gradient descent on an infinitely wide deterministic linear neural network. Moreover, even if the weights remain random, we get their precise law along the training dynamics, and prove a quantitative convergence result of the linear predictor in terms of the number of neurons. We finally study the continuous-time limit obtained for infinitely wide linear neural networks and show that the linear predictors of the neural network converge at an exponential rate to the minimal $\ell_2$-norm minimizer of the risk.