Mean Field Analysis of Neural Networks: A Law of Large Numbers

TL;DR

This paper provides a rigorous mathematical framework for understanding neural networks by analyzing their behavior in large-scale limits, demonstrating convergence to PDE solutions and the independence of parameters.

Contribution

It introduces a law of large numbers for neural networks, connecting their training dynamics to nonlinear PDEs and establishing propagation of chaos in the asymptotic regime.

Findings

Empirical distribution of parameters converges to a PDE solution.

Parameters become asymptotically independent during training.

Provides a mathematical foundation for neural network analysis.

Abstract

Machine learning, and in particular neural network models, have revolutionized fields such as image, text, and speech recognition. Today, many important real-world applications in these areas are driven by neural networks. There are also growing applications in engineering, robotics, medicine, and finance. Despite their immense success in practice, there is limited mathematical understanding of neural networks. This paper illustrates how neural networks can be studied via stochastic analysis, and develops approaches for addressing some of the technical challenges which arise. We analyze one-layer neural networks in the asymptotic regime of simultaneously (A) large network sizes and (B) large numbers of stochastic gradient descent training iterations. We rigorously prove that the empirical distribution of the neural network parameters converges to the solution of a nonlinear partial differential equation. This result can be considered a law of large numbers for neural networks. In addition, a consequence of our analysis is that the trained parameters of the neural network asymptotically become independent, a property which is commonly called "propagation of chaos".