Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks

TL;DR

This paper demonstrates that large multilayer neural networks exhibit a simplified limiting behavior independent of their size, described by a set of equations, revealing a new operational regime validated by experiments.

Contribution

It introduces a formalism capturing the mean field limit of neural network dynamics, uncovering a novel regime where behavior simplifies as network size grows.

Findings

Behavior becomes independent of the number of neurons at large scale

Development of a formal set of equations describing the limit behavior

Experimental validation of the limiting regime

Abstract

Can multilayer neural networks -- typically constructed as highly complex structures with many nonlinearly activated neurons across layers -- behave in a non-trivial way that yet simplifies away a major part of their complexities? In this work, we uncover a phenomenon in which the behavior of these complex networks -- under suitable scalings and stochastic gradient descent dynamics -- becomes independent of the number of neurons as this number grows sufficiently large. We develop a formalism in which this many-neurons limiting behavior is captured by a set of equations, thereby exposing a previously unknown operating regime of these networks. While the current pursuit is mathematically non-rigorous, it is complemented with several experiments that validate the existence of this behavior.