Random Neural Networks in the Infinite Width Limit as Gaussian Processes

TL;DR

This paper proves that fully connected neural networks with random weights converge to Gaussian processes as hidden layer widths grow infinitely large, under broad conditions on weight distributions and nonlinearities.

Contribution

It provides a new proof of Gaussian process convergence for neural networks, requiring only moment conditions and accommodating general nonlinearities.

Findings

Neural networks with random weights converge to Gaussian processes in the infinite width limit.

Convergence holds under minimal moment conditions on weight distributions.

The proof applies to a wide class of nonlinear activation functions.

Abstract

This article gives a new proof that fully connected neural networks with random weights and biases converge to Gaussian processes in the regime where the input dimension, output dimension, and depth are kept fixed, while the hidden layer widths tend to infinity. Unlike prior work, convergence is shown assuming only moment conditions for the distribution of weights and for quite general non-linearities.