Universality in Deep Neural Networks: An approach via the Lindeberg exchange principle

TL;DR

This paper investigates the behavior of deep neural networks as their width approaches infinity, providing bounds on their convergence to Gaussian limits using a novel Lindeberg principle.

Contribution

It introduces a Lindeberg exchange principle tailored for deep neural networks to quantify their convergence to Gaussian processes in the infinite-width limit.

Findings

Provides quantitative bounds on Wasserstein distance between finite and infinite-width networks.

Establishes a Lindeberg-based method for analyzing deep neural network limits.

Abstract

We consider the infinite-width limit of a fully connected deep neural network with general weights, and we prove quantitative general bounds on the $2$-Wasserstein distance between the network and its infinite-width Gaussian limit, under appropriate regularity assumptions on the activation function. Our main tool is a Lindeberg principle for Deep Neural Networks, which we use to successively replace the weights on each layer by Gaussian random variables.