Finite-Dimensional Gaussian Approximation for Deep Neural Networks: Universality in Random Weights

TL;DR

This paper proves that deep neural networks with random weights and finite moments can be approximated by Gaussian distributions, with convergence rates depending on layer widths and depth, under certain conditions.

Contribution

It establishes Gaussian approximation bounds for neural network distributions with finite moments, extending understanding of their universality properties.

Findings

Gaussian approximation bounds in Wasserstein-1 norm

Convergence rates depend on layer widths and depth

Applicable to networks with Lipschitz activation functions

Abstract

We study the Finite-Dimensional Distributions (FDDs) of deep neural networks with randomly initialized weights that have finite-order moments. Specifically, we establish Gaussian approximation bounds in the Wasserstein-$1$ norm between the FDDs and their Gaussian limit assuming a Lipschitz activation function and allowing the layer widths to grow to infinity at arbitrary relative rates. In the special case where all widths are proportional to a common scale parameter $n$ and there are $L-1$ hidden layers, we obtain convergence rates of order $n^{-({1}/{6})^{L-1} + \epsilon}$, for any $\epsilon > 0$.