Large deviation principles and functional limit theorems in the deep limit of wide random neural networks

TL;DR

This paper investigates large deviation principles and weak convergence for Gaussian fields modeling the statistical behavior of deep wide neural networks, revealing different regimes based on the covariance function's derivative.

Contribution

It introduces a recursive covariance evolution framework for neural networks and characterizes large deviations and convergence across three regimes.

Findings

Established functional large deviation principles in low-disorder regime

Proved weak convergence results for Gaussian fields in certain regimes

Identified failure of functional properties in the sparse regime due to covariance discontinuities

Abstract

This paper studies large deviation principles and weak convergence, both at the level of finite-dimensional distributions and in functional form, for a class of continuous, isotropic, centered Gaussian random fields defined on the unit sphere. The covariance functions of these fields evolve recursively through a nonlinear map induced by an activation function, reflecting the statistical dynamics of infinitely wide random neural networks as depth increases. We consider two types of centered fields, obtained by subtracting either the value at the North Pole or the spherical average. According to the behavior of the derivative at $t=1$ of the associated covariance function, we identify three regimes: low disorder, sparse, and high disorder. In the low-disorder regime, we establish functional large deviation principles and weak convergence results. In the sparse regime, we obtain large deviation principles and weak convergence for finite-dimensional distributions, while both properties fail at the functional level sense due to the emergence of discontinuities in the covariance recursion.