LDP for the covariance process in fully connected neural networks

TL;DR

This paper establishes a large deviation principle for the covariance process in fully connected Gaussian neural networks, providing insights into the probabilistic behavior of neural network covariance in high-dimensional regimes.

Contribution

It introduces a novel large deviation principle for the covariance process in neural networks, applicable in infinite-width and mean-field regimes, with a new functional framework and Markov process analysis.

Findings

LDP for the covariance process in neural networks

Posterior LDPs under Gaussian likelihood

Analysis in infinite-width and mean-field regimes

Abstract

In this work, we study large deviation properties of the covariance process in fully connected Gaussian deep neural networks. More precisely, we establish a large deviation principle (LDP) for the covariance process in a functional framework, viewing it as a process in the space of continuous functions. As key applications of our main results, we obtain posterior LDPs under Gaussian likelihood in both the infinite-width and mean-field regimes. The proof is based on an LDP for the covariance process as a Markov process valued in the space of non-negative, symmetric trace-class operators equipped with the trace norm.