Large and moderate deviations for Gaussian neural networks

TL;DR

This paper establishes large and moderate deviation principles for the outputs of Gaussian neural networks, including deep and shallow architectures with various activation functions.

Contribution

It provides the first comprehensive large and moderate deviation results for Gaussian neural networks, covering deep, shallow, and ReLU-activated models.

Findings

Large deviations hold for deep networks with bounded activations.

Results extend to ReLU activations in single-input deep networks.

Shallow networks exhibit large and moderate deviation principles for general activations.

Abstract

We prove large and moderate deviations for the output of Gaussian fully connected neural networks. The main achievements concern deep neural networks (i.e., when the model has more than one hidden layer) and hold for bounded and continuous pre-activation functions. However, for deep neural networks fed by a single input, we have results even if the pre-activation is ReLU. When the network is shallow (i.e., there is exactly one hidden layer) the large and moderate principles hold for quite general pre-activation functions.