Random ReLU Neural Networks as Non-Gaussian Processes

TL;DR

This paper demonstrates that shallow neural networks with ReLU activations and random parameters can be modeled as non-Gaussian stochastic processes, revealing new insights into their behavior beyond classical Gaussian limits.

Contribution

It introduces a non-asymptotic framework showing these networks form non-Gaussian processes driven by impulsive noise, extending classical Gaussian process convergence results.

Findings

Networks are well-defined non-Gaussian processes

Processes are isotropic and self-similar with Hurst exponent 3/2

Convergence to non-Gaussian processes under certain conditions

Abstract

We consider a large class of shallow neural networks with randomly initialized parameters and rectified linear unit activation functions. We prove that these random neural networks are well-defined non-Gaussian processes. As a by-product, we demonstrate that these networks are solutions to stochastic differential equations driven by impulsive white noise (combinations of random Dirac measures). These processes are parameterized by the law of the weights and biases as well as the density of activation thresholds in each bounded region of the input domain. We prove that these processes are isotropic and wide-sense self-similar with Hurst exponent 3/2. We also derive a remarkably simple closed-form expression for their autocovariance function. Our results are fundamentally different from prior work in that we consider a non-asymptotic viewpoint: The number of neurons in each bounded region of the input domain (i.e., the width) is itself a random variable with a Poisson law with mean proportional to the density parameter. Finally, we show that, under suitable hypotheses, as the expected width tends to infinity, these processes can converge in law not only to Gaussian processes, but also to non-Gaussian processes depending on the law of the weights. Our asymptotic results provide a new take on several classical results (wide networks converge to Gaussian processes) as well as some new ones (wide networks can converge to non-Gaussian processes).