Wide stable neural networks: Sample regularity, functional convergence and Bayesian inverse problems

TL;DR

This paper investigates the behavior of wide neural networks with heavy-tailed stable distributions, showing their sample functions converge in fractional Sobolev spaces and applying this to Bayesian inverse problems.

Contribution

It establishes the functional convergence of neural network outputs with stable weights in Sobolev spaces and applies this to Bayesian inverse problems.

Findings

Neural network outputs with stable weights lie in fractional Sobolev spaces.

Functional convergence of these processes is proven in the space of probability measures.

Application to edge-preserving Bayesian inverse problems with stable priors.

Abstract

We study the large-width asymptotics of random fully connected neural networks with weights drawn from $\alpha$-stable distributions, a family of heavy-tailed distributions arising as the limiting distributions in the Gnedenko-Kolmogorov heavy-tailed central limit theorem. We show that in an arbitrary bounded Euclidean domain $\mathcal{U}$ with smooth boundary, the random field at the infinite-width limit, characterized in previous literature in terms of finite-dimensional distributions, has sample functions in the fractional Sobolev-Slobodeckij-type quasi-Banach function space $W^{s,p}(\mathcal{U})$ for integrability indices $p < \alpha$ and suitable smoothness indices $s$ depending on the activation function of the neural network, and establish the functional convergence of the processes in the space of probability measures on $W^{s,p}(\mathcal{U})$. This convergence result is leveraged in the study of functional posteriors for edge-preserving Bayesian inverse problems with stable neural network priors.