Infinitely wide limits for deep Stable neural networks: sub-linear, linear and super-linear activation functions

TL;DR

This paper explores the behavior of deep neural networks with heavy-tailed Stable-distributed parameters as their width tends to infinity, extending previous Gaussian-based results to a broader class of activation functions using a generalized central limit theorem.

Contribution

It extends the characterization of infinitely wide limits of deep Stable neural networks to general activation functions under sequential growth, utilizing a generalized CLT for heavy tails.

Findings

Stable NN limits depend on activation function choice

Unified treatment of wide limits via heavy-tail CLT

Differences from Gaussian case in stability and scaling

Abstract

There is a growing literature on the study of large-width properties of deep Gaussian neural networks (NNs), i.e. deep NNs with Gaussian-distributed parameters or weights, and Gaussian stochastic processes. Motivated by some empirical and theoretical studies showing the potential of replacing Gaussian distributions with Stable distributions, namely distributions with heavy tails, in this paper we investigate large-width properties of deep Stable NNs, i.e. deep NNs with Stable-distributed parameters. For sub-linear activation functions, a recent work has characterized the infinitely wide limit of a suitable rescaled deep Stable NN in terms of a Stable stochastic process, both under the assumption of a ``joint growth" and under the assumption of a ``sequential growth" of the width over the NN's layers. Here, assuming a ``sequential growth" of the width, we extend such a characterization to a general class of activation functions, which includes sub-linear, asymptotically linear and super-linear functions. As a novelty with respect to previous works, our results rely on the use of a generalized central limit theorem for heavy tails distributions, which allows for an interesting unified treatment of infinitely wide limits for deep Stable NNs. Our study shows that the scaling of Stable NNs and the stability of their infinitely wide limits may depend on the choice of the activation function, bringing out a critical difference with respect to the Gaussian setting.