Global law of conjugate kernel random matrices with heavy-tailed weights

TL;DR

This paper analyzes the spectral distribution of conjugate kernel matrices in neural networks with heavy-tailed weight distributions, revealing unique spectral behaviors due to heavy tails.

Contribution

It extends the understanding of neural network spectral properties to heavy-tailed weight matrices, including stable laws and sparse matrices, with new theoretical results.

Findings

Heavy-tailed weights cause strong correlations in $Y$ entries.

Spectral distribution differs significantly from light-tailed cases.

Derived moments characterize the limiting eigenvalue distribution.

Abstract

We study the asymptotic spectral distribution of the conjugate kernel random matrix $YY^\top$, where $Y= f(WX)$ arises from a two-layer neural network model. We consider the setting where $W$ and $X$ are random rectangular matrices with i.i.d.\ entries, where the entries of $W$ follow a heavy-tailed distribution, while those of $X$ have light tails. Our assumptions on $W$ include a broad class of heavy-tailed distributions, such as symmetric $\alpha$-stable laws with $\alpha \in ]0,2[$ and sparse matrices with $\mathcal{O}(1)$ nonzero entries per row. The activation function $f$, applied entrywise, is bounded, smooth, odd, and nonlinear. We compute the limiting eigenvalue distribution of $YY^\top$ through its moments and show that heavy-tailed weights induce strong correlations between the entries of $Y$, resulting in richer and fundamentally different spectral behavior compared to the light-tailed case.