Eigenvalue distribution of nonlinear models of random matrices

TL;DR

This paper analyzes the asymptotic eigenvalue distribution of nonlinear random matrix ensembles, extending previous Gaussian results to sub-Gaussian cases and exploring multi-layer neural network models.

Contribution

It extends eigenvalue distribution analysis to sub-Gaussian matrices and multi-layer neural network models with nonlinear activations.

Findings

Derived the asymptotic empirical eigenvalue distribution for nonlinear random matrices.

Extended previous Gaussian matrix results to sub-Gaussian matrices.

Investigated eigenvalue distributions in multi-layer neural network models.

Abstract

This paper is concerned with the asymptotic empirical eigenvalue distribution of a non linear random matrix ensemble. More precisely we consider $M= \frac{1}{m} YY^*$ with $Y=f(WX)$ where $W$ and $X$ are random rectangular matrices with i.i.d. centered entries. The function $f$ is applied pointwise and can be seen as an activation function in (random) neural networks. We compute the asymptotic empirical distribution of this ensemble in the case where $W$ and $X$ have sub-Gaussian tails and $f$ is real analytic. This extends a previous result where the case of Gaussian matrices $W$ and $X$ is considered. We also investigate the same questions in the multi-layer case, regarding neural network applications.