On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case

TL;DR

This paper analyzes the singular value distribution of products of random matrices in deep neural networks, extending previous results to more general i.i.d. entries and validating the universality of these properties.

Contribution

It introduces a streamlined random matrix theory approach to generalize singular value distribution results to i.i.d. matrices with finite fourth moments in neural network models.

Findings

Extended universality results to i.i.d. matrices with finite fourth moments.

Validated the applicability of free probability techniques in neural network analysis.

Provided a more general framework for understanding singular values in deep learning models.

Abstract

We study the distribution of singular values of product of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices, however, an important difference is that the population covariance matrices assumed to be non-random or random but independent of the random data matrix in statistics and random matrix theory are now certain functions of random data matrices (synaptic weight matrices in the deep neural network terminology). The problem has been treated in recent work [25, 13] by using the techniques of free probability theory. Since, however, free probability theory deals with population covariance matrices which are independent of the data matrices, its applicability has to be justified. The justification has been given in [22] for Gaussian data matrices with independent entries, a standard analytical model of free probability, by using a version of the techniques of random matrix theory. In this paper we use another, more streamlined, version of the techniques of random matrix theory to generalize the results of [22] to the case where the entries of the synaptic weight matrices are just independent identically distributed random variables with zero mean and finite fourth moment. This, in particular, extends the property of the so-called macroscopic universality on the considered random matrices.