Approximate Spectral Decomposition of Fisher Information Matrix for Simple ReLU Networks

TL;DR

This paper provides an approximate spectral decomposition of the Fisher Information Matrix for simple ReLU networks, revealing the structure of its eigenvalues and eigenvectors under certain conditions.

Contribution

It offers a novel theoretical characterization of the FIM's eigenstructure for one hidden layer ReLU networks, which was previously not well-understood.

Findings

Eigenvalues cluster into three groups with specific eigenvectors.

The largest eigenvalue corresponds to the Perron-Frobenius eigenvalue.

Eigenvectors of the second cluster span the row space of W.

Abstract

We argue the Fisher information matrix (FIM) of one hidden layer networks with the ReLU activation function. For a network, let $W$ denote the $d \times p$ weight matrix from the $d$-dimensional input to the hidden layer consisting of $p$ neurons, and $v$ the $p$-dimensional weight vector from the hidden layer to the scalar output. We focus on the FIM of $v$, which we denote as $I$. Under certain conditions, we characterize the first three clusters of eigenvalues and eigenvectors of the FIM. Specifically, we show that 1) Since $I$ is non-negative owing to the ReLU, the first eigenvalue is the Perron-Frobenius eigenvalue. 2) For the cluster of the next maximum values, the eigenspace is spanned by the row vectors of $W$. 3) The direct sum of the eigenspace of the first eigenvalue and that of the third cluster is spanned by the set of all the vectors obtained as the Hadamard product of any pair of the row vectors of $W$. We confirmed by numerical calculation that the above is approximately correct when the number of hidden nodes is about 10000.