Approximating Simple ReLU Networks based on Spectral Decomposition of Fisher Information

TL;DR

This paper analyzes the spectral properties of Fisher information matrices in 2-layer ReLU networks, revealing dominant eigenspaces linked to spherical harmonic functions, which informs network approximation.

Contribution

It identifies the function spaces associated with the major eigenspaces of Fisher information matrices in ReLU networks, connecting spectral properties to spherical harmonics.

Findings

Eigenvalues concentrate on a few eigenspaces, accounting for 97.7% of the trace.

Major eigenspaces correspond to spherical harmonic functions of order up to 2.

Results relate to the Mercer decomposition of neural tangent kernels.

Abstract

Properties of Fisher information matrices of 2-layer neural ReLU networks with random hidden weights are studied. For these networks, it is known that the eigenvalue distribution highly concentrates on several eigenspaces approximately. In particular, the eigenvalues for the first three eigenspaces account for 97.7% of the trace of the Fisher information matrix, independently of the number of parameters. In this paper, we identify the function spaces which correspond to those major eigenspaces. This function space consists of the spherical harmonic functions whose orders are not greater than 2. This result relates to the Mercer decomposition of the neural tangent kernels.