Neural Tangent Kernels and Fisher Information Matrices for Simple ReLU Networks with Random Hidden Weights

TL;DR

This paper explores the relationship between Fisher information matrices and neural tangent kernels in 2-layer ReLU networks with random weights, providing spectral analysis and approximation formulas.

Contribution

It establishes a linear transformation relation between Fisher matrices and NTKs and derives spectral decompositions with explicit eigenfunctions.

Findings

Spectral decomposition of NTK with major eigenvalues identified

Approximation formulas for functions represented by 2-layer networks derived

Relation between Fisher information and NTK clarified

Abstract

Fisher information matrices and neural tangent kernels (NTK) for 2-layer ReLU networks with random hidden weight are argued. We discuss the relation between both notions as a linear transformation and show that spectral decomposition of NTK with concrete forms of eigenfunctions with major eigenvalues. We also obtain an approximation formula of the functions presented by the 2-layer neural networks.