On the validity of kernel approximations for orthogonally-initialized neural networks

TL;DR

This paper extends kernel approximation results from Gaussian to orthogonally-initialized neural networks using Haar-distributed matrices, leveraging recent random matrix theory insights.

Contribution

It introduces a novel extension of kernel approximation analysis to orthogonal initializations, broadening understanding of neural network behavior.

Findings

Kernel approximation results hold for orthogonally-initialized networks.

Uses random matrix theory to establish theoretical guarantees.

Extends prior Gaussian-based analyses to Haar-distributed orthogonal matrices.

Abstract

In this note we extend kernel function approximation results for neural networks with Gaussian-distributed weights to single-layer networks initialized using Haar-distributed random orthogonal matrices (with possible rescaling). This is accomplished using recent results from random matrix theory.