Condition Numbers and Eigenvalue Spectra of Shallow Networks on Spheres

TL;DR

This paper analyzes the condition numbers and eigenvalue spectra of matrices from shallow ReLU$^k$ neural networks on spheres, revealing how spectral properties relate to the network's approximation ability and stability.

Contribution

It provides sharp asymptotic estimates for eigenvalues and characterizes eigenspaces, linking spectral structure to approximation power and numerical stability.

Findings

Eigenvalues linked to polynomial degrees

Smallest eigenvalues associated with low-degree polynomials

Spectral estimates are sharp for antipodally quasi-uniform points

Abstract

We present an estimation of the condition numbers of the \emph{mass} and \emph{stiffness} matrices arising from shallow ReLU$^k$ neural networks defined on the unit sphere~$\mathbb{S}^d$. In particular, when $\{\theta_j^*\}_{j=1}^n \subset \mathbb{S}^d$ is \emph{antipodally quasi-uniform}, the condition number is sharp. Indeed, in this case, we obtain sharp asymptotic estimates for the full spectrum of eigenvalues and characterize the structure of the corresponding eigenspaces, showing that the smallest eigenvalues are associated with an eigenbasis of low-degree polynomials while the largest eigenvalues are linked to high-degree polynomials. This spectral analysis establishes a precise correspondence between the approximation power of the network and its numerical stability.