Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks

TL;DR

This paper provides a spectral analysis of the neural tangent kernel for deep residual networks, revealing their eigenfunctions, eigenvalue decay, and the impact of hyper-parameters on kernel stability, enhancing understanding of residual network behavior.

Contribution

It offers a novel spectral analysis of ResNTK, showing its eigenfunctions, eigenvalue decay, and the influence of hyper-parameters on kernel stability, connecting residual networks to classical kernels.

Findings

Eigenfunctions of ResNTK are spherical harmonics.

Eigenvalues decay polynomially as k^{-d}.

ResNTK can be tuned to be stable or spiky with depth.

Abstract

Deep residual network architectures have been shown to achieve superior accuracy over classical feed-forward networks, yet their success is still not fully understood. Focusing on massively over-parameterized, fully connected residual networks with ReLU activation through their respective neural tangent kernels (ResNTK), we provide here a spectral analysis of these kernels. Specifically, we show that, much like NTK for fully connected networks (FC-NTK), for input distributed uniformly on the hypersphere $\mathbb{S}^{d-1}$, the eigenfunctions of ResNTK are the spherical harmonics and the eigenvalues decay polynomially with frequency $k$ as $k^{-d}$. These in turn imply that the set of functions in their Reproducing Kernel Hilbert Space are identical to those of FC-NTK, and consequently also to those of the Laplace kernel. We further show, by drawing on the analogy to the Laplace kernel, that depending on the choice of a hyper-parameter that balances between the skip and residual connections ResNTK can either become spiky with depth, as with FC-NTK, or maintain a stable shape.