On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels

TL;DR

This paper analyzes the spectral properties of convolutional neural tangent and Gaussian process kernels, revealing how their eigenfunctions and eigenvalues behave, which helps understand the function spaces these over-parameterized models can represent.

Contribution

The paper provides a theoretical analysis of the eigenfunctions and eigenvalue decay of convolutional kernels, offering a quantitative understanding of their spectral bias in over-parameterized CNNs.

Findings

Eigenfunctions are products of spherical harmonics over pixel channels.

Eigenvalues decay polynomially with a quantifiable rate.

Measures are derived to reflect hierarchical feature composition.

Abstract

We study the properties of various over-parametrized convolutional neural architectures through their respective Gaussian process and neural tangent kernels. We prove that, with normalized multi-channel input and ReLU activation, the eigenfunctions of these kernels with the uniform measure are formed by products of spherical harmonics, defined over the channels of the different pixels. We next use hierarchical factorizable kernels to bound their respective eigenvalues. We show that the eigenvalues decay polynomially, quantify the rate of decay, and derive measures that reflect the composition of hierarchical features in these networks. Our results provide concrete quantitative characterization of over-parameterized convolutional network architectures.