Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networks

TL;DR

This paper analyzes how underparameterized neural networks optimize mean squared error via gradient flow, revealing a spectral bias towards eigenfunctions of an integral operator related to the NTK, and introduces the concept of damped deviations to unify understanding of network dynamics.

Contribution

It introduces the concept of damped deviations to analyze neural network training dynamics and characterizes the spectral bias in underparameterized regimes.

Findings

Neural networks learn eigenfunctions of an integral operator at rates tied to eigenvalues.

Spectral bias favors certain eigenfunctions, such as spherical harmonics for data on spheres.

Damped deviations help unify analysis of both under- and overparameterized regimes.

Abstract

We study the dynamics of a neural network in function space when optimizing the mean squared error via gradient flow. We show that in the underparameterized regime the network learns eigenfunctions of an integral operator $T_{K^\infty}$ determined by the Neural Tangent Kernel (NTK) at rates corresponding to their eigenvalues. For example, for uniformly distributed data on the sphere $S^{d - 1}$ and rotation invariant weight distributions, the eigenfunctions of $T_{K^\infty}$ are the spherical harmonics. Our results can be understood as describing a spectral bias in the underparameterized regime. The proofs use the concept of "Damped Deviations", where deviations of the NTK matter less for eigendirections with large eigenvalues due to the occurence of a damping factor. Aside from the underparameterized regime, the damped deviations point-of-view can be used to track the dynamics of the empirical risk in the overparameterized setting, allowing us to extend certain results in the literature. We conclude that damped deviations offers a simple and unifying perspective of the dynamics when optimizing the squared error.