The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies

TL;DR

This paper investigates how the frequency of a function affects the learning speed of neural networks, deriving theoretical predictions that match empirical results for both shallow and deep models.

Contribution

It introduces a theoretical framework linking function frequency to neural network convergence rates, including the impact of bias terms on low-frequency function learning.

Findings

Bias terms are crucial for learning odd-frequency functions.

Theoretical predictions align with empirical learning times.

Low-frequency functions are learned faster than high-frequency ones.

Abstract

We study the relationship between the frequency of a function and the speed at which a neural network learns it. We build on recent results that show that the dynamics of overparameterized neural networks trained with gradient descent can be well approximated by a linear system. When normalized training data is uniformly distributed on a hypersphere, the eigenfunctions of this linear system are spherical harmonic functions. We derive the corresponding eigenvalues for each frequency after introducing a bias term in the model. This bias term had been omitted from the linear network model without significantly affecting previous theoretical results. However, we show theoretically and experimentally that a shallow neural network without bias cannot represent or learn simple, low frequency functions with odd frequencies. Our results lead to specific predictions of the time it will take a network to learn functions of varying frequency. These predictions match the empirical behavior of both shallow and deep networks.