Is the Frequency Principle always valid?

TL;DR

This paper examines the frequency principle in shallow ReLU neural networks on the sphere, revealing conditions under which it holds or is violated, and how trainable directions influence learning dynamics and frequency emergence.

Contribution

It provides a harmonic analysis of the frequency principle on curved domains, highlighting the impact of initial conditions and trainable directions on learning behavior.

Findings

Spherical harmonic coefficients decay exponentially, supporting the frequency principle.

The principle can be violated under specific initial conditions or error distributions.

Trainable directions can either preserve or accelerate high-frequency learning.

Abstract

We investigate the learning dynamics of shallow ReLU neural networks on the unit sphere \(S^2\subset\mathbb{R}^3\) in polar coordinates \((\tau,\phi)\), considering both fixed and trainable neuron directions \(\{w_i\}\). For fixed weights, spherical harmonic expansions reveal an intrinsic low-frequency preference with coefficients decaying as \(O(\ell^{5/2}/2^\ell)\), typically leading to the Frequency Principle (FP) of lower-frequency-first learning. However, this principle can be violated under specific initial conditions or error distributions. With trainable weights, an additional rotation term in the harmonic evolution equations preserves exponential decay with decay order \(O(\ell^{7/2}/2^\ell)\) factor, also leading to the FP of lower-frequency-first learning. But like fixed weights case, the principle can be violated under specific initial conditions or error distributions. Our numerical results demonstrate that trainable directions increase learning complexity and can either maintain a low-frequency advantage or enable faster high-frequency emergence. This analysis suggests the FP should be viewed as a tendency rather than a rule on curved domains like \(S^2\), providing insights into how direction updates and harmonic expansions shape frequency-dependent learning.