A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insights

TL;DR

This paper analyzes neural network learning dynamics from a Fourier perspective, revealing how amplitude and phase information influence sample complexity and learning speed, especially in natural images.

Contribution

It introduces a Fourier-based framework to understand simplicity bias, demonstrating how power-law spectra accelerate phase learning and providing mechanistic insights into natural image processing.

Findings

Neural networks rely on amplitude information before phase information in image classification.

Classification based on phase alone is computationally hard without spectral advantages.

Power-law spectra significantly speed up phase learning, aiding natural image understanding.

Abstract

Neural networks trained with gradient-based methods exhibit a strong simplicity bias: they learn simpler statistical features of their data before moving to more complex features. Previous analyses of this phenomenon have largely focused on settings with (quasi-)isotropic inputs. In this work, we study the simplicity bias from a Fourier perspective, which allows us to include two key features of natural images in the analysis: approximate translation-invariance and power-law spectra. We first show experimentally that simple neural networks trained on image classification tasks first rely on amplitude information -- related to pair-wise correlations between pixels -- before exploiting phase information, which encodes edges and higher-order correlations. In view of this, we introduce a synthetic data model for translation-invariant inputs that allows precise control over amplitudes and phases while remaining tractable. We rigorously establish that for isotropic and high-dimensional inputs, classification based on phase information alone is a genuinely hard task: online stochastic gradient descent (SGD) cannot distinguish the structured inputs from noise within $n \ll N^3$ steps, but needs at least $n \gg N^3 \log^2{N}$ steps. In contrast, we show both experimentally and theoretically that power-law spectra can dramatically accelerate the speed of learning phase information, even if the spectra do not help with classification. Simulations with two-layer networks trained on textures and with deep convolutional networks on ImageNet and CIFAR100 confirm this non-trivial interaction between amplitudes and phases, providing mechanistic insights into how deep neural networks can learn natural image distributions efficiently.