Overview frequency principle/spectral bias in deep learning

TL;DR

This paper reviews the Frequency Principle in deep learning, highlighting how neural networks tend to learn low-frequency components first, and discusses its implications for understanding and designing deep learning models.

Contribution

It provides an overview of the F-Principle, summarizes recent research validating it, and discusses open problems for future exploration.

Findings

Neural networks learn low-frequency functions first

F-Principle explains neural network training dynamics

Understanding spectral bias informs model design

Abstract

Understanding deep learning is increasingly emergent as it penetrates more and more into industry and science. In recent years, a research line from Fourier analysis sheds lights on this magical "black box" by showing a Frequency Principle (F-Principle or spectral bias) of the training behavior of deep neural networks (DNNs) -- DNNs often fit functions from low to high frequency during the training. The F-Principle is first demonstrated by onedimensional synthetic data followed by the verification in high-dimensional real datasets. A series of works subsequently enhance the validity of the F-Principle. This low-frequency implicit bias reveals the strength of neural network in learning low-frequency functions as well as its deficiency in learning high-frequency functions. Such understanding inspires the design of DNN-based algorithms in practical problems, explains experimental phenomena emerging in various scenarios, and further advances the study of deep learning from the frequency perspective. Although incomplete, we provide an overview of F-Principle and propose some open problems for future research.