# Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

## TL;DR

This paper reveals that deep neural networks tend to learn target functions starting from low to high frequencies, which explains their generalization ability and contrasts with traditional numerical methods.

## Contribution

It introduces the Frequency Principle (F-Principle), showing that DNNs inherently fit low-frequency components first, supported by theoretical analysis and empirical validation on benchmark datasets.

## Key findings

- DNNs fit low to high frequencies during training
- F-Principle explains DNNs' good generalization on real data
- F-Principle contrasts with conventional iterative schemes

## Abstract

We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective. We demonstrate a very universal Frequency Principle (F-Principle) -- DNNs often fit target functions from low to high frequencies -- on high-dimensional benchmark datasets such as MNIST/CIFAR10 and deep neural networks such as VGG16. This F-Principle of DNNs is opposite to the behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibit faster convergence for higher frequencies for various scientific computing problems. With a simple theory, we illustrate that this F-Principle results from the regularity of the commonly used activation functions. The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function. This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or randomized dataset.