Understanding training and generalization in deep learning by Fourier analysis

TL;DR

This paper uses Fourier analysis to explain why deep neural networks prioritize low-frequency components during training and how small initialization improves generalization, supported by experiments on various datasets.

Contribution

It introduces a Fourier-based theoretical framework that explains the training dynamics and generalization properties of DNNs, emphasizing the role of initialization and frequency prioritization.

Findings

DNNs prioritize low-frequency components during training

Small initialization improves generalization while maintaining fitting ability

Experimental validation on natural images, 1D functions, and MNIST

Abstract

Background: It is still an open research area to theoretically understand why Deep Neural Networks (DNNs)---equipped with many more parameters than training data and trained by (stochastic) gradient-based methods---often achieve remarkably low generalization error. Contribution: We study DNN training by Fourier analysis. Our theoretical framework explains: i) DNN with (stochastic) gradient-based methods often endows low-frequency components of the target function with a higher priority during the training; ii) Small initialization leads to good generalization ability of DNN while preserving the DNN's ability to fit any function. These results are further confirmed by experiments of DNNs fitting the following datasets, that is, natural images, one-dimensional functions and MNIST dataset.