Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks

TL;DR

This paper explicitly models the implicit bias of the Frequency Principle in two-layer neural networks, showing it as a penalty on high-frequency components, which helps understand their generalization behavior.

Contribution

It introduces a linear F-Principle dynamics model for two-layer ReLU networks and explicitly formulates the implicit bias as a penalization of high frequencies, advancing theoretical understanding.

Findings

The LFP dynamics accurately predicts learning outcomes.

The implicit bias is equivalent to minimizing an FP-norm with high frequencies penalized.

Higher FP-norm of the target increases generalization error.

Abstract

It remains a puzzle that why deep neural networks (DNNs), with more parameters than samples, often generalize well. An attempt of understanding this puzzle is to discover implicit biases underlying the training process of DNNs, such as the Frequency Principle (F-Principle), i.e., DNNs often fit target functions from low to high frequencies. Inspired by the F-Principle, we propose an effective model of linear F-Principle (LFP) dynamics which accurately predicts the learning results of two-layer ReLU neural networks (NNs) of large widths. This LFP dynamics is rationalized by a linearized mean field residual dynamics of NNs. Importantly, the long-time limit solution of this LFP dynamics is equivalent to the solution of a constrained optimization problem explicitly minimizing an FP-norm, in which higher frequencies of feasible solutions are more heavily penalized. Using this optimization formulation, an a priori estimate of the generalization error bound is provided, revealing that a higher FP-norm of the target function increases the generalization error. Overall, by explicitizing the implicit bias of the F-Principle as an explicit penalty for two-layer NNs, our work makes a step towards a quantitative understanding of the learning and generalization of general DNNs.