From Frequency Bias to Spectral Balance: Operator-Aware Preconditioners for PINNs

TL;DR

This paper introduces an operator-aware preconditioning method for physics-informed neural networks (PINNs) that balances frequency biases, leading to improved training efficiency and accuracy in solving elliptic PDEs.

Contribution

The paper proposes a novel preconditioning strategy using an auxiliary integral operator to counteract frequency bias in PINNs, enhancing convergence and solution quality.

Findings

Restores balanced learning dynamics across frequency modes.

Significantly improves convergence speed.

Enhances accuracy in multiscale and variable-coefficient problems.

Abstract

When neural networks (NNs) are used as a type of nonlinear parametric representation to solve partial differential equations (PDEs), they often display frequency-dependent learning dynamics that can differ from those seen in direct function approximation tasks, resulting from a balance between the frequency bias of the NN representation and that of the underlying differential operator. Although many commonly used NNs exhibit a bias towards low-frequency modes in representation, the presence of differential operators in the loss function, which amplifies high-frequency components, can lead to high frequency bias. In this work, using second order elliptic PDEs as an example, we show how these two factors compete and lead to an overall frequency bias in different situations. Once the balance is determined, it is important to design computational strategies to counter the resulting bias to improve training efficiency. We propose a simple operator-aware preconditioning strategy that rebalances the optimization landscape and the learning dynamics by applying an auxiliary integral operator to the residual. The integral kernel can be the Green's function of a reference elliptic operator or an approximation, and integrates easily with common NN solvers for PDEs. Extensive experiments, including multiscale and variable-coefficient problems, show that the approach restores more balanced learning dynamics across modes and substantially improves both convergency and accuracy.