On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks

TL;DR

This paper investigates the eigenvector bias in Fourier feature neural networks used in physics-informed neural networks (PINNs), proposing multi-scale Fourier features to improve modeling of multi-scale PDEs, especially in high-frequency regimes.

Contribution

The authors identify the eigenvector bias in Fourier feature networks and introduce multi-scale Fourier features to enhance PINN performance on multi-scale PDE problems.

Findings

Multi-scale Fourier features improve PINN accuracy on high-frequency PDEs.

Proposed architectures outperform conventional PINNs in wave and reaction-diffusion problems.

Code and data are publicly available for reproducibility.

Abstract

Physics-informed neural networks (PINNs) are demonstrating remarkable promise in integrating physical models with gappy and noisy observational data, but they still struggle in cases where the target functions to be approximated exhibit high-frequency or multi-scale features. In this work we investigate this limitation through the lens of Neural Tangent Kernel (NTK) theory and elucidate how PINNs are biased towards learning functions along the dominant eigen-directions of their limiting NTK. Using this observation, we construct novel architectures that employ spatio-temporal and multi-scale random Fourier features, and justify how such coordinate embedding layers can lead to robust and accurate PINN models. Numerical examples are presented for several challenging cases where conventional PINN models fail, including wave propagation and reaction-diffusion dynamics, illustrating how the proposed methods can be used to effectively tackle both forward and inverse problems involving partial differential equations with multi-scale behavior. All code an data accompanying this manuscript will be made publicly available at \url{https://github.com/PredictiveIntelligenceLab/MultiscalePINNs}.