What Can One Expect When Solving PDEs Using Shallow Neural Networks?

TL;DR

This paper investigates the limitations and behaviors of shallow neural networks in solving elliptic PDEs, highlighting issues like ill-conditioning and frequency bias, and explores potential mitigation strategies.

Contribution

It provides a detailed analysis of the properties of shallow neural networks in PDE solutions, emphasizing the impact of activation functions and network formulation.

Findings

Linear SNNs with power ReLU are affected by ill-conditioning and spectral bias.

Using non-homogeneous activation functions can alleviate some issues.

Achieving adaptivity in nonlinear SNNs remains computationally costly.

Abstract

We use elliptic partial differential equations (PDEs) as examples to show various properties and behaviors when shallow neural networks (SNNs) are used to represent the solutions. In particular, we study the numerical ill-conditioning, frequency bias, and the balance between the differential operator and the shallow network representation for different formulations of the PDEs and with various activation functions. Our study shows that the performance of Physics-Informed Neural Networks (PINNs) or Deep Ritz Method (DRM) using linear SNNs with power ReLU activation is dominated by their inherent ill-conditioning and spectral bias against high frequencies. Although this can be alleviated by using non-homogeneous activation functions with proper scaling, achieving such adaptivity for nonlinear SNNs remains costly due to ill-conditioning.