Challenges in Training PINNs: A Loss Landscape Perspective

TL;DR

This paper investigates the training challenges of Physics-Informed Neural Networks (PINNs) from a loss landscape perspective, proposing new optimization strategies and analyzing the impact of differential operators on training difficulty.

Contribution

It introduces a novel second-order optimizer NysNewton-CG and analyzes the benefits of combining first- and second-order methods for PINNs.

Findings

Adam+L-BFGS outperforms individual optimizers.

NysNewton-CG significantly improves PINN training.

Ill-conditioning from differential operators affects loss landscape.

Abstract

This paper explores challenges in training Physics-Informed Neural Networks (PINNs), emphasizing the role of the loss landscape in the training process. We examine difficulties in minimizing the PINN loss function, particularly due to ill-conditioning caused by differential operators in the residual term. We compare gradient-based optimizers Adam, L-BFGS, and their combination Adam+L-BFGS, showing the superiority of Adam+L-BFGS, and introduce a novel second-order optimizer, NysNewton-CG (NNCG), which significantly improves PINN performance. Theoretically, our work elucidates the connection between ill-conditioned differential operators and ill-conditioning in the PINN loss and shows the benefits of combining first- and second-order optimization methods. Our work presents valuable insights and more powerful optimization strategies for training PINNs, which could improve the utility of PINNs for solving difficult partial differential equations.