A matrix preconditioning framework for physics-informed neural networks based on adjoint method

TL;DR

This paper introduces a matrix preconditioning framework using the adjoint method to enhance the convergence of physics-informed neural networks, especially for challenging multi-scale and high Reynolds number PDE problems.

Contribution

It proposes a novel preconditioning approach combining automatic differentiation, matrix coloring, and the adjoint method to improve PINNs' convergence and robustness.

Findings

Successfully solves multi-scale PDE problems.

Improves convergence in high Reynolds number scenarios.

Reduces Jacobian condition number for better training stability.

Abstract

Physics-informed neural networks (PINNs) have recently emerged as a popular approach for solving forward and inverse problems involving partial differential equations (PDEs). Compared to fully connected neural networks, PINNs based on convolutional neural networks offer advantages in the hard enforcement of boundary conditions and in reducing the computational cost of partial derivatives. However, the latter still struggles with slow convergence and even failure in some scenarios. In this study, we propose a matrix preconditioning method to improve the convergence of the latter. Specifically, we combine automatic differentiation with matrix coloring to compute the Jacobian matrix of the PDE system, which is used to construct the preconditioner via incomplete LU factorization. We subsequently use the preconditioner to scale the PDE residual in the loss function in order to reduce the condition number of the Jacobian matrix, which is key to improving the convergence of PINNs. To overcome the incompatibility between automatic differentiation and triangular solves in the preconditioning, we also design a framework based on the adjoint method to compute the gradients of the loss function with respect to the network parameters. By numerical experiments, we validate that the proposed method successfully and efficiently solves the multi-scale problem and the high Reynolds number problem, in both of which PINNs fail to obtain satisfactory results.