An analysis and solution of ill-conditioning in physics-informed neural networks

TL;DR

This paper analyzes the ill-conditioning problem in physics-informed neural networks (PINNs), links it to the Jacobian matrix's condition number, and proposes a method to improve stability and accuracy, enabling complex 3D flow simulations.

Contribution

It establishes a connection between PINN ill-conditioning and the Jacobian matrix, and introduces a general approach to mitigate this issue for complex PDE problems.

Findings

Reducing the Jacobian condition number improves PINN convergence and accuracy.

The proposed method enables successful 3D flow simulation around an M6 wing.

First successful PINN simulation of such complex industrial flow problems.

Abstract

Physics-informed neural networks (PINNs) have recently emerged as a novel and popular approach for solving forward and inverse problems involving partial differential equations (PDEs). However, achieving stable training and obtaining correct results remain a challenge in many cases, often attributed to the ill-conditioning of PINNs. Nonetheless, further analysis is still lacking, severely limiting the progress and applications of PINNs in complex engineering problems. Drawing inspiration from the ill-conditioning analysis in traditional numerical methods, we establish a connection between the ill-conditioning of PINNs and the ill-conditioning of the Jacobian matrix of the PDE system. Specifically, for any given PDE system, we construct its controlled system. This controlled system allows for adjustment of the condition number of the Jacobian matrix while retaining the same solution as the original system. Our numerical findings suggest that the ill-conditioning observed in PINNs predominantly stems from the Jacobian matrix. As the condition number of the Jacobian matrix decreases, PINNs exhibit faster convergence rates and higher accuracy. Building upon this understanding and the natural extension of controlled systems, we present a general approach to mitigate the ill-conditioning of PINNs, leading to successful simulations of the three-dimensional flow around the M6 wing at a Reynolds number of 5,000. To the best of our knowledge, this is the first time that PINNs have been successful in simulating such complex systems, offering a promising new technique for addressing industrial complexity problems. Our findings also offer valuable insights guiding the future development of PINNs.