Understanding the Difficulty of Solving Cauchy Problems with PINNs

TL;DR

This paper investigates why Physics-Informed Neural Networks (PINNs) struggle with accuracy in solving Cauchy problems, highlighting issues with residual-based loss functions and neural network approximation limits, especially near singularities.

Contribution

The paper identifies key theoretical limitations of PINNs in solving Cauchy problems, emphasizing the impact of loss function choice and neural network approximation gaps.

Findings

Residual-based loss functions do not capture solution dynamics.

Neural networks struggle with singularities in solutions.

Global minima may not exist, affecting accuracy.

Abstract

Physics-Informed Neural Networks (PINNs) have gained popularity in scientific computing in recent years. However, they often fail to achieve the same level of accuracy as classical methods in solving differential equations. In this paper, we identify two sources of this issue in the case of Cauchy problems: the use of $L^2$ residuals as objective functions and the approximation gap of neural networks. We show that minimizing the sum of $L^2$ residual and initial condition error is not sufficient to guarantee the true solution, as this loss function does not capture the underlying dynamics. Additionally, neural networks are not capable of capturing singularities in the solutions due to the non-compactness of their image sets. This, in turn, influences the existence of global minima and the regularity of the network. We demonstrate that when the global minimum does not exist, machine precision becomes the predominant source of achievable error in practice. We also present numerical experiments in support of our theoretical claims.