Global Convergence of Deep Galerkin and PINNs Methods for Solving Partial Differential Equations

TL;DR

This paper proves the global convergence of deep learning methods, specifically DGM and PINNs, for solving high-dimensional PDEs by analyzing neural tangent kernels and the wide network limit.

Contribution

It establishes the first rigorous convergence proofs for DGM and PINNs, showing their solutions approach the true PDE solutions in the wide network limit.

Findings

Neural network solutions converge to PDE solutions as network width increases.

The residual of the PDE approaches zero with infinite training time.

Both DGM and PINNs exhibit similar convergence properties under the analyzed conditions.

Abstract

Numerically solving high-dimensional partial differential equations (PDEs) is a major challenge. Conventional methods, such as finite difference methods, are unable to solve high-dimensional PDEs due to the curse-of-dimensionality. A variety of deep learning methods have been recently developed to try and solve high-dimensional PDEs by approximating the solution using a neural network. In this paper, we prove global convergence for one of the commonly-used deep learning algorithms for solving PDEs, the Deep Galerkin Method (DGM). DGM trains a neural network approximator to solve the PDE using stochastic gradient descent. We prove that, as the number of hidden units in the single-layer network goes to infinity (i.e., in the ``wide network limit"), the trained neural network converges to the solution of an infinite-dimensional linear ordinary differential equation (ODE). The PDE residual of the limiting approximator converges to zero as the training time $\rightarrow \infty$. Under mild assumptions, this convergence also implies that the neural network approximator converges to the solution of the PDE. A closely related class of deep learning methods for PDEs is Physics Informed Neural Networks (PINNs). Using the same mathematical techniques, we can prove a similar global convergence result for the PINN neural network approximators. Both proofs require analyzing a kernel function in the limit ODE governing the evolution of the limit neural network approximator. A key technical challenge is that the kernel function, which is a composition of the PDE operator and the neural tangent kernel (NTK) operator, lacks a spectral gap, therefore requiring a careful analysis of its properties.