Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks

TL;DR

This paper introduces an implicit stochastic gradient descent method to enhance the training stability of physics-informed neural networks, especially for complex multi-scale problems, with theoretical guarantees and empirical validation.

Contribution

The paper proposes a novel ISGD approach for PINNs, providing theoretical convergence analysis and demonstrating improved training stability over existing methods.

Findings

ISGD improves training stability for PINNs with multi-scale features

Theoretically, ISGD converges to a global optimum for certain neural network models

Empirically, ISGD outperforms SGD and Adam in challenging differential equation problems

Abstract

Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems, but they are still trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit stochastic gradient descent (ISGD) method to train PINNs for improving the stability of training process. We heuristically analyze how ISGD overcome stiffness in the gradient flow dynamics of PINNs, especially for problems with multi-scale solutions. We theoretically prove that for two-layer fully connected neural networks with large hidden nodes, randomly initialized ISGD converges to a globally optimal solution for the quadratic loss function. Empirical results demonstrate that ISGD works well in practice and compares favorably to other gradient-based optimization methods such as SGD and Adam, while can also effectively address the numerical stiffness in training dynamics via gradient descent.