Convergence of Implicit Gradient Descent for Training Two-Layer Physics-Informed Neural Networks

TL;DR

This paper analyzes the convergence of implicit gradient descent (IGD) in training over-parameterized two-layer physics-informed neural networks (PINNs), showing it converges globally at a linear rate and offers practical advantages over gradient descent.

Contribution

We provide the first convergence analysis of IGD for two-layer PINNs, demonstrating global linear convergence and milder network width requirements compared to prior methods.

Findings

IGD converges to a global optimum at a linear rate.

Learning rate for IGD can be chosen independently of sample size.

Empirical results confirm the theoretical convergence analysis.

Abstract

The optimization algorithms are crucial in training physics-informed neural networks (PINNs), as unsuitable methods may lead to poor solutions. Compared to the common gradient descent (GD) algorithm, implicit gradient descent (IGD) outperforms it in handling certain multi-scale problems. In this paper, we provide convergence analysis for the IGD in training over-parameterized two-layer PINNs. We first derive the training dynamics of IGD in training two-layer PINNs. Then, over-parameterization allows us to prove that the randomly initialized IGD converges to a globally optimal solution at a linear convergence rate. Moreover, due to the distinct training dynamics of IGD compared to GD, the learning rate can be selected independently of the sample size and the least eigenvalue of the Gram matrix. Additionally, the novel approach used in our convergence analysis imposes a milder requirement on the network width. Finally, empirical results validate our theoretical findings.