Convergence Analysis of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks

TL;DR

This paper analyzes the convergence of natural gradient descent for over-parameterized physics-informed neural networks, showing it achieves faster, Gram matrix-independent convergence rates, especially with smooth activation functions.

Contribution

It introduces a convergence analysis for NGD in training two-layer PINNs, demonstrating improved learning rates and independence from the Gram matrix compared to standard gradient descent.

Findings

NGD achieves a convergence rate independent of the Gram matrix.

For smooth activation functions, NGD's convergence is quadratic.

Numerical experiments confirm theoretical convergence improvements.

Abstract

In the context of over-parameterization, there is a line of work demonstrating that randomly initialized (stochastic) gradient descent (GD) converges to a globally optimal solution at a linear convergence rate for the quadratic loss function. However, the learning rate of GD for training two-layer neural networks exhibits poor dependence on the sample size and the Gram matrix, leading to a slow training process. In this paper, we show that for training two-layer $\text{ReLU}^3$ Physics-Informed Neural Networks (PINNs), the learning rate can be improved from $\mathcal{O}(\lambda_0)$ to $\mathcal{O}(1/\|\bm{H}^{\infty}\|_2)$, implying that GD actually enjoys a faster convergence rate. Despite such improvements, the convergence rate is still tied to the least eigenvalue of the Gram matrix, leading to slow convergence. We then develop the positive definiteness of Gram matrices with general smooth activation functions and provide the convergence analysis of natural gradient descent (NGD) in training two-layer PINNs, demonstrating that the learning rate can be $\mathcal{O}(1)$ and at this rate, the convergence rate is independent of the Gram matrix. In particular, for smooth activation functions, the convergence rate of NGD is quadratic. Numerical experiments are conducted to verify our theoretical results.