Sobolev Acceleration and Statistical Optimality for Learning Elliptic Equations via Gradient Descent

TL;DR

This paper analyzes the statistical limits of gradient descent in solving inverse elliptic PDE problems using Sobolev norms, demonstrating optimality and implicit acceleration effects in training neural methods like DRM and PINN.

Contribution

It introduces a unified theoretical framework showing gradient descent achieves statistical optimality for elliptic PDEs using Sobolev norms, explaining implicit acceleration in training.

Findings

Gradient descent achieves statistical optimality in elliptic PDE inverse problems.

Using Sobolev norms accelerates training, increasing optimal epochs with data size.

Both DRM and PINN can reach statistical optimality under the proposed framework.

Abstract

In this paper, we study the statistical limits in terms of Sobolev norms of gradient descent for solving inverse problem from randomly sampled noisy observations using a general class of objective functions. Our class of objective functions includes Sobolev training for kernel regression, Deep Ritz Methods (DRM), and Physics Informed Neural Networks (PINN) for solving elliptic partial differential equations (PDEs) as special cases. We consider a potentially infinite-dimensional parameterization of our model using a suitable Reproducing Kernel Hilbert Space and a continuous parameterization of problem hardness through the definition of kernel integral operators. We prove that gradient descent over this objective function can also achieve statistical optimality and the optimal number of passes over the data increases with sample size. Based on our theory, we explain an implicit acceleration of using a Sobolev norm as the objective function for training, inferring that the optimal number of epochs of DRM becomes larger than the number of PINN when both the data size and the hardness of tasks increase, although both DRM and PINN can achieve statistical optimality.