Automatic Differentiation is Essential in Training Neural Networks for Solving Differential Equations

TL;DR

This paper demonstrates that automatic differentiation (AD) significantly improves the training efficiency of neural networks in solving PDEs compared to finite difference methods, supported by experimental and theoretical analyses.

Contribution

The paper introduces the concept of truncated entropy to quantify training properties and shows AD's superiority over FD in neural network training for PDEs through comprehensive analysis.

Findings

AD outperforms FD in training neural networks for PDEs

Truncated entropy reliably measures residual loss and training speed

Theoretical and experimental results confirm AD's advantages

Abstract

Neural network-based approaches have recently shown significant promise in solving partial differential equations (PDEs) in science and engineering, especially in scenarios featuring complex domains or incorporation of empirical data. One advantage of the neural network methods for PDEs lies in its automatic differentiation (AD), which necessitates only the sample points themselves, unlike traditional finite difference (FD) approximations that require nearby local points to compute derivatives. In this paper, we quantitatively demonstrate the advantage of AD in training neural networks. The concept of truncated entropy is introduced to characterize the training property. Specifically, through comprehensive experimental and theoretical analyses conducted on random feature models and two-layer neural networks, we discover that the defined truncated entropy serves as a reliable metric for quantifying the residual loss of random feature models and the training speed of neural networks for both AD and FD methods. Our experimental and theoretical analyses demonstrate that, from a training perspective, AD outperforms FD in solving PDEs.