Adaptive quadratures for nonlinear approximation of low-dimensional PDEs using smooth neural networks

TL;DR

This paper introduces adaptive quadrature methods for neural network integration in PDE solvers, improving accuracy and efficiency in physics-informed neural networks by balancing activation functions and domain decomposition.

Contribution

It proposes a novel adaptive quadrature technique using CPWL approximations of smooth activations, enhancing neural network PDE solutions with fewer points and better convergence.

Findings

Quadratic convergence rate for CPWL approximation of smooth activations.

Adaptive quadrature reduces integration points compared to Monte Carlo.

Improved convergence and robustness in solving Poisson equations.

Abstract

Physics-informed neural networks (PINNs) and their variants have recently emerged as alternatives to traditional partial differential equation (PDE) solvers, but little literature has focused on devising accurate numerical integration methods for neural networks (NNs), which is essential for getting accurate solutions. In this work, we propose adaptive quadratures for the accurate integration of neural networks and apply them to loss functions appearing in low-dimensional PDE discretisations. We show that at opposite ends of the spectrum, continuous piecewise linear (CPWL) activation functions enable one to bound the integration error, while smooth activations ease the convergence of the optimisation problem. We strike a balance by considering a CPWL approximation of a smooth activation function. The CPWL activation is used to obtain an adaptive decomposition of the domain into regions where the network is almost linear, and we derive an adaptive global quadrature from this mesh. The loss function is then obtained by evaluating the smooth network (together with other quantities, e.g., the forcing term) at the quadrature points. We propose a method to approximate a class of smooth activations by CPWL functions and show that it has a quadratic convergence rate. We then derive an upper bound for the overall integration error of our proposed adaptive quadrature. The benefits of our quadrature are evaluated on a strong and weak formulation of the Poisson equation in dimensions one and two. Our numerical experiments suggest that compared to Monte-Carlo integration, our adaptive quadrature makes the convergence of NNs quicker and more robust to parameter initialisation while needing significantly fewer integration points and keeping similar training times.