Neural Networks for Singular Perturbations -- Finite Regularity

TL;DR

This paper analyzes the expressivity and approximation rates of neural networks and finite-element methods for singularly perturbed elliptic boundary value problems, emphasizing robustness to perturbation parameters and low regularity data.

Contribution

It provides explicit, -robust Sobolev norm bounds for neural networks and finite-element methods solving singular perturbation problems, including novel deep neural network expressivity results.

Findings

-robust algebraic expression rate bounds for finite elements on exponential and Shishkin meshes

Neural network approximation rates are robust to the perturbation parameter 

Deep neural networks with ReLU and tanh activations can achieve twice the convergence rate of finite elements under certain conditions.

Abstract

We study finite-element and deep feedforward neural network (DNN for short) expressivity rate bounds for solution sets of a model linear, second order singularly perturbed, elliptic two-point boundary value problem, in Sobolev norms on a bounded interval $(-1,1)$, with explicit dependence on the singular perturbation parameter $\e\in (0,1]$. Emphasis is on low Sobolev regularity of the data, i.e., source term $f$ and reaction coefficient $b$. A proof of $\e$-explicit solution regularity based on exponentially weighted energy-norm bounds is developed, and \emph{$\e$-robust, algebraic expression rate bounds} in Sobolev norms for $\mathbb{P}_1$ Finite-Elements on exponential and Shishkin type meshes is proved. Expression rates for shallow (fixed depth) $\ReLU$-NNs are shown which are robust w.r. to $\e$ and explicit in terms of the NN size. Robust NN expression rate bounds are further studied for deep feedforward DNNs with ReLU and tanh-activations. As in \cite{OSX24_1085}, tanh- and sigmoid-activated sub-NNs allow to include exponential boundary layer functions exactly into the NN feature space, leading to reduced NN sizes. Recent bitstring encoding techniques for deep NNs with ReLU activations afford, still under low data regularity $f,b \in H^1(I)$ \emph{twice the (robust) convergence rate of $\mathbb{P}_1$ Finite-Elements} achievable with ``eXp'' or Shishkin meshes.