Deep Neural Networks with General Activations: Super-Convergence in Sobolev Norms

TL;DR

This paper proves that deep neural networks with general activation functions can achieve super-convergence in Sobolev norms, surpassing classical numerical methods in approximating solutions to PDEs.

Contribution

It provides a unified theoretical framework showing deep networks can approximate PDE solutions with superior accuracy in Sobolev spaces, extending existing approximation theory.

Findings

Deep networks achieve super-convergence in Sobolev norms.

Neural networks outperform classical numerical methods in PDE approximation.

Theoretical foundation for neural network-based PDE solvers.

Abstract

This paper establishes a comprehensive approximation result for deep fully-connected neural networks with commonly-used and general activation functions in Sobolev spaces $W^{n,\infty}$, with errors measured in the $W^{m,p}$-norm for $m < n$ and $1\le p \le \infty$. The derived rates surpass those of classical numerical approximation techniques, such as finite element and spectral methods, exhibiting a phenomenon we refer to as \emph{super-convergence}. Our analysis shows that deep networks with general activations can approximate weak solutions of partial differential equations (PDEs) with superior accuracy compared to traditional numerical methods at the approximation level. Furthermore, this work closes a significant gap in the error-estimation theory for neural-network-based approaches to PDEs, offering a unified theoretical foundation for their use in scientific computing.