Approximation properties of Residual Neural Networks for Kolmogorov PDEs

TL;DR

This paper demonstrates that residual neural networks can efficiently approximate solutions to Kolmogorov PDEs in high dimensions without the curse of dimensionality, leveraging their Euler-Maruyama structure.

Contribution

It extends approximation results from feedforward networks to ResNets for Kolmogorov PDEs, showing ResNets' advantages in construction and activation function flexibility.

Findings

ResNets can approximate PDE solutions with polynomial complexity in accuracy and dimension.

The Euler-Maruyama structure simplifies ResNet construction for PDE approximation.

No need for identity map representation in ResNets, broadening activation function choices.

Abstract

In recent years residual neural networks (ResNets) as introduced by [He, K., Zhang, X., Ren, S., and Sun, J., Proceedings of the IEEE conference on computer vision and pattern recognition (2016), 770-778] have become very popular in a large number of applications, including in image classification and segmentation. They provide a new perspective in training very deep neural networks without suffering the vanishing gradient problem. In this article we show that ResNets are able to approximate solutions of Kolmogorov partial differential equations (PDEs) with constant diffusion and possibly nonlinear drift coefficients without suffering the curse of dimensionality, which is to say the number of parameters of the approximating ResNets grows at most polynomially in the reciprocal of the approximation accuracy $\varepsilon > 0$ and the dimension of the considered PDE $d\in\mathbb{N}$. We adapt a proof in [Jentzen, A., Salimova, D., and Welti, T., Commun. Math. Sci. 19, 5 (2021), 1167-1205] - who showed a similar result for feedforward neural networks (FNNs) - to ResNets. In contrast to FNNs, the Euler-Maruyama approximation structure of ResNets simplifies the construction of the approximating ResNets substantially. Moreover, contrary to the above work, in our proof using ResNets does not require the existence of an FNN (or a ResNet) representing the identity map, which enlarges the set of applicable activation functions.