Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activations

TL;DR

This paper analyzes how deep neural networks with piecewise-polynomial activations can simultaneously approximate smooth functions and their derivatives, providing bounds on network size and weight magnitudes for accurate approximation.

Contribution

It establishes the depth, width, and sparsity requirements for neural networks to effectively approximate smooth functions and derivatives with bounded weights, enhancing understanding of their approximation capabilities.

Findings

Neural networks can approximate smooth functions and derivatives simultaneously.

Bounded weights help control generalization errors.

Explicit size and sparsity bounds are derived for approximation accuracy.

Abstract

This paper investigates the approximation properties of deep neural networks with piecewise-polynomial activation functions. We derive the required depth, width, and sparsity of a deep neural network to approximate any H\"{o}lder smooth function up to a given approximation error in H\"{o}lder norms in such a way that all weights of this neural network are bounded by $1$. The latter feature is essential to control generalization errors in many statistical and machine learning applications.