Approximation smooth and sparse functions by deep neural networks without saturation

TL;DR

This paper demonstrates that deep neural networks with three hidden layers can optimally approximate smooth and sparse functions, and adding one more layer can overcome saturation issues, providing theoretical insights into deep learning.

Contribution

It constructs deep neural networks with three hidden layers that achieve optimal approximation rates for smooth and sparse functions, and shows that deepening the network can avoid saturation problems.

Findings

Deep nets can reach optimal approximation rates for smooth functions.

Adding one more hidden layer can prevent saturation in approximation.

Theoretical explanation for the advantages of deep neural networks.

Abstract

Constructing neural networks for function approximation is a classical and longstanding topic in approximation theory. In this paper, we aim at constructing deep neural networks (deep nets for short) with three hidden layers to approximate smooth and sparse functions. In particular, we prove that the constructed deep nets can reach the optimal approximation rate in approximating both smooth and sparse functions with controllable magnitude of free parameters. Since the saturation that describes the bottleneck of approximate is an insurmountable problem of constructive neural networks, we also prove that deepening the neural network with only one more hidden layer can avoid the saturation. The obtained results underlie advantages of deep nets and provide theoretical explanations for deep learning.