Approximation Power of Deep Neural Networks: an explanatory mathematical survey

TL;DR

This survey reviews the theoretical approximation capabilities of deep neural networks, analyzing their expressiveness, error bounds, and practical considerations, while highlighting recent advancements and introducing new findings.

Contribution

It provides a comprehensive overview of neural network approximation theory, including new insights into error complexity and expressiveness of deep architectures.

Findings

Deep ReLU networks can approximate continuous functions with specific error bounds.

Residual architectures enhance approximation power and training stability.

Recent theoretical results improve understanding of neural network expressiveness.

Abstract

This survey provides an in-depth and explanatory review of the approximation properties of deep neural networks, with a focus on feed-forward and residual architectures. The primary objective is to examine how effectively neural networks approximate target functions and to identify conditions under which they outperform traditional approximation methods. Key topics include the nonlinear, compositional structure of deep networks and the formalization of neural network tasks as optimization problems in regression and classification settings. The survey also addresses the training process, emphasizing the role of stochastic gradient descent and backpropagation in solving these optimization problems, and highlights practical considerations such as activation functions, overfitting, and regularization techniques. Additionally, the survey explores the density of neural networks in the space of continuous functions, comparing the approximation capabilities of deep ReLU networks with those of other approximation methods. It discusses recent theoretical advancements in understanding the expressiveness and limitations of these networks. A detailed error-complexity analysis is also presented, focusing on error rates and computational complexity for neural networks with ReLU and Fourier-type activation functions in the context of bounded target functions with minimal regularity assumptions. Alongside recent known results, the survey introduces new findings, offering a valuable resource for understanding the theoretical foundations of neural network approximation. Concluding remarks and further reading suggestions are provided.