New Approximation Results and Optimal Estimation for Fully Connected Deep Neural Networks

TL;DR

This paper improves the theoretical approximation bounds for fully connected deep neural networks, achieving near-optimal convergence rates and demonstrating their effectiveness in high-dimensional settings with compositional or manifold-structured functions.

Contribution

It derives sharper approximation bounds for fully connected networks, leading to improved convergence rates and insights into their ability to handle high-dimensional data.

Findings

Achieves near-optimal convergence rates for fully connected networks.

Shows neural networks can mitigate the curse of dimensionality.

Provides theoretical bounds for functions on manifolds.

Abstract

\citet{farrell2021deep} establish non-asymptotic high-probability bounds for general deep feedforward neural network (with rectified linear unit activation function) estimators, with \citet[Theorem 1]{farrell2021deep} achieving a suboptimal convergence rate for fully connected feedforward networks. The authors suggest that improved approximation of fully connected networks could yield sharper versions of \citet[Theorem 1]{farrell2021deep} without altering the theoretical framework. By deriving approximation bounds specifically for a narrower fully connected deep neural network, this note demonstrates that \citet[Theorem 1]{farrell2021deep} can be improved to achieve an optimal rate (up to a logarithmic factor). Furthermore, this note briefly shows that deep neural network estimators can mitigate the curse of dimensionality for functions with compositional structure and functions defined on manifolds.