Provable approximation properties for deep neural networks

TL;DR

This paper demonstrates that deep neural networks, specifically sparsely-connected depth-4 architectures, can effectively approximate functions on manifolds with error bounds depending on geometric and function complexity, leveraging wavelet representations.

Contribution

The authors construct a depth-4 neural network that approximates functions on manifolds with error bounds depending on manifold geometry and wavelet complexity, showing efficient approximation with weak ambient dimension dependence.

Findings

Network size depends on manifold curvature and wavelet complexity.

Neural network computes wavelet functions using ReLU units.

Approximation error bounds are established for functions on manifolds.

Abstract

We discuss approximation of functions using deep neural nets. Given a function $f$ on a $d$-dimensional manifold $\Gamma \subset \mathbb{R}^m$, we construct a sparsely-connected depth-4 neural network and bound its error in approximating $f$. The size of the network depends on dimension and curvature of the manifold $\Gamma$, the complexity of $f$, in terms of its wavelet description, and only weakly on the ambient dimension $m$. Essentially, our network computes wavelet functions, which are computed from Rectified Linear Units (ReLU)