A deep network construction that adapts to intrinsic dimensionality beyond the domain

TL;DR

This paper demonstrates that deep networks can efficiently approximate functions by focusing on their intrinsic dimensionality through feature maps, reducing dependence on ambient space complexity.

Contribution

It introduces a framework for deep network approximation that adapts to intrinsic dimension via specific feature maps, relaxing traditional low-dimensional manifold assumptions.

Findings

Near optimal approximation rates depend on the complexity of the feature map.

Deep nets capture intrinsic dimension rather than ambient space complexity.

The approach relaxes the need for functions to be defined on low-dimensional manifolds.

Abstract

We study the approximation of two-layer compositions $f(x) = g(\phi(x))$ via deep networks with ReLU activation, where $\phi$ is a geometrically intuitive, dimensionality reducing feature map. We focus on two intuitive and practically relevant choices for $\phi$: the projection onto a low-dimensional embedded submanifold and a distance to a collection of low-dimensional sets. We achieve near optimal approximation rates, which depend only on the complexity of the dimensionality reducing map $\phi$ rather than the ambient dimension. Since $\phi$ encapsulates all nonlinear features that are material to the function $f$, this suggests that deep nets are faithful to an intrinsic dimension governed by $f$ rather than the complexity of the domain of $f$. In particular, the prevalent assumption of approximating functions on low-dimensional manifolds can be significantly relaxed using functions of type $f(x) = g(\phi(x))$ with $\phi$ representing an orthogonal projection onto the same manifold.