Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations

TL;DR

This paper investigates the expressive power of deep ReLU neural networks with bounded width, establishing the minimal width needed for universal approximation of continuous functions and providing depth estimates for approximation rates.

Contribution

It proves that ReLU nets with width d+1 can approximate any convex function, and with width d+3 can approximate any continuous function on [0,1]^d.

Findings

ReLU nets with width d+1 can approximate any convex function.

ReLU nets with width d+3 can approximate any continuous function.

Quantitative depth estimates are provided for function approximation.

Abstract

This article concerns the expressive power of depth in neural nets with ReLU activations and bounded width. We are particularly interested in the following questions: what is the minimal width $w_{\text{min}}(d)$ so that ReLU nets of width $w_{\text{min}}(d)$ (and arbitrary depth) can approximate any continuous function on the unit cube $[0,1]^d$ aribitrarily well? For ReLU nets near this minimal width, what can one say about the depth necessary to approximate a given function? Our approach to this paper is based on the observation that, due to the convexity of the ReLU activation, ReLU nets are particularly well-suited for representing convex functions. In particular, we prove that ReLU nets with width $d+1$ can approximate any continuous convex function of $d$ variables arbitrarily well. These results then give quantitative depth estimates for the rate of approximation of any continuous scalar function on the $d$-dimensional cube $[0,1]^d$ by ReLU nets with width $d+3.$