Approximating Continuous Functions by ReLU Nets of Minimal Width

TL;DR

This paper determines the minimal width of ReLU neural networks needed to approximate any continuous function, showing that width must be at least the input dimension plus one, and provides explicit construction and depth estimates.

Contribution

It establishes that the minimal width for universal approximation by ReLU nets is exactly input dimension plus one, and offers constructive methods with quantitative depth bounds.

Findings

ReLU nets with width d_in+1 can approximate any continuous function.

Width less than or equal to d_in limits the expressive power.

Explicit construction with depth estimates for approximation.

Abstract

This article concerns the expressive power of depth in deep feed-forward neural nets with ReLU activations. Specifically, we answer the following question: for a fixed $d_{in}\geq 1,$ what is the minimal width $w$ so that neural nets with ReLU activations, input dimension $d_{in}$, hidden layer widths at most $w,$ and arbitrary depth can approximate any continuous, real-valued function of $d_{in}$ variables arbitrarily well? It turns out that this minimal width is exactly equal to $d_{in}+1.$ That is, if all the hidden layer widths are bounded by $d_{in}$, then even in the infinite depth limit, ReLU nets can only express a very limited class of functions, and, on the other hand, any continuous function on the $d_{in}$-dimensional unit cube can be approximated to arbitrary precision by ReLU nets in which all hidden layers have width exactly $d_{in}+1.$ Our construction in fact shows that any continuous function $f:[0,1]^{d_{in}}\to\mathbb R^{d_{out}}$ can be approximated by a net of width $d_{in}+d_{out}$. We obtain quantitative depth estimates for such an approximation in terms of the modulus of continuity of $f$.