Minimum width for universal approximation using ReLU networks on compact domain

TL;DR

This paper precisely characterizes the minimum width of ReLU-like neural networks needed for universal approximation on compact domains, revealing differences from known results on unbounded domains and establishing bounds for general activations.

Contribution

It provides an exact value for the minimum width for $L^p$ approximation with ReLU-like networks on compact domains and establishes lower bounds for uniform approximation with general activations.

Findings

Minimum width for $L^p$ approximation is $ ext{max}igrace{d_x, d_y, 2igrace}$ for ReLU-like activations.

On compact domains, the minimum width is smaller than on unbounded domains for ReLU networks.

Lower bounds for uniform approximation show a dichotomy between $L^p$ and uniform approximation for certain dimensions.

Abstract

It has been shown that deep neural networks of a large enough width are universal approximators but they are not if the width is too small. There were several attempts to characterize the minimum width $w_{\min}$ enabling the universal approximation property; however, only a few of them found the exact values. In this work, we show that the minimum width for $L^p$ approximation of $L^p$ functions from $[0,1]^{d_x}$ to $\mathbb R^{d_y}$ is exactly $\max\{d_x,d_y,2\}$ if an activation function is ReLU-Like (e.g., ReLU, GELU, Softplus). Compared to the known result for ReLU networks, $w_{\min}=\max\{d_x+1,d_y\}$ when the domain is $\smash{\mathbb R^{d_x}}$, our result first shows that approximation on a compact domain requires smaller width than on $\smash{\mathbb R^{d_x}}$. We next prove a lower bound on $w_{\min}$ for uniform approximation using general activation functions including ReLU: $w_{\min}\ge d_y+1$ if $d_x<d_y\le2d_x$. Together with our first result, this shows a dichotomy between $L^p$ and uniform approximations for general activation functions and input/output dimensions.