Function approximation by deep neural networks with parameters $\{0,\pm \frac{1}{2}, \pm 1, 2\}$

TL;DR

This paper demonstrates that deep neural networks with a limited set of parameters can effectively approximate smooth functions, matching the performance of more flexible networks in nonparametric regression tasks.

Contribution

It introduces a construction of neural networks with fixed, discrete parameters that achieve approximation rates comparable to those with continuous parameters.

Findings

Networks with fixed parameters approximate smooth functions effectively.

The approximation error dependence on network size matches that of networks with continuous parameters.

Nonparametric regression using these networks attains optimal convergence rates.

Abstract

In this paper it is shown that $C_\beta$-smooth functions can be approximated by deep neural networks with ReLU activation function and with parameters $\{0,\pm \frac{1}{2}, \pm 1, 2\}$. The $l_0$ and $l_1$ parameter norms of considered networks are thus equivalent. The depth, width and the number of active parameters of the constructed networks have, up to a logarithmic factor, the same dependence on the approximation error as the networks with parameters in $[-1,1]$. In particular, this means that the nonparametric regression estimation with the constructed networks attains the same convergence rate as with sparse networks with parameters in $[-1,1]$.