Approximating smooth functions by deep neural networks with sigmoid activation function

TL;DR

This paper investigates the approximation capabilities of deep neural networks with sigmoid activation, demonstrating that networks with fixed depth and width proportional to a power of the input dimension can approximate smooth functions at a specific rate, generalizing previous results.

Contribution

The paper extends approximation rate results to more general DNN architectures defined by width and depth, not just sparse networks, providing a quantitative understanding of their approximation power.

Findings

DNNs with fixed depth and width ~ M^d achieve approximation rate M^{-2p}

Approximation rate in terms of total weights W_0 is W_0^{-p/d}

Results help identify network topologies that guarantee target accuracy

Abstract

We study the power of deep neural networks (DNNs) with sigmoid activation function. Recently, it was shown that DNNs approximate any $d$-dimensional, smooth function on a compact set with a rate of order $W^{-p/d}$, where $W$ is the number of nonzero weights in the network and $p$ is the smoothness of the function. Unfortunately, these rates only hold for a special class of sparsely connected DNNs. We ask ourselves if we can show the same approximation rate for a simpler and more general class, i.e., DNNs which are only defined by its width and depth. In this article we show that DNNs with fixed depth and a width of order $M^d$ achieve an approximation rate of $M^{-2p}$. As a conclusion we quantitatively characterize the approximation power of DNNs in terms of the overall weights $W_0$ in the network and show an approximation rate of $W_0^{-p/d}$. This more general result finally helps us to understand which network topology guarantees a special target accuracy.