Neural networks: deep, shallow, or in between?

TL;DR

This paper provides theoretical estimates on the approximation error of neural networks, showing that only infinitely deep networks can surpass entropy number rates, with no advantage gained by increasing width at fixed depth.

Contribution

It establishes lower bounds for neural network approximation errors and clarifies the roles of depth and width in approximation capabilities.

Findings

Infinite depth can improve approximation rates beyond entropy numbers.

Increasing width alone at fixed depth does not improve approximation rates.

Theoretical bounds depend on network architecture and Lipschitz activation functions.

Abstract

We give estimates from below for the error of approximation of a compact subset from a Banach space by the outputs of feed-forward neural networks with width W, depth l and Lipschitz activation functions. We show that, modulo logarithmic factors, rates better that entropy numbers' rates are possibly attainable only for neural networks for which the depth l goes to infinity, and that there is no gain if we fix the depth and let the width W go to infinity.