Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks

TL;DR

This paper demonstrates that deeper neural networks can approximate certain natural functions more efficiently than shallower ones, with theoretical separation results and experimental validation showing depth's advantage over width.

Contribution

The paper provides new depth-based separation results for neural networks, showing depth can significantly improve approximation of natural functions over width.

Findings

Deeper networks better approximate indicator functions of geometric shapes.

Depth enhances learning of radial and smooth non-linear functions.

Experimental results confirm depth's advantage in learning indicator functions.

Abstract

We provide several new depth-based separation results for feed-forward neural networks, proving that various types of simple and natural functions can be better approximated using deeper networks than shallower ones, even if the shallower networks are much larger. This includes indicators of balls and ellipses; non-linear functions which are radial with respect to the $L_1$ norm; and smooth non-linear functions. We also show that these gaps can be observed experimentally: Increasing the depth indeed allows better learning than increasing width, when training neural networks to learn an indicator of a unit ball.