Limitations on approximation by deep and shallow neural networks

TL;DR

This paper establishes fundamental lower bounds on the approximation capabilities of deep and shallow neural networks for compact sets, using Carl's inequalities and Lipschitz widths, revealing inherent limitations.

Contribution

It introduces Carl's type inequalities for neural network approximation errors and connects these to Lipschitz widths, providing new theoretical bounds.

Findings

Lower bounds on approximation errors for neural networks

Connections between Lipschitz widths and neural network approximation

Limitations on the expressiveness of deep and shallow networks

Abstract

We prove Carl's type inequalities for the error of approximation of compact sets K by deep and shallow neural networks. This in turn gives lower bounds on how well we can approximate the functions in K when requiring the approximants to come from outputs of such networks. Our results are obtained as a byproduct of the study of the recently introduced Lipschitz widths.