Deep, Skinny Neural Networks are not Universal Approximators

TL;DR

This paper investigates how the architecture of neural networks, especially shallow and skinny ones, imposes topological constraints that limit their ability to approximate certain functions, challenging the notion of their universality.

Contribution

It introduces a novel topological perspective on neural network limitations, showing that certain architectures cannot approximate all functions regardless of depth.

Findings

Deep networks can overcome topological limitations of shallow ones.

Certain topological constraints are independent of network depth.

Skinny neural networks are not universal approximators.

Abstract

In order to choose a neural network architecture that will be effective for a particular modeling problem, one must understand the limitations imposed by each of the potential options. These limitations are typically described in terms of information theoretic bounds, or by comparing the relative complexity needed to approximate example functions between different architectures. In this paper, we examine the topological constraints that the architecture of a neural network imposes on the level sets of all the functions that it is able to approximate. This approach is novel for both the nature of the limitations and the fact that they are independent of network depth for a broad family of activation functions.