The effect of the choice of neural network depth and breadth on the size of its hypothesis space

TL;DR

This paper analyzes how the architecture of neural networks, specifically depth and breadth, influences the size of the set of functions they can represent, revealing an inverse relationship with factorials of neuron counts.

Contribution

It provides a mathematical characterization of the hypothesis space size based on network architecture, highlighting the impact of layer widths on expressiveness.

Findings

Hypothesis space size decreases with increasing neurons per layer.

Number of unique functions is inversely proportional to product of factorials of layer sizes.

Provides a formula linking network architecture to its expressive capacity.

Abstract

We show that the number of unique function mappings in a neural network hypothesis space is inversely proportional to $\prod_lU_l!$, where $U_{l}$ is the number of neurons in the hidden layer $l$.