A Numerical Investigation of the Minimum Width of a Neural Network

TL;DR

This paper numerically investigates the minimum width of neural networks necessary for approximation, testing theoretical bounds through experiments to better understand practical network design constraints.

Contribution

It provides empirical validation of Hanin's 2017 lower bounds on neural network width for function approximation.

Findings

Numerical results support Hanin's theoretical bounds

Minimum width requirements vary with function complexity

Practical implications for neural network architecture design

Abstract

Neural network width and depth are fundamental aspects of network topology. Universal approximation theorems provide that with increasing width or depth, there exists a neural network that approximates a function arbitrarily well. These theorems assume requirements, such as infinite data, that must be discretized in practice. Through numerical experiments, we seek to test the lower bounds established by Hanin in 2017.