Minimum Width for Deep, Narrow MLP: A Diffeomorphism Approach

TL;DR

This paper establishes a geometric framework to determine the minimum width of deep, narrow MLPs needed for universal approximation, using diffeomorphism approximation and the Whitney embedding theorem.

Contribution

It introduces a purely geometrical function to compute the minimum width for universality in deep MLPs based on input and output dimensions.

Findings

Upper bound for minimum width: max(2d_x+1, d_y) + alpha(sigma)

Lower bound for width when d_x = d_y = 2: at least 4

Deep MLPs can approximate C^2-diffeomorphisms with small additional width.

Abstract

Recently, there has been a growing focus on determining the minimum width requirements for achieving the universal approximation property in deep, narrow Multi-Layer Perceptrons (MLPs). Among these challenges, one particularly challenging task is approximating a continuous function under the uniform norm, as indicated by the significant disparity between its lower and upper bounds. To address this problem, we propose a framework that simplifies finding the minimum width for deep, narrow MLPs into determining a purely geometrical function denoted as $w(d_x, d_y)$. This function relies solely on the input and output dimensions, represented as $d_x$ and $d_y$, respectively. Two key steps support this framework. First, we demonstrate that deep, narrow MLPs, when provided with a small additional width, can approximate a $C^2$-diffeomorphism. Subsequently, using this result, we prove that $w(d_x, d_y)$ equates to the optimal minimum width required for deep, narrow MLPs to achieve universality. By employing the aforementioned framework and the Whitney embedding theorem, we provide an upper bound for the minimum width, given by $\operatorname{max}(2d_x+1, d_y) + \alpha(\sigma)$, where $0 \leq \alpha(\sigma) \leq 2$ represents a constant depending on the activation function. Furthermore, we provide a lower bound of $4$ for the minimum width in cases where the input and output dimensions are both equal to two.