ResNet with one-neuron hidden layers is a Universal Approximator

TL;DR

This paper proves that very deep ResNets with only one neuron per hidden layer can universally approximate any Lebesgue integrable function in multiple dimensions, highlighting their enhanced representational power over traditional narrow networks.

Contribution

It demonstrates that deep ResNets with one-neuron layers are universal approximators, contrasting with fully connected networks of the same width, thus revealing increased expressiveness.

Findings

ResNets with one-neuron layers can approximate any Lebesgue integrable function.

ResNet architecture enhances the expressiveness of narrow deep networks.

Contrast with fully connected networks which are not universal at the same width.

Abstract

We demonstrate that a very deep ResNet with stacked modules with one neuron per hidden layer and ReLU activation functions can uniformly approximate any Lebesgue integrable function in $d$ dimensions, i.e. $\ell_1(\mathbb{R}^d)$. Because of the identity mapping inherent to ResNets, our network has alternating layers of dimension one and $d$. This stands in sharp contrast to fully connected networks, which are not universal approximators if their width is the input dimension $d$ [Lu et al, 2017; Hanin and Sellke, 2017]. Hence, our result implies an increase in representational power for narrow deep networks by the ResNet architecture.