On the Universal Approximation Property of Deep Fully Convolutional Neural Networks

TL;DR

This paper proves that deep fully convolutional neural networks can universally approximate shift-invariant and equivariant functions, with specific architectural requirements, from a dynamical systems perspective.

Contribution

It establishes the universal approximation capabilities of deep residual and non-residual fully convolutional networks for symmetric functions, detailing necessary architectural conditions.

Findings

Residual networks achieve universality at constant channel width.

Non-residual networks require at least 2 channels per layer.

Fewer channels or smaller kernels are insufficient for universality.

Abstract

We study the approximation of shift-invariant or equivariant functions by deep fully convolutional networks from the dynamical systems perspective. We prove that deep residual fully convolutional networks and their continuous-layer counterpart can achieve universal approximation of these symmetric functions at constant channel width. Moreover, we show that the same can be achieved by non-residual variants with at least 2 channels in each layer and convolutional kernel size of at least 2. In addition, we show that these requirements are necessary, in the sense that networks with fewer channels or smaller kernels fail to be universal approximators.