Universal Approximation Property of Fully Convolutional Neural Networks with Zero Padding

TL;DR

This paper proves that fully convolutional neural networks with zero padding can universally approximate any continuous tensor-to-tensor function with the same input and output shape, detailing the minimal depth needed and confirming optimality.

Contribution

It establishes the universal approximation property of CNNs with zero padding for tensor-to-tensor functions, including depth bounds and applicability across dimensions and activation functions.

Findings

CNNs with zero padding can approximate any continuous tensor-to-tensor function.

Deep, narrow CNNs also possess the universal approximation property.

The minimal network depth for approximation is identified and proven optimal.

Abstract

The Convolutional Neural Network (CNN) is one of the most prominent neural network architectures in deep learning. Despite its widespread adoption, our understanding of its universal approximation properties has been limited due to its intricate nature. CNNs inherently function as tensor-to-tensor mappings, preserving the spatial structure of input data. However, limited research has explored the universal approximation properties of fully convolutional neural networks as arbitrary continuous tensor-to-tensor functions. In this study, we demonstrate that CNNs, when utilizing zero padding, can approximate arbitrary continuous functions in cases where both the input and output values exhibit the same spatial shape. Additionally, we determine the minimum depth of the neural network required for approximation and substantiate its optimality. We also verify that deep, narrow CNNs possess the UAP as tensor-to-tensor functions. The results encompass a wide range of activation functions, and our research covers CNNs of all dimensions.