Approximation Properties of Deep ReLU CNNs

TL;DR

This paper establishes the approximation capabilities of deep ReLU CNNs in 2D, linking their structure to universal approximation theorems and analyzing variants like ResNet and MgNet.

Contribution

It provides the first comprehensive analysis of approximation properties for deep ReLU CNNs, including their connection to one-hidden-layer ReLU networks and variants like ResNet and MgNet.

Findings

Deep ReLU CNNs can universally approximate functions in L^2 space.

The analysis connects CNN approximation properties to those of shallow ReLU networks.

Approximation properties are extended to ResNet, pre-act ResNet, and MgNet architectures.

Abstract

This paper focuses on establishing $L^2$ approximation properties for deep ReLU convolutional neural networks (CNNs) in two-dimensional space. The analysis is based on a decomposition theorem for convolutional kernels with a large spatial size and multi-channels. Given the decomposition result, the property of the ReLU activation function, and a specific structure for channels, a universal approximation theorem of deep ReLU CNNs with classic structure is obtained by showing its connection with one-hidden-layer ReLU neural networks (NNs). Furthermore, approximation properties are obtained for one version of neural networks with ResNet, pre-act ResNet, and MgNet architecture based on connections between these networks.