Equivalence of approximation by convolutional neural networks and fully-connected networks

TL;DR

This paper establishes a theoretical connection between convolutional neural networks and fully-connected networks, showing that approximation bounds for one translate to the other within certain function classes, specifically for translation-equivariant functions.

Contribution

It provides a mathematical link between CNNs and fully-connected networks, enabling transfer of approximation bounds and analysis techniques between these architectures.

Findings

Approximation bounds for fully-connected networks apply to CNNs for translation-equivariant functions.

The results are specific to CNNs without pooling and with circular convolutions.

The connection allows for unified theoretical analysis of both network types.

Abstract

Convolutional neural networks are the most widely used type of neural networks in applications. In mathematical analysis, however, mostly fully-connected networks are studied. In this paper, we establish a connection between both network architectures. Using this connection, we show that all upper and lower bounds concerning approximation rates of {fully-connected} neural networks for functions $f \in \mathcal{C}$ -- for an arbitrary function class $\mathcal{C}$ -- translate to essentially the same bounds concerning approximation rates of convolutional neural networks for functions $f \in {\mathcal{C}^{equi}}$, with the class ${\mathcal{C}^{equi}}$ consisting of all translation equivariant functions whose first coordinate belongs to $\mathcal{C}$. All presented results consider exclusively the case of convolutional neural networks without any pooling operation and with circular convolutions, i.e., not based on zero-padding.