Theory of Deep Convolutional Neural Networks III: Approximating Radial Functions

TL;DR

This paper develops an approximation theory for deep convolutional neural networks with explicit rates, demonstrating their superiority over shallow networks in approximating radial functions, and analyzes their generalization in regression tasks.

Contribution

It provides the first rigorous proof of deep CNNs outperforming shallow networks for radial functions and offers a generalization analysis with an error bound showing a depth trade-off.

Findings

Deep CNNs can approximate radial functions more efficiently than shallow networks.

The approximation error decreases then increases with network depth, indicating an optimal depth.

Theoretical validation of the depth trade-off phenomenon in neural networks.

Abstract

We consider a family of deep neural networks consisting of two groups of convolutional layers, a downsampling operator, and a fully connected layer. The network structure depends on two structural parameters which determine the numbers of convolutional layers and the width of the fully connected layer. We establish an approximation theory with explicit approximation rates when the approximated function takes a composite form $f\circ Q$ with a feature polynomial $Q$ and a univariate function $f$. In particular, we prove that such a network can outperform fully connected shallow networks in approximating radial functions with $Q(x) =|x|^2$, when the dimension $d$ of data from $\mathbb{R}^d$ is large. This gives the first rigorous proof for the superiority of deep convolutional neural networks in approximating functions with special structures. Then we carry out generalization analysis for empirical risk minimization with such a deep network in a regression framework with the regression function of the form $f\circ Q$. Our network structure which does not use any composite information or the functions $Q$ and $f$ can automatically extract features and make use of the composite nature of the regression function via tuning the structural parameters. Our analysis provides an error bound which decreases with the network depth to a minimum and then increases, verifying theoretically a trade-off phenomenon observed for network depths in many practical applications.