Approximation and Non-parametric Estimation of ResNet-type Convolutional Neural Networks

TL;DR

This paper demonstrates that ResNet-type CNNs can achieve minimax optimal approximation and estimation error rates in function classes like H"older and Barron, with constant width and sparsity, by replicating block-sparse FNNs.

Contribution

It introduces a novel approach showing ResNet CNNs can match the approximation power of block-sparse FNNs in more realistic settings.

Findings

ResNet CNNs attain minimax optimal error rates in H"older and Barron classes.

CNNs can be dense with constant width, channel, and filter sizes.

The theory translates approximation rates from block-sparse FNNs to CNNs.

Abstract

Convolutional neural networks (CNNs) have been shown to achieve optimal approximation and estimation error rates (in minimax sense) in several function classes. However, previous analyzed optimal CNNs are unrealistically wide and difficult to obtain via optimization due to sparse constraints in important function classes, including the H\"older class. We show a ResNet-type CNN can attain the minimax optimal error rates in these classes in more plausible situations -- it can be dense, and its width, channel size, and filter size are constant with respect to sample size. The key idea is that we can replicate the learning ability of Fully-connected neural networks (FNNs) by tailored CNNs, as long as the FNNs have \textit{block-sparse} structures. Our theory is general in a sense that we can automatically translate any approximation rate achieved by block-sparse FNNs into that by CNNs. As an application, we derive approximation and estimation error rates of the aformentioned type of CNNs for the Barron and H\"older classes with the same strategy.