Approximation and Estimation for High-Dimensional Deep Learning Networks

TL;DR

This paper provides a theoretical analysis of deep neural networks with ramp activation functions, showing their statistical risk can be bounded independently of the input dimension, explaining their surprising generalization ability.

Contribution

It introduces a risk bound for deep networks with $ ext{l}^1$ controls, demonstrating their effectiveness even in high-dimensional settings, and develops a sampling strategy for sparse covering.

Findings

Risk bound of $[(L^3 ext{log} d)/n]^{1/2}$ for deep ramp networks

Input dimension can be much larger than sample size with accurate estimation

Lower bounds indicate the risk bound is nearly optimal

Abstract

It has been experimentally observed in recent years that multi-layer artificial neural networks have a surprising ability to generalize, even when trained with far more parameters than observations. Is there a theoretical basis for this? The best available bounds on their metric entropy and associated complexity measures are essentially linear in the number of parameters, which is inadequate to explain this phenomenon. Here we examine the statistical risk (mean squared predictive error) of multi-layer networks with $\ell^1$-type controls on their parameters and with ramp activation functions (also called lower-rectified linear units). In this setting, the risk is shown to be upper bounded by $[(L^3 \log d)/n]^{1/2}$, where $d$ is the input dimension to each layer, $L$ is the number of layers, and $n$ is the sample size. In this way, the input dimension can be much larger than the sample size and the estimator can still be accurate, provided the target function has such $\ell^1$ controls and that the sample size is at least moderately large compared to $L^3\log d$. The heart of the analysis is the development of a sampling strategy that demonstrates the accuracy of a sparse covering of deep ramp networks. Lower bounds show that the identified risk is close to being optimal.