Deep Neural Networks Learn Non-Smooth Functions Effectively

TL;DR

This paper provides a theoretical analysis demonstrating that deep neural networks with ReLU activation effectively learn non-smooth functions, achieving near-optimal convergence rates and offering practical guidelines for network architecture.

Contribution

It introduces a theoretical framework for understanding DNNs' ability to learn non-smooth functions, showing their near-optimal generalization performance compared to other models.

Findings

DNNs with ReLU can nearly optimally learn non-smooth functions.

Convergence rates of DNNs are almost optimal for non-smooth functions.

Guidelines for choosing network depth and size are derived.

Abstract

We theoretically discuss why deep neural networks (DNNs) performs better than other models in some cases by investigating statistical properties of DNNs for non-smooth functions. While DNNs have empirically shown higher performance than other standard methods, understanding its mechanism is still a challenging problem. From an aspect of the statistical theory, it is known many standard methods attain the optimal rate of generalization errors for smooth functions in large sample asymptotics, and thus it has not been straightforward to find theoretical advantages of DNNs. This paper fills this gap by considering learning of a certain class of non-smooth functions, which was not covered by the previous theory. We derive the generalization error of estimators by DNNs with a ReLU activation, and show that convergence rates of the generalization by DNNs are almost optimal to estimate the non-smooth functions, while some of the popular models do not attain the optimal rate. In addition, our theoretical result provides guidelines for selecting an appropriate number of layers and edges of DNNs. We provide numerical experiments to support the theoretical results.