On Excess Risk Convergence Rates of Neural Network Classifiers

TL;DR

This paper analyzes the convergence rates of neural network classifiers' excess risk in a realistic setting, showing they achieve near-optimal, dimension-free rates under certain conditions, with implications for practical neural network training.

Contribution

It provides the first analysis of neural network excess risk convergence rates in a general, practical setting including Barron functions and surrogate loss minimization.

Findings

Neural networks can achieve near-minimax optimal excess risk rates.

Dimension-free convergence rates are possible in the considered regime.

The established rates are nearly optimal up to a logarithmic factor.

Abstract

The recent success of neural networks in pattern recognition and classification problems suggests that neural networks possess qualities distinct from other more classical classifiers such as SVMs or boosting classifiers. This paper studies the performance of plug-in classifiers based on neural networks in a binary classification setting as measured by their excess risks. Compared to the typical settings imposed in the literature, we consider a more general scenario that resembles actual practice in two respects: first, the function class to be approximated includes the Barron functions as a proper subset, and second, the neural network classifier constructed is the minimizer of a surrogate loss instead of the $0$-$1$ loss so that gradient descent-based numerical optimizations can be easily applied. While the class of functions we consider is quite large that optimal rates cannot be faster than $n^{-\frac{1}{3}}$, it is a regime in which dimension-free rates are possible and approximation power of neural networks can be taken advantage of. In particular, we analyze the estimation and approximation properties of neural networks to obtain a dimension-free, uniform rate of convergence for the excess risk. Finally, we show that the rate obtained is in fact minimax optimal up to a logarithmic factor, and the minimax lower bound shows the effect of the margin assumption in this regime.