Super-fast Rates of Convergence for Neural Network Classifiers under the Hard Margin Condition

TL;DR

This paper proves that deep neural network classifiers can achieve near-optimal fast convergence rates under low-noise and hard-margin conditions, with rates depending on the smoothness of the regression function.

Contribution

It establishes new excess risk bounds for DNN classifiers under Tsybakov's low-noise and hard-margin conditions, including a novel risk decomposition technique.

Findings

Achieves excess risk bounds of order n^{-eta} with eta close to 1 under low-noise conditions.

Attains arbitrarily fast rates under the hard-margin condition for certain activation functions.

Provides minimax lower bounds showing the optimality of these rates for q ≥ 2.

Abstract

We study the classical binary classification problem for hypothesis spaces of Deep Neural Networks (DNNs) under Tsybakov's low-noise condition with exponent $q>0$, as well as its limit case $q=\infty$, which we refer to as the \emph{hard margin condition}. We demonstrate that, for a wide range of commonly used activation functions (including but not limited to ReLU, LeakyReLU, ELU, CELU, SELU, Softplus, GELU, SiLU, Swish, Mish, and Softmax), DNN solutions to the empirical risk minimization (ERM) problem with square loss surrogate and $\ell_p$ penalty on the weights $(0<p<\infty)$ can achieve excess risk bounds of order $\mathcal{O}\left(n^{-\alpha}\right)$ for $\alpha$ close to $1$ under the low-noise condition, and for arbitrarily large $\alpha>1$ under the hard-margin condition, provided that the Bayes regression function $\eta$ satisfies a \emph{distribution-adapted smoothness} condition relative to the marginal data distribution $\rho_{X}$. Furthermore, when the activation function is chosen as $\tanh$ or sigmoid, we show that the same rates follow from the standard assumption that $\eta\in \mathcal{C}^s$. Finally, we establish minimax lower bounds, showing that these rates cannot be improved upon whenever $q\ge2$. Our proof relies on a novel decomposition of the excess risk for general ERM-based classifiers which might be of independent interest.