New Empirical Process Tools and Their Applications to Robust Deep ReLU Networks and Phase Transitions for Nonparametric Regression

TL;DR

This paper develops new empirical process tools that enable robust analysis of deep ReLU networks and nonparametric regression under heavy-tailed noise, broadening theoretical understanding and practical guarantees.

Contribution

It introduces Dudley-type inequalities for empirical processes that handle heavy tails and complex classes, leading to robustness guarantees for deep learning models.

Findings

Robust sub-Gaussian bounds for deep ReLU networks under infinite-variance noise

First non-asymptotic bounds for deep quantile regression without moment assumptions

Estimation error bounds for nonparametric least-squares estimators with broad applicability

Abstract

This paper introduces new empirical process tools for analyzing a broad class of statistical learning models under heavy-tailed noise and complex function classes. Our primary contribution is the derivation of two Dudley-type maximal inequalities for expected empirical processes that remove restrictive assumptions such as light tails and uniform boundedness of the function class. These inequalities enlarge the scope of empirical process theory available for statistical learning and nonparametric estimation. Exploiting the new bounds, we establish robustness guarantees for deep ReLU network estimators in Huber and quantile regression. In particular, we prove a unified non-asymptotic sub-Gaussian concentration bound that remains valid even under infinite-variance noise and provide a comprehensive analysis of non-asymptotic robustness for deep Huber estimators across all noise regimes. For deep quantile regression, we provide the first non-asymptotic sub-Gaussian bounds without requiring moment assumptions. As an additional application, our framework yields estimation error bounds for nonparametric least-squares estimators that simultaneously accommodate infinite-variance noise, non-Donsker function classes, and approximation error. Moreover, unlike prior approaches based on specialized multiplier processes, our framework extends to broader empirical risk minimization problems, including the nonparametric generalized linear models and the ``set-structured'' models.