Exponential Tail Local Rademacher Complexity Risk Bounds Without the Bernstein Condition

TL;DR

This paper introduces a new risk bound based on offset Rademacher complexities that applies to non-convex, improper estimators and does not require the Bernstein condition, broadening the scope of statistical learning theory.

Contribution

It develops an exponential-tail excess risk bound using offset Rademacher complexities under an estimator-dependent geometric condition, extending localization theory beyond classical constraints.

Findings

Bounds are at least as sharp as classical results.

Applicable to improper and non-convex estimators.

Operates without the Bernstein condition.

Abstract

The local Rademacher complexity framework is one of the most successful general-purpose toolboxes for establishing sharp excess risk bounds for statistical estimators based on the framework of empirical risk minimization. Applying this toolbox typically requires using the Bernstein condition, which often restricts applicability to convex and proper settings. Recent years have witnessed several examples of problems where optimal statistical performance is only achievable via non-convex and improper estimators originating from aggregation theory, including the fundamental problem of model selection. These examples are currently outside of the reach of the classical localization theory. In this work, we build upon the recent approach to localization via offset Rademacher complexities, for which a general high-probability theory has yet to be established. Our main result is an exponential-tail excess risk bound expressed in terms of the offset Rademacher complexity that yields results at least as sharp as those obtainable via the classical theory. However, our bound applies under an estimator-dependent geometric condition (the "offset condition") instead of the estimator-independent (but, in general, distribution-dependent) Bernstein condition on which the classical theory relies. Our results apply to improper prediction regimes not directly covered by the classical theory.