Instance-dependent uniform tail bounds for empirical processes

TL;DR

This paper introduces a novel uniform tail bound for empirical processes that depends on individual function deviations rather than the entire class, improving understanding of tail behaviors in statistical learning.

Contribution

It develops a new tail bound using a deflation step in generic chaining, incorporating individual deviations and complexity measures, extending previous bounds to broader moment conditions.

Findings

Provides a tail bound based on individual deviations and class complexity.

Extends bounds to cases without finite exponential moments.

Offers approximations using Orlicz norms under moment conditions.

Abstract

We formulate a uniform tail bound for empirical processes indexed by a class of functions, in terms of the individual deviations of the functions rather than the worst-case deviation in the considered class. The tail bound is established by introducing an initial ``deflation'' step to the standard generic chaining argument. The resulting tail bound is the sum of the complexity of the ``deflated function class'' in terms of a generalization of Talagrand's $\gamma$ functional, and the deviation of the function instance, both of which are formulated based on the natural seminorm induced by the corresponding Cram\'{e}r functions. Leveraging another less demanding natural seminorm, we also show similar bounds, though with implicit dependence on the sample size, in the more general case where finite exponential moments cannot be assumed. We also provide approximations of the tail bounds in terms of the more prevalent Orlicz norms or their ``incomplete'' versions under suitable moment conditions.