Simple and Sharp Generalization Bounds via Lifting

TL;DR

This paper introduces an information-theoretic framework with a lifting technique that simplifies and sharpens generalization bounds, improving upon classical methods and applying to empirical risk minimization.

Contribution

It presents a novel lifting argument that yields sharper, simpler generalization bounds and offers new insights into empirical risk minimization over Sobolev ellipsoids.

Findings

Provides a simpler proof of the majorizing measure theorem.

Yields sharp convergence rates for empirical risk minimization.

Offers soft localized complexity bounds without slicing arguments.

Abstract

We develop an information-theoretic framework for bounding the supremum of stochastic processes, offering a simpler and sharper alternative to classical chaining and slicing arguments for generalization bounds. The key idea is a lifting argument that produces information-theoretic analogues of empirical process bounds, such as Dudley's entropy integral. Lifting introduces permutation symmetry, yielding sharp bounds when the classical Dudley integral is loose. This gives a simple proof of the majorizing measure theorem via the sharpness of Dudley's entropy integral for stationary processes, a result known well before the proof of the majorizing measure theorem. Furthermore, the information-theoretic formulation provides soft versions of classical localized complexity bounds in generalization theory, but is simpler and does not require the slicing argument. We apply this approach to empirical risk minimization over Sobolev ellipsoids, obtaining sharp convergence rates in settings where previous methods are suboptimal.