Generalization Guarantees via Algorithm-dependent Rademacher Complexity

TL;DR

This paper introduces a new data- and algorithm-dependent complexity measure based on empirical Rademacher complexity to derive tighter and more versatile generalization bounds for machine learning algorithms.

Contribution

It proposes a novel complexity measure that extends fractal dimension bounds, simplifies existing proofs, and recovers classical results without mutual information dependence.

Findings

Derived bounds based on finite fractal dimension.

Simplified proof of stochastic gradient descent generalization bounds.

Recovered classical VC and compression scheme results.

Abstract

Algorithm- and data-dependent generalization bounds are required to explain the generalization behavior of modern machine learning algorithms. In this context, there exists information theoretic generalization bounds that involve (various forms of) mutual information, as well as bounds based on hypothesis set stability. We propose a conceptually related, but technically distinct complexity measure to control generalization error, which is the empirical Rademacher complexity of an algorithm- and data-dependent hypothesis class. Combining standard properties of Rademacher complexity with the convenient structure of this class, we are able to (i) obtain novel bounds based on the finite fractal dimension, which (a) extend previous fractal dimension-type bounds from continuous to finite hypothesis classes, and (b) avoid a mutual information term that was required in prior work; (ii) we greatly simplify the proof of a recent dimension-independent generalization bound for stochastic gradient descent; and (iii) we easily recover results for VC classes and compression schemes, similar to approaches based on conditional mutual information.