Chained Generalisation Bounds

TL;DR

This paper develops a theoretical framework for deriving tighter generalisation error bounds in supervised learning using chaining techniques, connecting regularity assumptions with gradient-based bounds and introducing novel information-theoretic bounds.

Contribution

It establishes a duality between regularity-based and chaining-based generalisation bounds, and introduces new bounds using Wasserstein distance and other metrics.

Findings

Chained bounds can be significantly tighter than standard bounds in concentrated hypothesis distributions.

Re-derivation of the chaining mutual information bound from existing literature.

Introduction of novel chained information-theoretic bounds based on Wasserstein distance.

Abstract

This work discusses how to derive upper bounds for the expected generalisation error of supervised learning algorithms by means of the chaining technique. By developing a general theoretical framework, we establish a duality between generalisation bounds based on the regularity of the loss function, and their chained counterparts, which can be obtained by lifting the regularity assumption from the loss onto its gradient. This allows us to re-derive the chaining mutual information bound from the literature, and to obtain novel chained information-theoretic generalisation bounds, based on the Wasserstein distance and other probability metrics. We show on some toy examples that the chained generalisation bound can be significantly tighter than its standard counterpart, particularly when the distribution of the hypotheses selected by the algorithm is very concentrated. Keywords: Generalisation bounds; Chaining; Information-theoretic bounds; Mutual information; Wasserstein distance; PAC-Bayes.