From Generalisation Error to Transportation-cost Inequalities and Back

TL;DR

This paper establishes a theoretical connection between generalisation error bounds and transportation-cost inequalities, generalising existing results and introducing new bounds involving arbitrary divergence measures.

Contribution

It generalises the link between transportation-cost inequalities and generalisation bounds, extending beyond Kullback-Leibler divergence and sub-Gaussian measures, and shows equivalence between functionals and measure-based inequalities.

Findings

Established equivalence between two families of inequalities.

Recovered standard bounds involving mutual information.

Introduced new bounds with arbitrary divergence measures.

Abstract

In this work, we connect the problem of bounding the expected generalisation error with transportation-cost inequalities. Exposing the underlying pattern behind both approaches we are able to generalise them and go beyond Kullback-Leibler Divergences/Mutual Information and sub-Gaussian measures. In particular, we are able to provide a result showing the equivalence between two families of inequalities: one involving functionals and one involving measures. This result generalises the one proposed by Bobkov and G\"otze that connects transportation-cost inequalities with concentration of measure. Moreover, it allows us to recover all standard generalisation error bounds involving mutual information and to introduce new, more general bounds, that involve arbitrary divergence measures.