Formal limitations of sample-wise information-theoretic generalization bounds

TL;DR

This paper investigates the limitations of sample-wise information-theoretic bounds on generalization, showing that such bounds cannot be extended to squared gaps or single-draw scenarios, but pairwise bounds are possible.

Contribution

It proves the non-existence of sample-wise bounds for squared generalization gaps and single draws, highlighting the importance of pairwise information bounds.

Findings

Sample-wise bounds do not exist for expected squared generalization gap.

Sample-wise bounds do not exist for single-draw generalization.

Pairwise information bounds are possible for PAC-Bayes and related scenarios.

Abstract

Some of the tightest information-theoretic generalization bounds depend on the average information between the learned hypothesis and a single training example. However, these sample-wise bounds were derived only for expected generalization gap. We show that even for expected squared generalization gap no such sample-wise information-theoretic bounds exist. The same is true for PAC-Bayes and single-draw bounds. Remarkably, PAC-Bayes, single-draw and expected squared generalization gap bounds that depend on information in pairs of examples exist.