Tighter CMI-Based Generalization Bounds via Stochastic Projection and Quantization

TL;DR

This paper introduces tighter CMI-based generalization bounds using stochastic projection and quantization, demonstrating improved guarantees and insights into the necessity of memorization for generalization in learning algorithms.

Contribution

It presents novel CMI bounds leveraging stochastic projection and quantization, improving over existing bounds and analyzing memorization's role in generalization.

Findings

New CMI bounds are tighter than existing ones.

Bounds achieve (1/\u221a{n}) guarantees for certain problems.

Memorization is not necessary for good generalization.

Abstract

In this paper, we leverage stochastic projection and lossy compression to establish new conditional mutual information (CMI) bounds on the generalization error of statistical learning algorithms. It is shown that these bounds are generally tighter than the existing ones. In particular, we prove that for certain problem instances for which existing MI and CMI bounds were recently shown in Attias et al. [2024] and Livni [2023] to become vacuous or fail to describe the right generalization behavior, our bounds yield suitable generalization guarantees of the order of $\mathcal{O}(1/\sqrt{n})$, where $n$ is the size of the training dataset. Furthermore, we use our bounds to investigate the problem of data "memorization" raised in those works, and which asserts that there are learning problem instances for which any learning algorithm that has good prediction there exist distributions under which the algorithm must "memorize" a big fraction of the training dataset. We show that for every learning algorithm, there exists an auxiliary algorithm that does not memorize and which yields comparable generalization error for any data distribution. In part, this shows that memorization is not necessary for good generalization.