Fast Rate Information-theoretic Bounds on Generalization Errors

TL;DR

This paper improves information-theoretic bounds on the generalization error by establishing conditions under which these bounds are asymptotically tight and introducing new bounds based on the $(\,eta, c)$-central condition.

Contribution

It demonstrates that fast convergence rates can be achieved under certain assumptions and introduces new bounds based on the $(\,eta, c)$-central condition that directly relate mutual information to convergence rates.

Findings

Bounds can be asymptotically tight with appropriate assumptions.

The $(\eta, c)$-central condition simplifies verification of bounds.

Numerical examples confirm the effectiveness of the new bounds.

Abstract

The generalization error of a learning algorithm refers to the discrepancy between the loss of a learning algorithm on training data and that on unseen testing data. Various information-theoretic bounds on the generalization error have been derived in the literature, where the mutual information between the training data and the hypothesis (the output of the learning algorithm) plays an important role. Focusing on the individual sample mutual information bound by Bu et al., which itself is a tightened version of the first bound on the topic by Russo et al. and Xu et al., this paper investigates the tightness of these bounds, in terms of the dependence of their convergence rates on the sample size $n$. It has been recognized that these bounds are in general not tight, readily verified for the exemplary quadratic Gaussian mean estimation problem, where the individual sample mutual information bound scales as $O(\sqrt{1/n})$ while the true generalization error scales as $O(1/n)$. The first contribution of this paper is to show that the same bound can in fact be asymptotically tight if an appropriate assumption is made. In particular, we show that the fast rate can be recovered when the assumption is made on the excess risk instead of the loss function, which was usually done in existing literature. A theoretical justification is given for this choice. The second contribution of the paper is a new set of generalization error bounds based on the $(\eta, c)$-central condition, a condition relatively easy to verify and has the property that the mutual information term directly determines the convergence rate of the bound. Several analytical and numerical examples are given to show the effectiveness of these bounds.