Information Complexity and Generalization Bounds

TL;DR

This paper unifies PAC-Bayesian and mutual information bounds on generalization error, introduces new bounds for complex scenarios, and discusses practical algorithms like Entropy-SGD and PAC-Bayes-SGD for neural networks.

Contribution

It provides a unifying framework using Tong Zhang's IEI to derive various generalization bounds, including new bounds for data-dependent priors and unbounded losses.

Findings

Several existing bounds are derived as corollaries of IEI.

New bounds are established for data-dependent priors and unbounded loss functions.

Practical variants of Gibbs algorithms are discussed for neural network training.

Abstract

We present a unifying picture of PAC-Bayesian and mutual information-based upper bounds on the generalization error of randomized learning algorithms. As we show, Tong Zhang's information exponential inequality (IEI) gives a general recipe for constructing bounds of both flavors. We show that several important results in the literature can be obtained as simple corollaries of the IEI under different assumptions on the loss function. Moreover, we obtain new bounds for data-dependent priors and unbounded loss functions. Optimizing the bounds gives rise to variants of the Gibbs algorithm, for which we discuss two practical examples for learning with neural networks, namely, Entropy- and PAC-Bayes- SGD. Further, we use an Occam's factor argument to show a PAC-Bayesian bound that incorporates second-order curvature information of the training loss.