PAC-Bayes Analysis Beyond the Usual Bounds

TL;DR

This paper extends PAC-Bayes analysis to more general settings, deriving new bounds for stochastic predictors that improve understanding of their generalization capabilities beyond traditional assumptions.

Contribution

It introduces a basic PAC-Bayes inequality for stochastic kernels and derives new bounds, including for data-dependent priors and unbounded losses, broadening the scope of PAC-Bayes analysis.

Findings

Basic PAC-Bayes inequality for stochastic kernels

New bounds for data-dependent priors

Bounds applicable to unbounded square loss

Abstract

We focus on a stochastic learning model where the learner observes a finite set of training examples and the output of the learning process is a data-dependent distribution over a space of hypotheses. The learned data-dependent distribution is then used to make randomized predictions, and the high-level theme addressed here is guaranteeing the quality of predictions on examples that were not seen during training, i.e. generalization. In this setting the unknown quantity of interest is the expected risk of the data-dependent randomized predictor, for which upper bounds can be derived via a PAC-Bayes analysis, leading to PAC-Bayes bounds. Specifically, we present a basic PAC-Bayes inequality for stochastic kernels, from which one may derive extensions of various known PAC-Bayes bounds as well as novel bounds. We clarify the role of the requirements of fixed 'data-free' priors, bounded losses, and i.i.d. data. We highlight that those requirements were used to upper-bound an exponential moment term, while the basic PAC-Bayes theorem remains valid without those restrictions. We present three bounds that illustrate the use of data-dependent priors, including one for the unbounded square loss.