PAC-Bayes-Chernoff bounds for unbounded losses

TL;DR

This paper introduces a new PAC-Bayes oracle bound for unbounded losses, extending Cramér-Chernoff bounds to the PAC-Bayesian setting, enabling tighter and more informative generalization bounds.

Contribution

It develops a novel PAC-Bayes bound for unbounded losses using Cramér-Chernoff techniques, allowing for richer assumptions and tighter bounds compared to previous results.

Findings

Recovers and generalizes existing PAC-Bayes bounds

Allows bounds based on parameter norms and log-Sobolev inequalities

Enables derivation of bounds beyond Gibbs posteriors

Abstract

We introduce a new PAC-Bayes oracle bound for unbounded losses that extends Cram\'er-Chernoff bounds to the PAC-Bayesian setting. The proof technique relies on controlling the tails of certain random variables involving the Cram\'er transform of the loss. Our approach naturally leverages properties of Cram\'er-Chernoff bounds, such as exact optimization of the free parameter in many PAC-Bayes bounds. We highlight several applications of the main theorem. Firstly, we show that our bound recovers and generalizes previous results. Additionally, our approach allows working with richer assumptions that result in more informative and potentially tighter bounds. In this direction, we provide a general bound under a new \textit{model-dependent} assumption from which we obtain bounds based on parameter norms and log-Sobolev inequalities. Notably, many of these bounds can be minimized to obtain distributions beyond the Gibbs posterior and provide novel theoretical coverage to existing regularization techniques.