PAC-Bayes unleashed: generalisation bounds with unbounded losses

TL;DR

This paper develops new PAC-Bayesian generalisation bounds applicable to unbounded loss functions by introducing the HYPE notion, broadening the framework's applicability beyond bounded loss scenarios, with practical considerations included.

Contribution

It introduces the HYPE concept to handle unbounded losses and derives novel PAC-Bayesian bounds, extending the framework's scope to more general learning problems.

Findings

Derived PAC-Bayesian bounds for unbounded loss functions

Introduced the HYPE notion for predictor-dependent loss ranges

Applied bounds to linear regression with practical discussion

Abstract

We present new PAC-Bayesian generalisation bounds for learning problems with unbounded loss functions. This extends the relevance and applicability of the PAC-Bayes learning framework, where most of the existing literature focuses on supervised learning problems with a bounded loss function (typically assumed to take values in the interval [0;1]). In order to relax this assumption, we propose a new notion called HYPE (standing for \emph{HYPothesis-dependent rangE}), which effectively allows the range of the loss to depend on each predictor. Based on this new notion we derive a novel PAC-Bayesian generalisation bound for unbounded loss functions, and we instantiate it on a linear regression problem. To make our theory usable by the largest audience possible, we include discussions on actual computation, practicality and limitations of our assumptions.