Tighter PAC-Bayes Generalisation Bounds by Leveraging Example Difficulty

TL;DR

This paper presents a new PAC-Bayesian generalisation bound that incorporates example difficulty to achieve tighter, faster convergence rates, supported by empirical evaluations on real datasets.

Contribution

It introduces a modified excess risk leveraging data difficulty and a new bound for dependent signed losses, along with a novel technical result for interdependent random vectors.

Findings

Tighter PAC-Bayes bounds achieved on real datasets.

Effective leverage of example difficulty reduces variance in bounds.

New theoretical tools for dependent random vectors introduced.

Abstract

We introduce a modified version of the excess risk, which can be used to obtain tighter, fast-rate PAC-Bayesian generalisation bounds. This modified excess risk leverages information about the relative hardness of data examples to reduce the variance of its empirical counterpart, tightening the bound. We combine this with a new bound for $[-1, 1]$-valued (and potentially non-independent) signed losses, which is more favourable when they empirically have low variance around $0$. The primary new technical tool is a novel result for sequences of interdependent random vectors which may be of independent interest. We empirically evaluate these new bounds on a number of real-world datasets.