Change of measure through the Legendre transform

TL;DR

This paper introduces new PAC-Bayes generalisation bounds using change-of-measure inequalities based on f-divergences and the Legendre transform, extending applicability beyond traditional bounded exponential moment assumptions.

Contribution

It develops change-of-measure inequalities with f-divergences via the Legendre transform, broadening PAC-Bayes bounds under varied empirical risk conditions.

Findings

New PAC-Bayes bounds under relaxed assumptions on empirical risk

Inequalities based on f-divergences and Legendre transform are effective

Extends PAC-Bayesian guarantees to broader settings

Abstract

PAC-Bayes generalisation bounds are derived via change-of-measure inequalities that transfer concentration properties from a reference measure to all posterior measures. The specific choice of change of measure determines the assumptions required on the empirical risk; in particular, the classical Donsker--Varadhan theorem leads to bounds relying on bounded exponential moments. We study change-of-measure inequalities based on \(f\)-divergences, obtained by combining the Legendre transform of \(f\) with the Fenchel--Young inequality. Beyond their intrinsic interest in probability theory, we show how these inequalities are helpful in learning theory and yield PAC-Bayes bounds under tailored assumptions on the empirical risk, thereby extending the range of conditions under which PAC-Bayesian guarantees can be established.