Novel Change of Measure Inequalities with Applications to PAC-Bayesian Bounds and Monte Carlo Estimation

TL;DR

This paper develops new change of measure inequalities for $f$-divergences and $\alpha$-divergences, with applications to PAC-Bayesian bounds and Monte Carlo estimation, enhancing theoretical tools for statistical learning.

Contribution

It introduces novel change of measure inequalities for $f$-divergences and $\alpha$-divergences, along with their applications to PAC-Bayesian bounds and Monte Carlo methods.

Findings

New change of measure inequalities for $f$-divergences.

Multiplicative change of measure inequality for $\alpha$-divergences.

Non-asymptotic bounds for Monte Carlo estimates.

Abstract

We introduce several novel change of measure inequalities for two families of divergences: $f$-divergences and $\alpha$-divergences. We show how the variational representation for $f$-divergences leads to novel change of measure inequalities. We also present a multiplicative change of measure inequality for $\alpha$-divergences and a generalized version of Hammersley-Chapman-Robbins inequality. Finally, we present several applications of our change of measure inequalities, including PAC-Bayesian bounds for various classes of losses and non-asymptotic intervals for Monte Carlo estimates.