PAC$^m$-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime

TL;DR

This paper introduces PAC$^m$-Bayes, a new approach that reduces the gap caused by model misspecification in Bayesian inference, improving predictive performance with theoretical guarantees.

Contribution

It develops a multi-sample loss function that narrows the empirical risk gap in misspecified Bayesian models, with computational efficiency and PAC guarantees.

Findings

Improved predictive distribution performance in empirical tests

Theoretical PAC guarantees for the proposed loss

Effective trade-off between inferential and predictive risks

Abstract

The Bayesian posterior minimizes the "inferential risk" which itself bounds the "predictive risk". This bound is tight when the likelihood and prior are well-specified. However since misspecification induces a gap, the Bayesian posterior predictive distribution may have poor generalization performance. This work develops a multi-sample loss (PAC$^m$) which can close the gap by spanning a trade-off between the two risks. The loss is computationally favorable and offers PAC generalization guarantees. Empirical study demonstrates improvement to the predictive distribution.