Robust Bayesian Inference via Variational Approximations of Generalized Rho-Posteriors

TL;DR

This paper develops a robust Bayesian inference method called the $ ilde{ ho}$-posterior, which offers theoretical guarantees and practical robustness against model misspecification and data contamination, with efficient variational approximations.

Contribution

It introduces the $ ilde{ ho}$-posterior with PAC-Bayesian analysis and extends guarantees to variational approximations, enhancing robustness and computational efficiency.

Findings

Achieves robustness comparable to theoretical predictions.

Provides finite-sample oracle inequalities with explicit convergence rates.

Demonstrates effectiveness on real-world datasets and exponential families.

Abstract

We introduce the $\widetilde{\rho}$-posterior, a modified version of the $\rho$-posterior, obtained by replacing the supremum over competitor parameters with a softmax aggregation. This modification allows a PAC-Bayesian analysis of the $\widetilde{\rho}$-posterior. This yields finite-sample oracle inequalities with explicit convergence rates that inherit the key robustness properties of the original framework, in particular, graceful degradation under model misspecification and data contamination. Crucially, the PAC-Bayesian oracle inequalities extend to variational approximations of the $\widetilde{\rho}$-posterior, providing theoretical guarantees for tractable inference. Numerical experiments on exponential families, regression, and real-world datasets confirm that the resulting variational procedures achieve robustness competitive with theoretical predictions at computational cost comparable to standard variational Bayes.