Variational predictive resampling

TL;DR

The paper introduces variational predictive resampling (VPR), a scalable method that enhances posterior sampling by combining variational inference's predictive capabilities with a resampling framework, improving uncertainty quantification.

Contribution

VPR leverages variational predictive distributions within a resampling framework to better approximate the true Bayesian posterior, addressing limitations of mean-field variational inference.

Findings

VPR converges to the exact Bayesian posterior in a Gaussian location model.

VPR improves posterior uncertainty quantification in linear and logistic regression.

VPR is computationally competitive with MCMC and often more efficient.

Abstract

Bayesian inference provides principled uncertainty quantification, but accurate posterior sampling with MCMC can be computationally prohibitive for modern applications. Variational inference (VI) offers a scalable alternative and often yields accurate predictive distributions, but cheap variational families such as mean-field (MF) can produce over-concentrated approximations that miss posterior dependence. We propose variational predictive resampling (VPR), a scalable posterior sampling method that exploits VI's predictive strength within a predictive-resampling framework to better approximate the Bayesian posterior. Given a prior-likelihood pair, VPR repeatedly imputes future observations from the current variational predictive, updates the variational approximation after each imputation, and records the parameter value implied by the completed sample. We establish conditions under which the law of the parameter returned by VPR is well defined and show that its finite-horizon approximation converges to this limit. In a tractable Gaussian location model, we show that VPR with MF variational predictives converges to the exact Bayesian posterior, whereas the optimal MF-VI approximation retains a non-vanishing asymptotic gap. Experiments on linear regression, logistic regression, and hierarchical linear mixed-effects models demonstrate that VPR substantially improves posterior uncertainty quantification and recovers posterior dependence missed by MF-VI, while remaining computationally competitive with, and often more efficient than, MCMC.