Variational MCMC
Nando de Freitas, Pedro Hojen-Sorensen, Michael I. Jordan, Stuart, Russell

TL;DR
This paper introduces a hybrid variational-MCMC algorithm that combines the strengths of both methods to improve convergence speed and estimate higher moments more accurately in Bayesian inference tasks.
Contribution
It presents a novel MCMC algorithm integrating variational approximations with multiple kernels, enhancing efficiency and accuracy over traditional methods.
Findings
Outperforms pure variational methods in estimating moments.
Speeds up convergence compared to standard MCMC.
Effective in Bayesian logistic network parameter estimation.
Abstract
We propose a new class of learning algorithms that combines variational approximation and Markov chain Monte Carlo (MCMC) simulation. Naive algorithms that use the variational approximation as proposal distribution can perform poorly because this approximation tends to underestimate the true variance and other features of the data. We solve this problem by introducing more sophisticated MCMC algorithms. One of these algorithms is a mixture of two MCMC kernels: a random walk Metropolis kernel and a blockMetropolis-Hastings (MH) kernel with a variational approximation as proposaldistribution. The MH kernel allows one to locate regions of high probability efficiently. The Metropolis kernel allows us to explore the vicinity of these regions. This algorithm outperforms variationalapproximations because it yields slightly better estimates of the mean and considerably better estimates of…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsBayesian Methods and Mixture Models · Domain Adaptation and Few-Shot Learning · Machine Learning and Algorithms
