Robust Coordinate Ascent Variational Inference with Markov chain Monte Carlo simulations

TL;DR

This paper introduces Hybrid CAVI, a novel method combining Coordinate Ascent Variational Inference with MCMC to enhance convergence and initialization robustness, especially for non-conjugate posteriors.

Contribution

The paper proposes Hybrid CAVI, integrating MCMC-based initialization with CAVI, addressing limitations of both methods and extending applicability to non-conjugate models.

Findings

Hybrid CAVI improves convergence over standard CAVI.

Initialization with MCMC reduces sensitivity to starting points.

Effective for non-conjugate posterior distributions.

Abstract

Variational Inference (VI) is a method that approximates a difficult-to-compute posterior density using better behaved distributional families. VI is an alternative to the already well-studied Markov chain Monte Carlo (MCMC) method of approximating densities. With each algorithm, there are of course benefits and drawbacks; does there exist a combination of the two that mitigates the flaws of both? We propose a method to combine Coordinate Ascent Variational Inference (CAVI) with MCMC. This new methodology, termed Hybrid CAVI, seeks to improve the sensitivity to initialization and convergence problems of CAVI by proposing an initialization using method of moments estimates obtained from a short MCMC burn-in period. Unlike CAVI, Hybrid CAVI proves to also be effective when the posterior is not from a conditionally conjugate exponential family.