Gibbs sampler and coordinate ascent variational inference: a set-theoretical review

TL;DR

This paper offers a set-theoretical perspective on Gibbs sampling and coordinate ascent variational inference, clarifying their mechanisms and providing pedagogical insights into their approximation of Bayesian posteriors.

Contribution

It introduces a set-theoretical framework to analyze and compare Gibbs sampler and variational inference methods, enhancing understanding of their fundamental properties.

Findings

Set-theoretical definitions of densities used in Bayesian inference

Clarification of Gibbs sampler and variational inference schemes

New pedagogical insights into approximation mechanisms

Abstract

One of the fundamental problems in Bayesian statistics is the approximation of the posterior distribution. Gibbs sampler and coordinate ascent variational inference are renownedly utilized approximation techniques that rely on stochastic and deterministic approximations. In this paper, we define fundamental sets of densities frequently used in Bayesian inference. We shall be concerned with the clarification of the two schemes from the set-theoretical point of view. This new way provides an alternative mechanism for analyzing the two schemes endowed with pedagogical insights.