Copula Variational Bayes inference via information geometry

TL;DR

This paper introduces copula variational Bayes (CVB), a generalized Bayesian inference method that relaxes independence constraints using information geometry, enabling potentially globally optimal approximations in complex Bayesian networks.

Contribution

The paper develops a novel copula-based VB framework that generalizes existing mean-field methods and introduces an augmented hierarchy for improved global optimality in Bayesian network inference.

Findings

CVB outperforms traditional VB, EM, and k-means in Gaussian mixture clustering accuracy.

All mean-field algorithms are special cases of CVB, unifying various inference methods.

Augmented CVB can achieve near-global optimal solutions in complex networks.

Abstract

Variational Bayes (VB), also known as independent mean-field approximation, has become a popular method for Bayesian network inference in recent years. Its application is vast, e.g. in neural network, compressed sensing, clustering, etc. to name just a few. In this paper, the independence constraint in VB will be relaxed to a conditional constraint class, called copula in statistics. Since a joint probability distribution always belongs to a copula class, the novel copula VB (CVB) approximation is a generalized form of VB. Via information geometry, we will see that CVB algorithm iteratively projects the original joint distribution to a copula constraint space until it reaches a local minimum Kullback-Leibler (KL) divergence. By this way, all mean-field approximations, e.g. iterative VB, Expectation-Maximization (EM), Iterated Conditional Mode (ICM) and k-means algorithms, are special cases of CVB approximation. For a generic Bayesian network, an augmented hierarchy form of CVB will also be designed. While mean-field algorithms can only return a locally optimal approximation for a correlated network, the augmented CVB network, which is an optimally weighted average of a mixture of simpler network structures, can potentially achieve the globally optimal approximation for the first time. Via simulations of Gaussian mixture clustering, the classification's accuracy of CVB will be shown to be far superior to that of state-of-the-art VB, EM and k-means algorithms.