Distributed Variational Bayesian Algorithms Over Sensor Networks

TL;DR

This paper introduces two novel distributed variational Bayesian algorithms for sensor networks, enabling efficient Bayesian inference with performance close to centralized methods, applicable to conjugate-exponential models.

Contribution

The paper proposes two new distributed VB algorithms using stochastic natural gradients and ADMM, applicable to a broad class of models, with demonstrated effectiveness on Gaussian mixture models.

Findings

Algorithms perform nearly as well as centralized VB.

Effective on synthetic and real datasets.

Applicable to general conjugate-exponential models.

Abstract

Distributed inference/estimation in Bayesian framework in the context of sensor networks has recently received much attention due to its broad applicability. The variational Bayesian (VB) algorithm is a technique for approximating intractable integrals arising in Bayesian inference. In this paper, we propose two novel distributed VB algorithms for general Bayesian inference problem, which can be applied to a very general class of conjugate-exponential models. In the first approach, the global natural parameters at each node are optimized using a stochastic natural gradient that utilizes the Riemannian geometry of the approximation space, followed by an information diffusion step for cooperation with the neighbors. In the second method, a constrained optimization formulation for distributed estimation is established in natural parameter space and solved by alternating direction method of multipliers (ADMM). An application of the distributed inference/estimation of a Bayesian Gaussian mixture model is then presented, to evaluate the effectiveness of the proposed algorithms. Simulations on both synthetic and real datasets demonstrate that the proposed algorithms have excellent performance, which are almost as good as the corresponding centralized VB algorithm relying on all data available in a fusion center.