# Variational Bayes on Manifolds

## TL;DR

This paper extends Variational Bayes to Riemannian manifolds, developing efficient algorithms that leverage geometric structures, with proven convergence and superior performance in numerical experiments.

## Contribution

It introduces manifold-based VB algorithms that incorporate geometric and information-theoretic structures, broadening VB applicability beyond Euclidean spaces.

## Key findings

- Algorithms are provably convergent with specific rates.
- Manifold VB algorithms are stable and less sensitive to initialization.
- Numerical experiments show improved performance over existing VB methods.

## Abstract

Variational Bayes (VB) has become a widely-used tool for Bayesian inference in statistics and machine learning. Nonetheless, the development of the existing VB algorithms is so far generally restricted to the case where the variational parameter space is Euclidean, which hinders the potential broad application of VB methods. This paper extends the scope of VB to the case where the variational parameter space is a Riemannian manifold. We develop an efficient manifold-based VB algorithm that exploits both the geometric structure of the constraint parameter space and the information geometry of the manifold of VB approximating probability distributions. Our algorithm is provably convergent and achieves a convergence rate of order $\mathcal O(1/\sqrt{T})$ and $\mathcal O(1/T^{2-2\epsilon})$ for a non-convex evidence lower bound function and a strongly retraction-convex evidence lower bound function, respectively. We develop in particular two manifold VB algorithms, Manifold Gaussian VB and Manifold Neural Net VB, and demonstrate through numerical experiments that the proposed algorithms are stable, less sensitive to initialization and compares favourably to existing VB methods.