Stability of Sequential and Parallel Coordinate Ascent Variational Inference

TL;DR

This paper compares the convergence properties of sequential and parallel coordinate ascent variational inference algorithms, revealing that the sequential version, despite being slower, has more relaxed convergence guarantees in high-dimensional linear regression.

Contribution

It provides a theoretical analysis of the differing convergence behaviors of sequential and parallel variational inference algorithms in complex models.

Findings

Sequential algorithm has more relaxed convergence conditions.

Parallel algorithm is faster but less guaranteed to converge.

Differences are significant in high-dimensional linear regression.

Abstract

We highlight a striking difference in behavior between two widely used variants of coordinate ascent variational inference: the sequential and parallel algorithms. While such differences were known in the numerical analysis literature in simpler settings, they remain largely unexplored in the optimization-focused literature on variational inference in more complex models. Focusing on the moderately high-dimensional linear regression problem, we show that the sequential algorithm, although typically slower, enjoys convergence guarantees under more relaxed conditions than the parallel variant, which is often employed to facilitate block-wise updates and improve computational efficiency.