Optimization Guarantees for Square-Root Natural-Gradient Variational Inference

TL;DR

This paper provides new theoretical convergence guarantees for natural-gradient variational inference using a square-root parameterization of Gaussian covariance, supported by experiments showing its practical advantages.

Contribution

It introduces a square-root parameterization that enables convergence guarantees for natural-gradient variational inference with Gaussian approximations.

Findings

Natural-gradient methods outperform Euclidean-based algorithms.

Square-root parameterization improves convergence guarantees.

Experiments confirm the effectiveness of the proposed approach.

Abstract

Variational inference with natural-gradient descent often shows fast convergence in practice, but its theoretical convergence guarantees have been challenging to establish. This is true even for the simplest cases that involve concave log-likelihoods and use a Gaussian approximation. We show that the challenge can be circumvented for such cases using a square-root parameterization for the Gaussian covariance. This approach establishes novel convergence guarantees for natural-gradient variational-Gaussian inference and its continuous-time gradient flow. Our experiments demonstrate the effectiveness of natural gradient methods and highlight their advantages over algorithms that use Euclidean or Wasserstein geometries.