Analytic natural gradient updates for Cholesky factor in Gaussian variational approximation

TL;DR

This paper derives analytic natural gradient updates for the Cholesky factor in Gaussian variational inference, addressing positive definiteness and sparsity, and proposes a stochastic normalized natural gradient method for complex models.

Contribution

It introduces a novel analytic natural gradient update for the Cholesky factor, enabling efficient and stable optimization in high-dimensional Gaussian variational approximations.

Findings

Analytic natural gradient updates improve convergence stability.

Method ensures positive definiteness of covariance matrices.

Effective in generalized linear mixed models and neural networks.

Abstract

Natural gradients can improve convergence in stochastic variational inference significantly but inverting the Fisher information matrix is daunting in high dimensions. Moreover, in Gaussian variational approximation, natural gradient updates of the precision matrix do not ensure positive definiteness. To tackle this issue, we derive analytic natural gradient updates of the Cholesky factor of the covariance or precision matrix, and consider sparsity constraints representing different posterior correlation structures. Stochastic normalized natural gradient ascent with momentum is proposed for implementation in generalized linear mixed models and deep neural networks.