The Variational Predictive Natural Gradient

TL;DR

This paper introduces the Variational Predictive Natural Gradient (VPNG), a new natural gradient method for variational inference that better accounts for correlations between model and variational parameters, improving optimization in complex models.

Contribution

The paper proposes VPNG, a novel natural gradient that incorporates the relationship between model and variational parameters, addressing limitations of traditional natural gradients.

Findings

VPNG improves optimization in variational inference tasks.

Demonstrated on classification, image generation, and recommendation models.

Outperforms traditional natural gradients in experiments.

Abstract

Variational inference transforms posterior inference into parametric optimization thereby enabling the use of latent variable models where otherwise impractical. However, variational inference can be finicky when different variational parameters control variables that are strongly correlated under the model. Traditional natural gradients based on the variational approximation fail to correct for correlations when the approximation is not the true posterior. To address this, we construct a new natural gradient called the Variational Predictive Natural Gradient (VPNG). Unlike traditional natural gradients for variational inference, this natural gradient accounts for the relationship between model parameters and variational parameters. We demonstrate the insight with a simple example as well as the empirical value on a classification task, a deep generative model of images, and probabilistic matrix factorization for recommendation.