# Natural Gradient Hybrid Variational Inference with Application to Deep Mixed Models

## TL;DR

This paper introduces a fast, accurate natural gradient variational inference method for stochastic models with global parameters and latent variables, demonstrating improved efficiency and accuracy over existing methods, especially in deep mixed models.

## Contribution

The paper proposes a hybrid natural gradient variational inference approach that combines global parameter updates with latent variable sampling, enhancing speed and accuracy in high-dimensional models.

## Key findings

- Natural gradient significantly outperforms ordinary gradient in efficiency.
- The method is faster and more accurate than existing natural gradient VI techniques.
- Application to financial models improves asset pricing accuracy.

## Abstract

Stochastic models with global parameters and latent variables are common, and for which variational inference (VI) is popular. However, existing methods are often either slow or inaccurate in high dimensions. We suggest a fast and accurate VI method for this case that employs a well-defined natural gradient variational optimization that targets the joint posterior of the global parameters and latent variables. It is a hybrid method, where at each step the global parameters are updated using the natural gradient and the latent variables are generated from their conditional posterior. A fast to compute expression for the Tikhonov damped Fisher information matrix is used, along with the re-parameterization trick, to provide a stable natural gradient. We apply the approach to deep mixed models, which are an emerging class of Bayesian neural networks with random output layer coefficients to allow for heterogeneity. A range of simulations show that using the natural gradient is substantially more efficient than using the ordinary gradient, and that the approach is faster and more accurate than two cutting-edge natural gradient VI methods. In a financial application we show that accounting for industry level heterogeneity using the deep mixed model improves the accuracy of asset pricing models. MATLAB code to implement the method can be found at: https://github.com/WeibenZhang07/NG-HVI.