You Only Accept Samples Once: Fast, Self-Correcting Stochastic Variational Inference

TL;DR

YOASOVI is a novel stochastic variational inference algorithm that uses acceptance sampling to improve convergence speed and accuracy on large Bayesian hierarchical models.

Contribution

The paper introduces YOASOVI, a self-correcting stochastic optimization method that replaces traditional sampling with acceptance sampling for faster VI convergence.

Findings

YOASOVI converges faster than Monte Carlo and Quasi-Monte Carlo VI.

It achieves better optimal neighborhoods in less clock time.

Empirical results validate improved efficiency on benchmark datasets.

Abstract

We introduce YOASOVI, an algorithm for performing fast, self-correcting stochastic optimization for Variational Inference (VI) on large Bayesian heirarchical models. To accomplish this, we take advantage of available information on the objective function used for stochastic VI at each iteration and replace regular Monte Carlo sampling with acceptance sampling. Rather than spend computational resources drawing and evaluating over a large sample for the gradient, we draw only one sample and accept it with probability proportional to the expected improvement in the objective. The following paper develops two versions of the algorithm: the first one based on a naive intuition, and another building up the algorithm as a Metropolis-type scheme. Empirical results based on simulations and benchmark datasets for multivariate Gaussian mixture models show that YOASOVI consistently converges faster (in clock time) and within better optimal neighborhoods than both regularized Monte Carlo and Quasi-Monte Carlo VI algorithms.