Differentiable Antithetic Sampling for Variance Reduction in Stochastic Variational Inference

TL;DR

This paper introduces a differentiable antithetic sampling method that reduces variance in stochastic gradient estimates for variational inference, improving the efficiency of deep generative model training.

Contribution

It proposes a novel differentiable antithetic sampling technique that generates correlated samples matching true moments, enhancing variance reduction in stochastic variational inference.

Findings

Reduced estimator variance with antithetic sampling

Improved training efficiency for deep generative models

Demonstrated effectiveness on variational inference tasks

Abstract

Stochastic optimization techniques are standard in variational inference algorithms. These methods estimate gradients by approximating expectations with independent Monte Carlo samples. In this paper, we explore a technique that uses correlated, but more representative , samples to reduce estimator variance. Specifically, we show how to generate antithetic samples that match sample moments with the true moments of an underlying importance distribution. Combining a differentiable antithetic sampler with modern stochastic variational inference, we showcase the effectiveness of this approach for learning a deep generative model.