REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models

TL;DR

This paper introduces REBAR, a novel method that combines control variates and continuous relaxations to produce low-variance, unbiased gradient estimates for models with discrete latent variables, improving training efficiency.

Contribution

The paper presents a new control variate technique that yields unbiased, low-variance gradient estimates and an adaptive relaxation method, advancing discrete latent variable optimization.

Findings

Achieves state-of-the-art variance reduction on benchmark tasks.

Leads to faster convergence and better final log-likelihood.

Removes the need for hyperparameter tuning of relaxation tightness.

Abstract

Learning in models with discrete latent variables is challenging due to high variance gradient estimators. Generally, approaches have relied on control variates to reduce the variance of the REINFORCE estimator. Recent work (Jang et al. 2016, Maddison et al. 2016) has taken a different approach, introducing a continuous relaxation of discrete variables to produce low-variance, but biased, gradient estimates. In this work, we combine the two approaches through a novel control variate that produces low-variance, \emph{unbiased} gradient estimates. Then, we introduce a modification to the continuous relaxation and show that the tightness of the relaxation can be adapted online, removing it as a hyperparameter. We show state-of-the-art variance reduction on several benchmark generative modeling tasks, generally leading to faster convergence to a better final log-likelihood.