Learning variational autoencoders via MCMC speed measures

TL;DR

This paper introduces an entropy-based method to adapt short-run MCMC chains within variational autoencoders, resulting in tighter bounds and improved generative performance on complex posterior distributions.

Contribution

It proposes a novel entropy-based adaptation of MCMC proposals for VAEs, enhancing the expressiveness of variational densities and optimizing tighter bounds.

Findings

Higher held-out log-likelihoods achieved

Improved generative metrics observed

Better adaptation to complex posterior geometries

Abstract

Variational autoencoders (VAEs) are popular likelihood-based generative models which can be efficiently trained by maximizing an Evidence Lower Bound (ELBO). There has been much progress in improving the expressiveness of the variational distribution to obtain tighter variational bounds and increased generative performance. Whilst previous work has leveraged Markov chain Monte Carlo (MCMC) methods for the construction of variational densities, gradient-based methods for adapting the proposal distributions for deep latent variable models have received less attention. This work suggests an entropy-based adaptation for a short-run Metropolis-adjusted Langevin (MALA) or Hamiltonian Monte Carlo (HMC) chain while optimising a tighter variational bound to the log-evidence. Experiments show that this approach yields higher held-out log-likelihoods as well as improved generative metrics. Our implicit variational density can adapt to complicated posterior geometries of latent hierarchical representations arising in hierarchical VAEs.