Langevin Autoencoders for Learning Deep Latent Variable Models

TL;DR

This paper introduces Langevin autoencoders and amortized Langevin dynamics, enabling efficient deep latent variable modeling by replacing costly MCMC sampling with encoder updates, and proves their validity as MCMC algorithms.

Contribution

The paper proposes ALD to replace datapoint-wise MCMC with encoder updates, and introduces the Langevin autoencoder, a novel deep latent variable model based on this method.

Findings

ALD accurately samples from target posteriors in synthetic datasets.

LAE outperforms variational autoencoders in image generation tasks.

LAE surpasses existing MCMC-based methods in test likelihood.

Abstract

Markov chain Monte Carlo (MCMC), such as Langevin dynamics, is valid for approximating intractable distributions. However, its usage is limited in the context of deep latent variable models owing to costly datapoint-wise sampling iterations and slow convergence. This paper proposes the amortized Langevin dynamics (ALD), wherein datapoint-wise MCMC iterations are entirely replaced with updates of an encoder that maps observations into latent variables. This amortization enables efficient posterior sampling without datapoint-wise iterations. Despite its efficiency, we prove that ALD is valid as an MCMC algorithm, whose Markov chain has the target posterior as a stationary distribution under mild assumptions. Based on the ALD, we also present a new deep latent variable model named the Langevin autoencoder (LAE). Interestingly, the LAE can be implemented by slightly modifying the traditional autoencoder. Using multiple synthetic datasets, we first validate that ALD can properly obtain samples from target posteriors. We also evaluate the LAE on the image generation task, and show that our LAE can outperform existing methods based on variational inference, such as the variational autoencoder, and other MCMC-based methods in terms of the test likelihood.