Variational Lossy Autoencoder

TL;DR

This paper introduces a variational autoencoder framework combined with autoregressive models to learn global, lossy representations of data, improving generative performance and enabling control over the information captured in the latent space.

Contribution

It proposes a novel VAE architecture that leverages autoregressive models for better control of learned representations and enhanced generative modeling.

Findings

Achieved state-of-the-art results on MNIST, OMNIGLOT, and Caltech-101 Silhouettes.

Demonstrated control over the type of information encoded in the latent space.

Improved generative modeling performance with autoregressive priors and decoders.

Abstract

Representation learning seeks to expose certain aspects of observed data in a learned representation that's amenable to downstream tasks like classification. For instance, a good representation for 2D images might be one that describes only global structure and discards information about detailed texture. In this paper, we present a simple but principled method to learn such global representations by combining Variational Autoencoder (VAE) with neural autoregressive models such as RNN, MADE and PixelRNN/CNN. Our proposed VAE model allows us to have control over what the global latent code can learn and , by designing the architecture accordingly, we can force the global latent code to discard irrelevant information such as texture in 2D images, and hence the VAE only "autoencodes" data in a lossy fashion. In addition, by leveraging autoregressive models as both prior distribution $p(z)$ and decoding distribution $p(x|z)$, we can greatly improve generative modeling performance of VAEs, achieving new state-of-the-art results on MNIST, OMNIGLOT and Caltech-101 Silhouettes density estimation tasks.