Discrete Variational Autoencoders

TL;DR

This paper introduces a novel training method for discrete variational autoencoders that enables backpropagation through discrete variables, improving unsupervised learning of object classes and details.

Contribution

It presents a new approach to train probabilistic models with discrete latent variables within the variational autoencoder framework, including backpropagation capabilities.

Findings

Outperforms state-of-the-art on MNIST, Omniglot, Caltech-101 datasets

Efficiently learns object classes and details from unsupervised data

Handles discrete and continuous components in probabilistic models

Abstract

Probabilistic models with discrete latent variables naturally capture datasets composed of discrete classes. However, they are difficult to train efficiently, since backpropagation through discrete variables is generally not possible. We present a novel method to train a class of probabilistic models with discrete latent variables using the variational autoencoder framework, including backpropagation through the discrete latent variables. The associated class of probabilistic models comprises an undirected discrete component and a directed hierarchical continuous component. The discrete component captures the distribution over the disconnected smooth manifolds induced by the continuous component. As a result, this class of models efficiently learns both the class of objects in an image, and their specific realization in pixels, from unsupervised data, and outperforms state-of-the-art methods on the permutation-invariant MNIST, Omniglot, and Caltech-101 Silhouettes datasets.