Deep AutoRegressive Networks

TL;DR

This paper presents a deep autoregressive generative model with hierarchical representations, enabling fast sampling and state-of-the-art performance on various datasets.

Contribution

It introduces a novel deep autoencoder with autoregressive connections and an efficient MDL-based training method for improved generative modeling.

Findings

Achieved state-of-the-art results on UCI datasets, MNIST, and Atari games.

Developed a fast, exact ancestral sampling procedure.

Proposed an efficient variational inference approach based on MDL.

Abstract

We introduce a deep, generative autoencoder capable of learning hierarchies of distributed representations from data. Successive deep stochastic hidden layers are equipped with autoregressive connections, which enable the model to be sampled from quickly and exactly via ancestral sampling. We derive an efficient approximate parameter estimation method based on the minimum description length (MDL) principle, which can be seen as maximising a variational lower bound on the log-likelihood, with a feedforward neural network implementing approximate inference. We demonstrate state-of-the-art generative performance on a number of classic data sets: several UCI data sets, MNIST and Atari 2600 games.