Iterative Neural Autoregressive Distribution Estimator (NADE-k)

TL;DR

This paper introduces NADE-k, an extension of the neural autoregressive density estimator that improves learning by multi-step inference, offering analytical likelihood computation, easy sampling, and competitive density estimation performance.

Contribution

NADE-k extends NADE by using multi-step inference, enhancing learning and performance in density estimation tasks.

Findings

NADE-k achieves competitive density estimation results.

It allows analytical computation of test likelihood.

It facilitates easy generation of independent samples.

Abstract

Training of the neural autoregressive density estimator (NADE) can be viewed as doing one step of probabilistic inference on missing values in data. We propose a new model that extends this inference scheme to multiple steps, arguing that it is easier to learn to improve a reconstruction in $k$ steps rather than to learn to reconstruct in a single inference step. The proposed model is an unsupervised building block for deep learning that combines the desirable properties of NADE and multi-predictive training: (1) Its test likelihood can be computed analytically, (2) it is easy to generate independent samples from it, and (3) it uses an inference engine that is a superset of variational inference for Boltzmann machines. The proposed NADE-k is competitive with the state-of-the-art in density estimation on the two datasets tested.