RNADE: The real-valued neural autoregressive density-estimator

TL;DR

RNADE is a novel neural autoregressive model for estimating joint densities of real-valued data, combining mixture density networks with shared parameters for efficient and tractable density computation.

Contribution

RNADE introduces a new autoregressive density estimator that learns distributed data representations with tractable likelihoods, outperforming traditional mixture models in most cases.

Findings

RNADE outperforms mixture models on several datasets.

The model provides a tractable likelihood for direct comparison and training.

RNADE effectively models heterogeneous and perceptual data.

Abstract

We introduce RNADE, a new model for joint density estimation of real-valued vectors. Our model calculates the density of a datapoint as the product of one-dimensional conditionals modeled using mixture density networks with shared parameters. RNADE learns a distributed representation of the data, while having a tractable expression for the calculation of densities. A tractable likelihood allows direct comparison with other methods and training by standard gradient-based optimizers. We compare the performance of RNADE on several datasets of heterogeneous and perceptual data, finding it outperforms mixture models in all but one case.