Autoregressive Energy Machines

TL;DR

The paper introduces the Autoregressive Energy Machine, a novel energy-based model that efficiently estimates high-dimensional densities by learning unnormalized energies and normalizing constants autoregressively, achieving state-of-the-art results.

Contribution

It presents a new autoregressive energy-based model that jointly learns unnormalized densities and estimates normalization constants, overcoming previous limitations.

Findings

Achieves state-of-the-art density estimation performance

Effectively estimates normalization constants autoregressively

Demonstrates improved flexibility over traditional density estimators

Abstract

Neural density estimators are flexible families of parametric models which have seen widespread use in unsupervised machine learning in recent years. Maximum-likelihood training typically dictates that these models be constrained to specify an explicit density. However, this limitation can be overcome by instead using a neural network to specify an energy function, or unnormalized density, which can subsequently be normalized to obtain a valid distribution. The challenge with this approach lies in accurately estimating the normalizing constant of the high-dimensional energy function. We propose the Autoregressive Energy Machine, an energy-based model which simultaneously learns an unnormalized density and computes an importance-sampling estimate of the normalizing constant for each conditional in an autoregressive decomposition. The Autoregressive Energy Machine achieves state-of-the-art performance on a suite of density-estimation tasks.