An Improved Training Procedure for Neural Autoregressive Data Completion

TL;DR

This paper introduces OA++, an improved training method for neural autoregressive models that enhances data completion performance, reduces computational costs, and leverages prior knowledge about inference queries.

Contribution

The paper proposes OA++, a novel training procedure that outperforms the previous order-agnostic method in data completion tasks by training fewer distributions and utilizing prior knowledge.

Findings

OA++ achieves better accuracy in data completion.

OA++ requires fewer computations than OA.

OA++ effectively uses prior knowledge about inference queries.

Abstract

Neural autoregressive models are explicit density estimators that achieve state-of-the-art likelihoods for generative modeling. The D-dimensional data distribution is factorized into an autoregressive product of one-dimensional conditional distributions according to the chain rule. Data completion is a more involved task than data generation: the model must infer missing variables for any partially observed input vector. Previous work introduced an order-agnostic training procedure for data completion with autoregressive models. Missing variables in any partially observed input vector can be imputed efficiently by choosing an ordering where observed dimensions precede unobserved ones and by computing the autoregressive product in this order. In this paper, we provide evidence that the order-agnostic (OA) training procedure is suboptimal for data completion. We propose an alternative procedure (OA++) that reaches better performance in fewer computations. It can handle all data completion queries while training fewer one-dimensional conditional distributions than the OA procedure. In addition, these one-dimensional conditional distributions are trained proportionally to their expected usage at inference time, reducing overfitting. Finally, our OA++ procedure can exploit prior knowledge about the distribution of inference completion queries, as opposed to OA. We support these claims with quantitative experiments on standard datasets used to evaluate autoregressive generative models.