Improving Variational Inference with Inverse Autoregressive Flow

TL;DR

This paper introduces inverse autoregressive flow (IAF), a new normalizing flow method that enhances variational inference by efficiently modeling high-dimensional posteriors, leading to improved approximation and faster image synthesis.

Contribution

The paper proposes IAF, a scalable normalizing flow based on autoregressive neural networks, significantly improving variational inference for high-dimensional latent spaces.

Findings

IAF outperforms diagonal Gaussian posteriors in experiments.

A new variational autoencoder with IAF achieves competitive log-likelihoods.

IAF enables faster image synthesis compared to autoregressive models.

Abstract

The framework of normalizing flows provides a general strategy for flexible variational inference of posteriors over latent variables. We propose a new type of normalizing flow, inverse autoregressive flow (IAF), that, in contrast to earlier published flows, scales well to high-dimensional latent spaces. The proposed flow consists of a chain of invertible transformations, where each transformation is based on an autoregressive neural network. In experiments, we show that IAF significantly improves upon diagonal Gaussian approximate posteriors. In addition, we demonstrate that a novel type of variational autoencoder, coupled with IAF, is competitive with neural autoregressive models in terms of attained log-likelihood on natural images, while allowing significantly faster synthesis.