Discrete Flows: Invertible Generative Models of Discrete Data

TL;DR

This paper extends normalizing flows to discrete data using a simple change-of-variables approach, enabling bidirectional and efficient non-autoregressive discrete data modeling with promising empirical results.

Contribution

It introduces a novel method for applying invertible flow models to discrete data, including architectures for autoregressive and bipartite flows, without requiring log-determinant-Jacobian computations.

Findings

Discrete autoregressive flows outperform baselines on synthetic and Potts models.

Bipartite flows achieve competitive results on character-level language modeling.

The proposed methods enable bidirectional and efficient discrete data modeling.

Abstract

While normalizing flows have led to significant advances in modeling high-dimensional continuous distributions, their applicability to discrete distributions remains unknown. In this paper, we show that flows can in fact be extended to discrete events---and under a simple change-of-variables formula not requiring log-determinant-Jacobian computations. Discrete flows have numerous applications. We consider two flow architectures: discrete autoregressive flows that enable bidirectionality, allowing, for example, tokens in text to depend on both left-to-right and right-to-left contexts in an exact language model; and discrete bipartite flows that enable efficient non-autoregressive generation as in RealNVP. Empirically, we find that discrete autoregressive flows outperform autoregressive baselines on synthetic discrete distributions, an addition task, and Potts models; and bipartite flows can obtain competitive performance with autoregressive baselines on character-level language modeling for Penn Tree Bank and text8.