Latent Normalizing Flows for Discrete Sequences

TL;DR

This paper introduces a VAE-based model with normalizing flows in the latent space to effectively generate discrete sequences like text and music, balancing flexibility and speed.

Contribution

It proposes novel flow architectures for highly multimodal latent distributions and demonstrates their effectiveness on language and music generation tasks.

Findings

Autoregressive flow models match baseline performance.

Non-autoregressive models improve generation speed.

Flow architectures enhance modeling of discrete sequences.

Abstract

Normalizing flows are a powerful class of generative models for continuous random variables, showing both strong model flexibility and the potential for non-autoregressive generation. These benefits are also desired when modeling discrete random variables such as text, but directly applying normalizing flows to discrete sequences poses significant additional challenges. We propose a VAE-based generative model which jointly learns a normalizing flow-based distribution in the latent space and a stochastic mapping to an observed discrete space. In this setting, we find that it is crucial for the flow-based distribution to be highly multimodal. To capture this property, we propose several normalizing flow architectures to maximize model flexibility. Experiments consider common discrete sequence tasks of character-level language modeling and polyphonic music generation. Our results indicate that an autoregressive flow-based model can match the performance of a comparable autoregressive baseline, and a non-autoregressive flow-based model can improve generation speed with a penalty to performance.