Variations and Relaxations of Normalizing Flows

TL;DR

This paper surveys recent variations and relaxations of Normalizing Flows, highlighting methods that enhance their expressivity and flexibility by combining them with other generative models like VAEs and diffusion, while addressing their limitations.

Contribution

It provides a comprehensive overview of recent approaches that relax the bijectivity constraints of NFs, improving their ability to model complex distributions.

Findings

Combining NFs with VAEs and diffusion models increases expressivity.

Relaxations enable modeling of non-homeomorphic distributions.

Trade-offs between expressivity and computational efficiency are discussed.

Abstract

Normalizing Flows (NFs) describe a class of models that express a complex target distribution as the composition of a series of bijective transformations over a simpler base distribution. By limiting the space of candidate transformations to diffeomorphisms, NFs enjoy efficient, exact sampling and density evaluation, enabling NFs to flexibly behave as both discriminative and generative models. Their restriction to diffeomorphisms, however, enforces that input, output and all intermediary spaces share the same dimension, limiting their ability to effectively represent target distributions with complex topologies. Additionally, in cases where the prior and target distributions are not homeomorphic, Normalizing Flows can leak mass outside of the support of the target. This survey covers a selection of recent works that combine aspects of other generative model classes, such as VAEs and score-based diffusion, and in doing so loosen the strict bijectivity constraints of NFs to achieve a balance of expressivity, training speed, sample efficiency and likelihood tractability.