Relaxing Bijectivity Constraints with Continuously Indexed Normalising Flows

TL;DR

This paper introduces Continuously Indexed Flows (CIFs), a novel approach that overcomes topological limitations of traditional normalising flows by using a family of bijections, leading to improved modeling of complex data supports.

Contribution

The paper proposes CIFs, a new class of flow models that relax bijectivity constraints, with theoretical advantages and empirical improvements over existing normalising flows.

Findings

CIFs are not limited by topological constraints of normalising flows.

CIFs outperform traditional flows on various benchmarks.

Normalising flows become noninvertible with complex supports.

Abstract

We show that normalising flows become pathological when used to model targets whose supports have complicated topologies. In this scenario, we prove that a flow must become arbitrarily numerically noninvertible in order to approximate the target closely. This result has implications for all flow-based models, and especially Residual Flows (ResFlows), which explicitly control the Lipschitz constant of the bijection used. To address this, we propose Continuously Indexed Flows (CIFs), which replace the single bijection used by normalising flows with a continuously indexed family of bijections, and which can intuitively "clean up" mass that would otherwise be misplaced by a single bijection. We show theoretically that CIFs are not subject to the same topological limitations as normalising flows, and obtain better empirical performance on a variety of models and benchmarks.