Generalization of the Change of Variables Formula with Applications to Residual Flows

TL;DR

This paper extends the Change of Variables Formula to include non-smooth transformations called $\\mathcal{L}$-diffeomorphisms, enabling the use of non-smooth activations in normalizing flows, and applies this to residual flow models.

Contribution

It introduces $\\mathcal{L}$-diffeomorphisms as a generalization of diffeomorphisms, broadening the design space of normalizing flows to include non-smooth transformations.

Findings

Enlarged the class of transformations for normalizing flows.

Allowed the use of ReLU and similar non-smooth activations.

Applied the theory to residual flow architectures.

Abstract

Normalizing flows leverage the Change of Variables Formula (CVF) to define flexible density models. Yet, the requirement of smooth transformations (diffeomorphisms) in the CVF poses a significant challenge in the construction of these models. To enlarge the design space of flows, we introduce $\mathcal{L}$-diffeomorphisms as generalized transformations which may violate these requirements on zero Lebesgue-measure sets. This relaxation allows e.g. the use of non-smooth activation functions such as ReLU. Finally, we apply the obtained results to planar, radial, and contractive residual flows.