Invertible Monotone Operators for Normalizing Flows

TL;DR

This paper introduces Monotone Flows, a novel invertible model using monotone operators and a new activation function, CPila, to enhance the flexibility and performance of ResNet-based normalizing flows for density estimation.

Contribution

It proposes a monotone operator formulation to address Lipschitz constraints and introduces the CPila activation function, advancing the design of ResNet-based normalizing flows.

Findings

Achieved state-of-the-art results on MNIST, CIFAR-10, ImageNet32, and ImageNet64.

Demonstrated improved gradient flow and model flexibility.

Provided theoretical analysis of monotone operator-based flows.

Abstract

Normalizing flows model probability distributions by learning invertible transformations that transfer a simple distribution into complex distributions. Since the architecture of ResNet-based normalizing flows is more flexible than that of coupling-based models, ResNet-based normalizing flows have been widely studied in recent years. Despite their architectural flexibility, it is well-known that the current ResNet-based models suffer from constrained Lipschitz constants. In this paper, we propose the monotone formulation to overcome the issue of the Lipschitz constants using monotone operators and provide an in-depth theoretical analysis. Furthermore, we construct an activation function called Concatenated Pila (CPila) to improve gradient flow. The resulting model, Monotone Flows, exhibits an excellent performance on multiple density estimation benchmarks (MNIST, CIFAR-10, ImageNet32, ImageNet64). Code is available at https://github.com/mlvlab/MonotoneFlows.