Ordering Dimensions with Nested Dropout Normalizing Flows

TL;DR

This paper introduces a method for ordering latent dimensions in normalizing flows using nested dropout, enabling the learning of low-dimensional, semantically meaningful representations while maintaining full data support.

Contribution

It proposes a novel approach to order latent variables in normalizing flows with full support, balancing likelihood and meaningfulness of the learned representations.

Findings

Ordered latent variables improve interpretability.

Trade-off observed between likelihood and ordering quality.

Method maintains full data support while learning low-dimensional representations.

Abstract

The latent space of normalizing flows must be of the same dimensionality as their output space. This constraint presents a problem if we want to learn low-dimensional, semantically meaningful representations. Recent work has provided compact representations by fitting flows constrained to manifolds, but hasn't defined a density off that manifold. In this work we consider flows with full support in data space, but with ordered latent variables. Like in PCA, the leading latent dimensions define a sequence of manifolds that lie close to the data. We note a trade-off between the flow likelihood and the quality of the ordering, depending on the parameterization of the flow.