Normalizing Flows Across Dimensions

TL;DR

This paper introduces noisy injective flows (NIF), a novel generative model that extends normalizing flows to learn across dimensions, effectively capturing low-dimensional structures in high-dimensional data.

Contribution

NIF generalizes normalizing flows by enabling dimension-crossing mappings and explicitly modeling data manifolds with injective transformations and noise.

Findings

Improved sample quality with NIF applied to existing architectures.

NIF produces more separable and meaningful data embeddings.

Significant enhancement over traditional normalizing flows.

Abstract

Real-world data with underlying structure, such as pictures of faces, are hypothesized to lie on a low-dimensional manifold. This manifold hypothesis has motivated state-of-the-art generative algorithms that learn low-dimensional data representations. Unfortunately, a popular generative model, normalizing flows, cannot take advantage of this. Normalizing flows are based on successive variable transformations that are, by design, incapable of learning lower-dimensional representations. In this paper we introduce noisy injective flows (NIF), a generalization of normalizing flows that can go across dimensions. NIF explicitly map the latent space to a learnable manifold in a high-dimensional data space using injective transformations. We further employ an additive noise model to account for deviations from the manifold and identify a stochastic inverse of the generative process. Empirically, we demonstrate that a simple application of our method to existing flow architectures can significantly improve sample quality and yield separable data embeddings.