# A Linear Systems Theory of Normalizing Flows

## TL;DR

This paper introduces a linear systems theory perspective on Normalizing Flows, revealing they learn local covariances, and proposes a new method for extracting interpretable components, addressing stability issues with theoretical solutions.

## Contribution

It offers a novel theoretical framework for Normalizing Flows using linear systems theory and develops an algorithm for interpretable component extraction, along with stability improvements.

## Key findings

- Normalizing Flows learn local covariance structures.
- The proposed algorithm extracts interpretable components.
- Stability issues in learning are theoretically addressed.

## Abstract

Normalizing Flows are a promising new class of algorithms for unsupervised learning based on maximum likelihood optimization with change of variables. They offer to learn a factorized component representation for complex nonlinear data and, simultaneously, yield a density function that can evaluate likelihoods and generate samples. Despite these diverse offerings, applications of Normalizing Flows have focused primarily on sampling and likelihoods, with little emphasis placed on feature representation. A lack of theoretical foundation has left many open questions about how to interpret and apply the learned components of the model. We provide a new theoretical perspective of Normalizing Flows using the lens of linear systems theory, showing that optimal flows learn to represent the local covariance at each region of input space. Using this insight, we develop a new algorithm to extract interpretable component representations from the learned model, where components correspond to Cartesian dimensions and are scaled according to their manifold significance. In addition, we highlight a stability concern for the learning algorithm that was previously unaddressed, providing a theoretically-grounded solution to mediate the problem. Experiments with toy manifold learning datasets, as well as the MNIST image dataset, provide convincing support for our theory and tools.