Learning and Generalization in Overparameterized Normalizing Flows

TL;DR

This paper investigates the effects of overparameterization in normalizing flows, revealing that it can hinder training in constrained models but enables efficient learning in unconstrained models, both theoretically and empirically.

Contribution

It provides the first theoretical and empirical analysis of overparameterization effects in normalizing flows, distinguishing between constrained and unconstrained models.

Findings

Overparameterization hampers training in constrained NFs.

Unconstrained NFs can efficiently learn data distributions when overparameterized.

Empirical results support theoretical claims.

Abstract

In supervised learning, it is known that overparameterized neural networks with one hidden layer provably and efficiently learn and generalize, when trained using stochastic gradient descent with a sufficiently small learning rate and suitable initialization. In contrast, the benefit of overparameterization in unsupervised learning is not well understood. Normalizing flows (NFs) constitute an important class of models in unsupervised learning for sampling and density estimation. In this paper, we theoretically and empirically analyze these models when the underlying neural network is a one-hidden-layer overparametrized network. Our main contributions are two-fold: (1) On the one hand, we provide theoretical and empirical evidence that for constrained NFs (this class of NFs underlies many NF constructions) with the one-hidden-layer network, overparametrization hurts training. (2) On the other hand, we prove that unconstrained NFs, a recently introduced model, can efficiently learn any reasonable data distribution under minimal assumptions when the underlying network is overparametrized and has one hidden-layer.