From optimal transport to generative modeling: the VEGAN cookbook

TL;DR

This paper connects optimal transport theory with generative modeling, providing theoretical insights into auto-encoders and GANs, and introduces a penalized OT approach that unifies several existing methods.

Contribution

It introduces the penalized optimal transport framework for generative modeling, offering theoretical justification for adversarial auto-encoders and linking Wasserstein GANs with OT.

Findings

POT for 2-Wasserstein aligns with adversarial auto-encoders.

Provides theoretical explanation for VAE image blurriness.

Establishes duality between Wasserstein GAN and OT for 1-Wasserstein.

Abstract

We study unsupervised generative modeling in terms of the optimal transport (OT) problem between true (but unknown) data distribution $P_X$ and the latent variable model distribution $P_G$. We show that the OT problem can be equivalently written in terms of probabilistic encoders, which are constrained to match the posterior and prior distributions over the latent space. When relaxed, this constrained optimization problem leads to a penalized optimal transport (POT) objective, which can be efficiently minimized using stochastic gradient descent by sampling from $P_X$ and $P_G$. We show that POT for the 2-Wasserstein distance coincides with the objective heuristically employed in adversarial auto-encoders (AAE) (Makhzani et al., 2016), which provides the first theoretical justification for AAEs known to the authors. We also compare POT to other popular techniques like variational auto-encoders (VAE) (Kingma and Welling, 2014). Our theoretical results include (a) a better understanding of the commonly observed blurriness of images generated by VAEs, and (b) establishing duality between Wasserstein GAN (Arjovsky and Bottou, 2017) and POT for the 1-Wasserstein distance.