Implicit Manifold Learning on Generative Adversarial Networks

TL;DR

This paper investigates how different training objectives in GANs influence the learned data support manifold, revealing that Jensen-Shannon divergence aligns the model support with real data, while Wasserstein distances have different properties.

Contribution

It introduces an implicit manifold learning perspective in GANs and compares the effects of Jensen-Shannon and Wasserstein distances on the learned support.

Findings

Jensen-Shannon divergence aligns the model support with real data support.

Wasserstein distance does not necessarily match the supports.

Wasserstein W2^2 may reduce mode collapse.

Abstract

This paper raises an implicit manifold learning perspective in Generative Adversarial Networks (GANs), by studying how the support of the learned distribution, modelled as a submanifold $\mathcal{M}_{\theta}$, perfectly match with $\mathcal{M}_{r}$, the support of the real data distribution. We show that optimizing Jensen-Shannon divergence forces $\mathcal{M}_{\theta}$ to perfectly match with $\mathcal{M}_{r}$, while optimizing Wasserstein distance does not. On the other hand, by comparing the gradients of the Jensen-Shannon divergence and the Wasserstein distances ($W_1$ and $W_2^2$) in their primal forms, we conjecture that Wasserstein $W_2^2$ may enjoy desirable properties such as reduced mode collapse. It is therefore interesting to design new distances that inherit the best from both distances.