On Unifying Deep Generative Models

TL;DR

This paper unifies GANs and VAEs under a common framework by interpreting their training as KL divergence minimization of posterior and inference distributions, enabling cross-method improvements.

Contribution

It introduces a formal connection between GANs and VAEs, extending the wake-sleep algorithm, and demonstrates how techniques can be transferred between the models.

Findings

Transferred importance weighting improves GAN training.

Enhanced VAEs with adversarial mechanisms show better performance.

Unified framework facilitates analysis and development of generative models.

Abstract

Deep generative models have achieved impressive success in recent years. Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), as emerging families for generative model learning, have largely been considered as two distinct paradigms and received extensive independent studies respectively. This paper aims to establish formal connections between GANs and VAEs through a new formulation of them. We interpret sample generation in GANs as performing posterior inference, and show that GANs and VAEs involve minimizing KL divergences of respective posterior and inference distributions with opposite directions, extending the two learning phases of classic wake-sleep algorithm, respectively. The unified view provides a powerful tool to analyze a diverse set of existing model variants, and enables to transfer techniques across research lines in a principled way. For example, we apply the importance weighting method in VAE literatures for improved GAN learning, and enhance VAEs with an adversarial mechanism that leverages generated samples. Experiments show generality and effectiveness of the transferred techniques.