Bridging the ELBO and MMD

TL;DR

This paper introduces a new objective replacing the KL divergence in VAEs with an MMD-based term and a latent clipping technique, improving training stability and sample quality, especially on small datasets.

Contribution

It proposes a novel objective function and latent clipping method for VAEs, reducing posterior collapse and enhancing generative and representation quality.

Findings

$ta$-VAE outperforms ELBO and $eta$-VAE in experiments

Latent representations are effective for downstream tasks

Model generates high-quality reconstructions and samples

Abstract

One of the challenges in training generative models such as the variational auto encoder (VAE) is avoiding posterior collapse. When the generator has too much capacity, it is prone to ignoring latent code. This problem is exacerbated when the dataset is small, and the latent dimension is high. The root of the problem is the ELBO objective, specifically the Kullback-Leibler (KL) divergence term in objective function \citep{zhao2019infovae}. This paper proposes a new objective function to replace the KL term with one that emulates the maximum mean discrepancy (MMD) objective. It also introduces a new technique, named latent clipping, that is used to control distance between samples in latent space. A probabilistic autoencoder model, named $\mu$-VAE, is designed and trained on MNIST and MNIST Fashion datasets, using the new objective function and is shown to outperform models trained with ELBO and $\beta$-VAE objective. The $\mu$-VAE is less prone to posterior collapse, and can generate reconstructions and new samples in good quality. Latent representations learned by $\mu$-VAE are shown to be good and can be used for downstream tasks such as classification.