A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text

TL;DR

This paper introduces a simple yet effective fix for posterior collapse in VAEs for text, improving likelihood and representations despite worse ELBO scores, challenging the adequacy of ELBO as a training objective.

Contribution

The paper combines two heuristics to significantly improve VAE training for text, addressing posterior collapse and enhancing representation learning.

Findings

Improved held-out likelihood and reconstruction.

Better latent representations compared to previous methods.

Worse ELBO scores despite better practical performance.

Abstract

When trained effectively, the Variational Autoencoder (VAE) is both a powerful language model and an effective representation learning framework. In practice, however, VAEs are trained with the evidence lower bound (ELBO) as a surrogate objective to the intractable marginal data likelihood. This approach to training yields unstable results, frequently leading to a disastrous local optimum known as posterior collapse. In this paper, we investigate a simple fix for posterior collapse which yields surprisingly effective results. The combination of two known heuristics, previously considered only in isolation, substantially improves held-out likelihood, reconstruction, and latent representation learning when compared with previous state-of-the-art methods. More interestingly, while our experiments demonstrate superiority on these principle evaluations, our method obtains a worse ELBO. We use these results to argue that the typical surrogate objective for VAEs may not be sufficient or necessarily appropriate for balancing the goals of representation learning and data distribution modeling.