Improving latent variable descriptiveness with AutoGen

TL;DR

This paper introduces AutoGen, a new latent variable modeling approach combining data likelihood and autoencoder reconstruction, ensuring better use of latent variables and providing a formal interpretation of VAE training pre-factors.

Contribution

AutoGen offers a unified objective that improves latent variable informativeness and clarifies the role of pre-factors in VAE training, bridging autoencoding and variational methods.

Findings

AutoGen's lower bound matches the standard VAE bound with an added pre-factor.

The approach guarantees the latent variable captures observation information.

Provides a theoretical foundation for ad-hoc VAE training pre-factors.

Abstract

Powerful generative models, particularly in Natural Language Modelling, are commonly trained by maximizing a variational lower bound on the data log likelihood. These models often suffer from poor use of their latent variable, with ad-hoc annealing factors used to encourage retention of information in the latent variable. We discuss an alternative and general approach to latent variable modelling, based on an objective that combines the data log likelihood as well as the likelihood of a perfect reconstruction through an autoencoder. Tying these together ensures by design that the latent variable captures information about the observations, whilst retaining the ability to generate well. Interestingly, though this approach is a priori unrelated to VAEs, the lower bound attained is identical to the standard VAE bound but with the addition of a simple pre-factor; thus, providing a formal interpretation of the commonly used, ad-hoc pre-factors in training VAEs.