Benefiting Deep Latent Variable Models via Learning the Prior and Removing Latent Regularization

TL;DR

This paper challenges the necessity of latent regularization in deep latent variable models like VAEs, showing that learned expressive priors can improve image quality and benefit tasks such as disentanglement and image translation.

Contribution

It demonstrates that latent regularization can be omitted when using expressive learned priors, leading to better image quality and improved performance in vision tasks.

Findings

Latent regularization may harm image quality with expressive priors.

Learned priors enhance disentanglement in latent spaces.

Improved diversity in image-to-image translation.

Abstract

There exist many forms of deep latent variable models, such as the variational autoencoder and adversarial autoencoder. Regardless of the specific class of model, there exists an implicit consensus that the latent distribution should be regularized towards the prior, even in the case where the prior distribution is learned. Upon investigating the effect of latent regularization on image generation our results indicate that in the case where a sufficiently expressive prior is learned, latent regularization is not necessary and may in fact be harmful insofar as image quality is concerned. We additionally investigate the benefit of learned priors on two common problems in computer vision: latent variable disentanglement, and diversity in image-to-image translation.