Variational Autoencoder with Implicit Optimal Priors

TL;DR

This paper introduces a novel method for VAEs that employs an implicit optimal prior estimated via a density ratio trick, leading to improved density estimation performance without explicit modeling of complex priors.

Contribution

The authors propose a new approach to incorporate the aggregated posterior as an implicit prior in VAEs using a density ratio trick, overcoming computational challenges.

Findings

Achieves higher density estimation accuracy on multiple datasets.

Effectively estimates KL divergence without explicit prior modeling.

Improves VAE regularization with implicit optimal priors.

Abstract

The variational autoencoder (VAE) is a powerful generative model that can estimate the probability of a data point by using latent variables. In the VAE, the posterior of the latent variable given the data point is regularized by the prior of the latent variable using Kullback Leibler (KL) divergence. Although the standard Gaussian distribution is usually used for the prior, this simple prior incurs over-regularization. As a sophisticated prior, the aggregated posterior has been introduced, which is the expectation of the posterior over the data distribution. This prior is optimal for the VAE in terms of maximizing the training objective function. However, KL divergence with the aggregated posterior cannot be calculated in a closed form, which prevents us from using this optimal prior. With the proposed method, we introduce the density ratio trick to estimate this KL divergence without modeling the aggregated posterior explicitly. Since the density ratio trick does not work well in high dimensions, we rewrite this KL divergence that contains the high-dimensional density ratio into the sum of the analytically calculable term and the low-dimensional density ratio term, to which the density ratio trick is applied. Experiments on various datasets show that the VAE with this implicit optimal prior achieves high density estimation performance.