On Kernel-based Variational Autoencoder

TL;DR

This paper introduces a kernel-based approach to variational autoencoders, using KDEs to approximate posteriors and improve the flexibility and quality of generated images, especially with the Epanechnikov kernel.

Contribution

It proposes a novel kernel-based VAE framework that leverages KDEs for posterior approximation and identifies the Epanechnikov kernel as optimal for minimizing KL divergence.

Findings

Epanechnikov kernel improves image quality over Gaussian in VAEs

EVAE achieves lower FID scores on benchmark datasets

Kernel-based approach enhances posterior flexibility in VAEs

Abstract

In this paper, we bridge Variational Autoencoders (VAEs) and kernel density estimations (KDEs) by approximating the posterior by KDEs and deriving an upper bound of the Kullback-Leibler (KL) divergence in the evidence lower bound (ELBO). The flexibility of KDEs makes the optimization of posteriors in VAEs possible, which not only addresses the limitations of Gaussian latent space in vanilla VAE but also provides a new perspective of estimating the KL-divergence in ELBO. Under appropriate conditions, we show that the Epanechnikov kernel is the optimal choice in minimizing the derived upper bound of KL-divergence asymptotically. Compared with Gaussian kernel, Epanechnikov kernel has compact support which should make the generated sample less noisy and blurry. The implementation of Epanechnikov kernel in ELBO is straightforward as it lies in the "location-scale" family of distributions where the reparametrization tricks can be directly employed. A series of experiments on benchmark datasets such as MNIST, Fashion-MNIST, CIFAR-10 and CelebA further demonstrate the superiority of Epanechnikov Variational Autoenocoder (EVAE) over vanilla VAE in the quality of reconstructed images, as measured by the FID score and Sharpness.