ED-VAE: Entropy Decomposition of ELBO in Variational Autoencoders

TL;DR

ED-VAE introduces a new ELBO formulation that explicitly incorporates entropy components, enabling VAEs to handle complex priors and improve interpretability and generative quality.

Contribution

The paper presents ED-VAE, a reformulation of the ELBO that explicitly includes entropy terms, enhancing flexibility and interpretability of VAEs with complex priors.

Findings

Enhanced generative performance with complex priors

Improved interpretability of latent representations

Better modeling of latent-data interactions

Abstract

Traditional Variational Autoencoders (VAEs) are constrained by the limitations of the Evidence Lower Bound (ELBO) formulation, particularly when utilizing simplistic, non-analytic, or unknown prior distributions. These limitations inhibit the VAE's ability to generate high-quality samples and provide clear, interpretable latent representations. This work introduces the Entropy Decomposed Variational Autoencoder (ED-VAE), a novel re-formulation of the ELBO that explicitly includes entropy and cross-entropy components. This reformulation significantly enhances model flexibility, allowing for the integration of complex and non-standard priors. By providing more detailed control over the encoding and regularization of latent spaces, ED-VAE not only improves interpretability but also effectively captures the complex interactions between latent variables and observed data, thus leading to better generative performance.