Tutorial: Deriving the Standard Variational Autoencoder (VAE) Loss Function

TL;DR

This tutorial provides a detailed derivation of the standard VAE loss function, including the variational lower bound, KL divergence, and assumptions like Gaussian priors, clarifying the mathematical foundations of VAEs.

Contribution

It offers a comprehensive, step-by-step derivation of the VAE loss function, making the mathematical underpinnings explicit and accessible.

Findings

Closed-form solution for the KL divergence under Gaussian assumptions

Clear derivation from Bayes' theorem to the VAE loss function

Enhanced understanding of the variational lower bound in VAEs

Abstract

In Bayesian machine learning, the posterior distribution is typically computationally intractable, hence variational inference is often required. In this approach, an evidence lower bound on the log likelihood of data is maximized during training. Variational Autoencoders (VAE) are one important example where variational inference is utilized. In this tutorial, we derive the variational lower bound loss function of the standard variational autoencoder. We do so in the instance of a gaussian latent prior and gaussian approximate posterior, under which assumptions the Kullback-Leibler term in the variational lower bound has a closed form solution. We derive essentially everything we use along the way; everything from Bayes' theorem to the Kullback-Leibler divergence.