Simple and Effective VAE Training with Calibrated Decoders

TL;DR

This paper demonstrates that using calibrated decoders in VAEs, especially with a novel analytical variance computation, simplifies training and improves performance across various image and video datasets.

Contribution

It introduces a comprehensive analysis of calibrated decoders in VAEs and proposes a simple analytical variance modification to enhance training effectiveness.

Findings

Calibrated decoders improve VAE training without heuristic hyperparameters.

Analytical variance computation in Gaussian decoders is effective.

The method performs well across diverse datasets and models.

Abstract

Variational autoencoders (VAEs) provide an effective and simple method for modeling complex distributions. However, training VAEs often requires considerable hyperparameter tuning to determine the optimal amount of information retained by the latent variable. We study the impact of calibrated decoders, which learn the uncertainty of the decoding distribution and can determine this amount of information automatically, on the VAE performance. While many methods for learning calibrated decoders have been proposed, many of the recent papers that employ VAEs rely on heuristic hyperparameters and ad-hoc modifications instead. We perform the first comprehensive comparative analysis of calibrated decoder and provide recommendations for simple and effective VAE training. Our analysis covers a range of image and video datasets and several single-image and sequential VAE models. We further propose a simple but novel modification to the commonly used Gaussian decoder, which computes the prediction variance analytically. We observe empirically that using heuristic modifications is not necessary with our method. Project website is at https://orybkin.github.io/sigma-vae/