Theoretical Convergence Guarantees for Variational Autoencoders

TL;DR

This paper provides the first non-asymptotic convergence guarantees for Variational Autoencoders trained with common optimization algorithms, clarifying their theoretical properties and influencing future research.

Contribution

It derives explicit convergence rates for VAE training with SGD and Adam, applicable to various VAE models and variants, bridging theory and practice.

Findings

Convergence rate of $\\mathcal{O}(\log n / \sqrt{n})$ established.

Explicit dependencies on batch size and hyperparameters identified.

Empirical analysis shows hyperparameters significantly affect convergence.

Abstract

Variational Autoencoders (VAE) are popular generative models used to sample from complex data distributions. Despite their empirical success in various machine learning tasks, significant gaps remain in understanding their theoretical properties, particularly regarding convergence guarantees. This paper aims to bridge that gap by providing non-asymptotic convergence guarantees for VAE trained using both Stochastic Gradient Descent and Adam algorithms. We derive a convergence rate of $\mathcal{O}(\log n / \sqrt{n})$, where $n$ is the number of iterations of the optimization algorithm, with explicit dependencies on the batch size, the number of variational samples, and other key hyperparameters. Our theoretical analysis applies to both Linear VAE and Deep Gaussian VAE, as well as several VAE variants, including $\beta$-VAE and IWAE. Additionally, we empirically illustrate the impact of hyperparameters on convergence, offering new insights into the theoretical understanding of VAE training.