A Generalised Linear Model Framework for $\beta$-Variational Autoencoders based on Exponential Dispersion Families

TL;DR

This paper introduces a theoretical framework linking $eta$-VAE to generalized linear models, enabling systematic analysis and improved initialization for training, while explaining auto-pruning and posterior collapse phenomena.

Contribution

It provides a novel connection between $eta$-VAE and GLMs based on exponential dispersion families, allowing for better analysis and initialization strategies.

Findings

MLE-based initialization improves training performance

Analytical description of auto-pruning property

Explanation of posterior collapse in $eta$-VAE

Abstract

Although variational autoencoders (VAE) are successfully used to obtain meaningful low-dimensional representations for high-dimensional data, the characterization of critical points of the loss function for general observation models is not fully understood. We introduce a theoretical framework that is based on a connection between $\beta$-VAE and generalized linear models (GLM). The equality between the activation function of a $\beta$-VAE and the inverse of the link function of a GLM enables us to provide a systematic generalization of the loss analysis for $\beta$-VAE based on the assumption that the observation model distribution belongs to an exponential dispersion family (EDF). As a result, we can initialize $\beta$-VAE nets by maximum likelihood estimates (MLE) that enhance the training performance on both synthetic and real world data sets. As a further consequence, we analytically describe the auto-pruning property inherent in the $\beta$-VAE objective and reason for posterior collapse.