Posterior Collapse of a Linear Latent Variable Model

TL;DR

This paper investigates a specific type of posterior collapse in linear latent variable models, revealing its causes and potential connections to neural and dimensional collapse, with implications for deep learning architectures.

Contribution

It precisely characterizes the cause of posterior collapse in linear latent variable models as a competition between likelihood and prior regularization.

Findings

Identifies the cause of posterior collapse in linear models.

Links posterior collapse to neural and dimensional collapse.

Provides insights into learning challenges in deep architectures.

Abstract

This work identifies the existence and cause of a type of posterior collapse that frequently occurs in the Bayesian deep learning practice. For a general linear latent variable model that includes linear variational autoencoders as a special case, we precisely identify the nature of posterior collapse to be the competition between the likelihood and the regularization of the mean due to the prior. Our result suggests that posterior collapse may be related to neural collapse and dimensional collapse and could be a subclass of a general problem of learning for deeper architectures.