Posterior Collapse as Automatic Spectral Pruning

TL;DR

This paper demonstrates that posterior collapse in $eta$-VAEs functions as an automatic spectral pruning process, where latent modes diminish based on their contribution to reconstruction relative to a $eta$-dependent cutoff.

Contribution

It provides a theoretical framework linking posterior collapse to spectral pruning, including stability analysis and a new order parameter for ranking latent modes.

Findings

Collapse thresholds predict which latent modes will decouple.

In the linear Gaussian case, collapse, utility, and PCA spectra coincide.

Predictions validated on the WorldClim dataset.

Abstract

We show that posterior collapse in $\beta$-VAEs implements automatic spectral pruning. A latent mode collapses if its contribution to reconstruction is below the cutoff set by $\beta$. Equilibrium solutions with different $\beta$ thus reveal a cascade of collapses as latent modes decouple from least to most useful. We derive this as a consequence of the loss via a Landau stability analysis. We define a latent-rescaling-invariant order parameter that ranks active latent modes and whose collapse thresholds identify which effective variables to inspect first. In the linear Gaussian case, the collapse spectrum, utility spectrum, and normalized PCA spectrum coincide, and each collapse follows a mean-field law. We test these predictions on the WorldClim dataset.