Entropy-Based Characterisation of the Polarised Regime in Latent Variable Models

TL;DR

This paper introduces an entropy-based method to identify active, passive, and mixed latent dimensions in Variational Autoencoders, providing a theoretical foundation and empirical validation across multiple models.

Contribution

It proposes a novel information-theoretic criterion based on entropy of the mean representation to characterize the polarised regime in VAEs, extending beyond Gaussian priors.

Findings

The entropy criterion reliably detects polarised regimes across various autoencoder models.

Passive dimensions can improve downstream task performance when latent codes are normalized.

The entropy of the mean alone cannot distinguish active from mixed dimensions without variance signals.

Abstract

Variational Autoencoders (VAEs) often exhibit a polarised regime in which latent variables separate into active, passive, and mixed subsets. Existing criteria for identifying active dimensions depend on a Gaussian prior, limiting their applicability to variational models and specific priors. We propose a simple information-theoretic classification of the polarised regime based on the entropy of the mean representation. We show theoretically how this entropy couples to KL minimisation through entropy--variance bounds, and we relate the resulting criterion to Bonheme's active/passive conditions. We also clarify a key limitation: entropy of the mean alone cannot reliably distinguish active from mixed dimensions without additional signals from the variance representation. Empirically, we evaluate the entropy criterion on $\beta$-VAEs, identifiable VAEs, Least-Volume Autoencoders, and L2-regularised autoencoders, and find that it consistently recovers a polarised regime when such a regime is present across the model classes studied. Finally, we show that passive dimensions can yield small but consistent improvements on downstream tasks when latent codes are appropriately normalised, suggesting that collapse is often a matter of scale rather than absolute information removal.