Disentanglement as Identifiable Pushforward Factorisation

TL;DR

This paper characterizes disentanglement in smooth generative models like VAEs and GANs, linking it to the Jacobian's SVD and conditions on the generator, with experiments on various datasets.

Contribution

It provides a theoretical framework connecting disentanglement to the Jacobian's SVD and identifies conditions for disentanglement and its identifiability.

Findings

Disentanglement corresponds to the SVD of the generator's Jacobian.

Diagonal posteriors in Gaussian VAEs promote disentanglement conditions.

Experiments confirm the theoretical mechanism on multiple datasets.

Abstract

We characterise disentanglement in smooth generative pushforward models, such as in VAEs and GANs. For a generator/decoder $g:Z\to X$ and factorised prior $p(z)=\prod_i p_i(z_i)$, we define disentanglement as factorisation of the pushforward density $p_\mu= g_\#p$ into one-dimensional "seam" factors, where each latent dimension controls an independent generative factor of the data. We prove that $p_\mu$ factorises according to the SVD of $g$'s Jacobian; that disentanglement equates to two conditions on $g$ (C1-C2); and that under those conditions the seam factors are identifiable, up to permutation and sign. In the particular case of Gaussian ($\beta$-)VAEs, we show via an identity how diagonal posteriors promote C1-C2, in expectation, explaining why disentanglement arises modulated by $\beta$. Experiments illustrate this mechanism on Gaussian data, dSprites, and CelebA.