Understanding disentangling in $\beta$-VAE

TL;DR

This paper offers new theoretical insights into how disentangled representations emerge in $eta$-VAE, proposing a training modification that improves disentanglement without sacrificing reconstruction quality.

Contribution

It introduces a rate-distortion perspective to understand disentanglement and proposes a progressive capacity increase method for better $eta$-VAE training.

Findings

Disentanglement emerges under specific rate-distortion conditions.

Progressive capacity increase improves disentanglement and reconstruction.

Theoretical assessment clarifies conditions for disentangled representation emergence.

Abstract

We present new intuitions and theoretical assessments of the emergence of disentangled representation in variational autoencoders. Taking a rate-distortion theory perspective, we show the circumstances under which representations aligned with the underlying generative factors of variation of data emerge when optimising the modified ELBO bound in $\beta$-VAE, as training progresses. From these insights, we propose a modification to the training regime of $\beta$-VAE, that progressively increases the information capacity of the latent code during training. This modification facilitates the robust learning of disentangled representations in $\beta$-VAE, without the previous trade-off in reconstruction accuracy.