Variance Constrained Autoencoding

TL;DR

This paper introduces the Variance Constrained Autoencoder (VCAE), a new model that improves generative and disentangled representations by enforcing only a variance constraint on the latent space, avoiding issues with distribution enforcement.

Contribution

The paper proposes VCAE, which enforces a variance constraint instead of a full distribution constraint, leading to better reconstruction, generation, and disentanglement performance.

Findings

VCAE outperforms Wasserstein and Variational Autoencoders on MNIST and CelebA.

VCAE with total correlation penalty matches FactorVAE in disentanglement on 3D-Shapes.

Enforcing only variance constraints simplifies training and improves quality.

Abstract

Recent state-of-the-art autoencoder based generative models have an encoder-decoder structure and learn a latent representation with a pre-defined distribution that can be sampled from. Implementing the encoder networks of these models in a stochastic manner provides a natural and common approach to avoid overfitting and enforce a smooth decoder function. However, we show that for stochastic encoders, simultaneously attempting to enforce a distribution constraint and minimising an output distortion leads to a reduction in generative and reconstruction quality. In addition, attempting to enforce a latent distribution constraint is not reasonable when performing disentanglement. Hence, we propose the variance-constrained autoencoder (VCAE), which only enforces a variance constraint on the latent distribution. Our experiments show that VCAE improves upon Wasserstein Autoencoder and the Variational Autoencoder in both reconstruction and generative quality on MNIST and CelebA. Moreover, we show that VCAE equipped with a total correlation penalty term performs equivalently to FactorVAE at learning disentangled representations on 3D-Shapes while being a more principled approach.