Disentangled Representation Learning and Generation with Manifold Optimization

TL;DR

This paper introduces a novel representation learning framework that explicitly promotes disentanglement by encouraging orthogonal variations, improving both interpretability and generation quality in latent space models.

Contribution

It proposes a new objective combining autoencoder and PCA errors, interpreted as a Restricted Kernel Machine on the Stiefel manifold, with an optimization scheme for enhanced disentanglement.

Findings

Improves disentanglement in latent representations.

Enhances generation quality over existing VAE variants.

Theoretically justified and empirically validated.

Abstract

Disentanglement is a useful property in representation learning which increases the interpretability of generative models such as Variational autoencoders (VAE), Generative Adversarial Models, and their many variants. Typically in such models, an increase in disentanglement performance is traded-off with generation quality. In the context of latent space models, this work presents a representation learning framework that explicitly promotes disentanglement by encouraging orthogonal directions of variations. The proposed objective is the sum of an autoencoder error term along with a Principal Component Analysis reconstruction error in the feature space. This has an interpretation of a Restricted Kernel Machine with the eigenvector matrix-valued on the Stiefel manifold. Our analysis shows that such a construction promotes disentanglement by matching the principal directions in the latent space with the directions of orthogonal variation in data space. In an alternating minimization scheme, we use Cayley ADAM algorithm - a stochastic optimization method on the Stiefel manifold along with the ADAM optimizer. Our theoretical discussion and various experiments show that the proposed model improves over many VAE variants in terms of both generation quality and disentangled representation learning.