Auto-Encoding Total Correlation Explanation

TL;DR

This paper introduces a new information-theoretic framework for understanding and improving unsupervised representation learning, specifically through a relaxed variational bound to total correlation, leading to more interpretable and realistic generative models.

Contribution

It relaxes the restrictive assumptions of the CorEx principle by introducing a flexible variational lower bound, connecting it to VAEs, and proposing AnchorVAE for enhanced interpretability and sample quality.

Findings

The variational lower bound to CorEx is equivalent to the VAE bound under certain conditions.

AnchorVAE improves interpretability of latent codes.

AnchorVAE generates richer, more realistic samples.

Abstract

Advances in unsupervised learning enable reconstruction and generation of samples from complex distributions, but this success is marred by the inscrutability of the representations learned. We propose an information-theoretic approach to characterizing disentanglement and dependence in representation learning using multivariate mutual information, also called total correlation. The principle of total Cor-relation Ex-planation (CorEx) has motivated successful unsupervised learning applications across a variety of domains, but under some restrictive assumptions. Here we relax those restrictions by introducing a flexible variational lower bound to CorEx. Surprisingly, we find that this lower bound is equivalent to the one in variational autoencoders (VAE) under certain conditions. This information-theoretic view of VAE deepens our understanding of hierarchical VAE and motivates a new algorithm, AnchorVAE, that makes latent codes more interpretable through information maximization and enables generation of richer and more realistic samples.