High Mutual Information in Representation Learning with Symmetric Variational Inference

TL;DR

This paper introduces the Mutual Information Machine (MIM), a new representation learning framework that emphasizes symmetry and mutual information to learn useful, high mutual information representations without posterior collapse.

Contribution

The paper presents MIM, a novel symmetric variational inference approach that bounds mutual information with a tractable loss, extending and relating to VAEs.

Findings

MIM learns symmetric factorizations with high mutual information.

MIM avoids posterior collapse in representation learning.

MIM relates to and extends variational autoencoders.

Abstract

We introduce the Mutual Information Machine (MIM), a novel formulation of representation learning, using a joint distribution over the observations and latent state in an encoder/decoder framework. Our key principles are symmetry and mutual information, where symmetry encourages the encoder and decoder to learn different factorizations of the same underlying distribution, and mutual information, to encourage the learning of useful representations for downstream tasks. Our starting point is the symmetric Jensen-Shannon divergence between the encoding and decoding joint distributions, plus a mutual information encouraging regularizer. We show that this can be bounded by a tractable cross entropy loss function between the true model and a parameterized approximation, and relate this to the maximum likelihood framework. We also relate MIM to variational autoencoders (VAEs) and demonstrate that MIM is capable of learning symmetric factorizations, with high mutual information that avoids posterior collapse.