Learning deep representations by mutual information estimation and maximization

TL;DR

This paper introduces Deep InfoMax (DIM), an unsupervised method that maximizes mutual information between input and neural network output, improving representation learning for downstream tasks by incorporating locality and adversarial prior matching.

Contribution

The paper presents a novel mutual information-based approach, Deep InfoMax, that enhances unsupervised representation learning by leveraging input locality and adversarial prior matching.

Findings

DIM outperforms popular unsupervised methods.

DIM competes with fully-supervised learning on classification.

Incorporating locality improves representation quality.

Abstract

In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality of the input to the objective can greatly influence a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and competes with fully-supervised learning on several classification tasks. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation-learning objectives for specific end-goals.