DEMI: Discriminative Estimator of Mutual Information

TL;DR

DEMI introduces a classifier-based mutual information estimator that overcomes the limitations of variational methods, providing accurate, low-variance estimates especially useful in high-dimensional data and representation learning.

Contribution

The paper presents a novel mutual information estimator based on classification probabilities, establishing a direct connection to mutual information and surpassing variational approaches in accuracy and stability.

Findings

High accuracy in mutual information estimation demonstrated.

Advantages shown in representation learning tasks.

Theoretical equivalence to variational methods at optimum.

Abstract

Estimating mutual information between continuous random variables is often intractable and extremely challenging for high-dimensional data. Recent progress has leveraged neural networks to optimize variational lower bounds on mutual information. Although showing promise for this difficult problem, the variational methods have been theoretically and empirically proven to have serious statistical limitations: 1) many methods struggle to produce accurate estimates when the underlying mutual information is either low or high; 2) the resulting estimators may suffer from high variance. Our approach is based on training a classifier that provides the probability that a data sample pair is drawn from the joint distribution rather than from the product of its marginal distributions. Moreover, we establish a direct connection between mutual information and the average log odds estimate produced by the classifier on a test set, leading to a simple and accurate estimator of mutual information. We show theoretically that our method and other variational approaches are equivalent when they achieve their optimum, while our method sidesteps the variational bound. Empirical results demonstrate high accuracy of our approach and the advantages of our estimator in the context of representation learning. Our demo is available at https://github.com/RayRuizhiLiao/demi_mi_estimator.