Mutual Information Multinomial Estimation

TL;DR

This paper introduces a novel mutual information estimator that leverages a preliminary data distribution estimate to improve accuracy, demonstrated through synthetic and real-world experiments.

Contribution

It presents a new mutual information estimator that uses a preliminary distribution estimate to effectively bridge joint and marginal distributions, enhancing estimation accuracy.

Findings

Effective in non-Gaussian synthetic problems with known ground-truth

Demonstrates advantages in real-world applications

Outperforms existing MI estimation methods

Abstract

Estimating mutual information (MI) is a fundamental yet challenging task in data science and machine learning. This work proposes a new estimator for mutual information. Our main discovery is that a preliminary estimate of the data distribution can dramatically help estimate. This preliminary estimate serves as a bridge between the joint and the marginal distribution, and by comparing with this bridge distribution we can easily obtain the true difference between the joint distributions and the marginal distributions. Experiments on diverse tasks including non-Gaussian synthetic problems with known ground-truth and real-world applications demonstrate the advantages of our method.