Efficient Estimation of Mutual Information for Strongly Dependent Variables

TL;DR

This paper introduces a new mutual information estimator that overcomes the limitations of existing methods by accurately estimating MI between strongly dependent variables with limited data, addressing a key challenge in information theory applications.

Contribution

A novel mutual information estimator that is robust to local non-uniformity and effective with small sample sizes, improving estimation accuracy for strongly dependent variables.

Findings

Outperforms existing estimators on synthetic data

Effective in real-world data scenarios

Capable of capturing a wide range of relationship strengths

Abstract

We demonstrate that a popular class of nonparametric mutual information (MI) estimators based on k-nearest-neighbor graphs requires number of samples that scales exponentially with the true MI. Consequently, accurate estimation of MI between two strongly dependent variables is possible only for prohibitively large sample size. This important yet overlooked shortcoming of the existing estimators is due to their implicit reliance on local uniformity of the underlying joint distribution. We introduce a new estimator that is robust to local non-uniformity, works well with limited data, and is able to capture relationship strengths over many orders of magnitude. We demonstrate the superior performance of the proposed estimator on both synthetic and real-world data.