Estimation and Confidence Intervals for Mutual Information: Issues in Convergence for Non-Normal Distributions

TL;DR

This paper investigates how different empirical estimators of mutual information perform with normal and non-normal data, highlighting issues in convergence, bias, and confidence interval accuracy, especially for non-normal distributions.

Contribution

It reveals the impact of nonlinear transformations on MI estimation and confidence intervals, and discusses strategies to improve estimation precision for non-normal data.

Findings

Confidence intervals are larger for non-normal data.

Convergence of confidence intervals is slower for non-normal samples.

Strong biases can lead to confidence intervals not containing the true MI value.

Abstract

By employing various empirical estimators for the Mutual Information (MI) measure, we calculate and compare the estimates and their confidence intervals for both normal and non-normal bivariate data samples. We find that certain nonlinear invertible transformations of the random variables can significantly affect both the estimated MI value and the precision and asymptotic behavior of its confidence intervals. Generally, for non-normal samples, the confidence intervals are larger than those for normal samples, and the convergence of the confidence intervals is slower even as the data sample size increases. In some cases, due to strong biases, the estimated confidence interval may not contain the true value at all. We discuss various strategies to improve the precision of the estimated Mutual Information.