Formal Limitations on the Measurement of Mutual Information

TL;DR

This paper proves fundamental statistical limitations on measuring mutual information from finite samples, showing that any distribution-free high-confidence lower bound cannot grow faster than logarithmically with the number of samples.

Contribution

It establishes a theoretical lower bound on the accuracy of mutual information estimation methods, highlighting inherent limitations regardless of the approach used.

Findings

Any distribution-free high-confidence lower bound is at most O(ln N)

Mutual information measurement from finite data has fundamental statistical constraints

Variational methods cannot surpass these inherent limitations

Abstract

Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confidence lower bound on mutual information estimated from N samples cannot be larger than O(ln N ).