On the use of Mutual Information for Testing Independence

TL;DR

This paper develops an asymptotic distribution for Mutual Information using the delta-method, enabling a new independence test with connections to classical statistical tests, and explores geometric and alternative measures.

Contribution

It introduces a delta-method-based framework for asymptotic analysis of Mutual Information and related measures, linking them to classical tests and exploring geometric interpretations.

Findings

The asymptotic distribution of Mutual Information is derived as a combination of chi-square variables.

The proposed test is asymptotically equivalent to a linear combination of chi-squares under independence.

Alternative statistical measures and geometric insights are proposed for testing independence.

Abstract

In this paper we use a well know method in statistics, the $\delta$-method, to provide an asymptotic distribution for the Mutual Information, and construct and independence test based on it. Interesting connections are found with the likelihood ratio test and the chi-square goodness of fit test. In general, the difference between the Mutual Information evaluated at the true probabilities and at the empirical distribution, can be approximated by the sum of a normal random variable and a linear combination of chi-squares random variables. This summands are not independent, however the normal terms vanishes when testing independence, making the test statistic being asymptotically a linear combination of chi-squares. The $\delta$-method gives a general framework for computing the asymptotic distribution of other information based measures. A common difficulty is calculating the first and second-order derivatives, which is already challenging in the case of Mutual Information. However, this difficulty can be circumvallated by using advance symbolic software such as Mathematica. Finally, we explore the underlying geometry of the Mutual Information and propose other statical measures which may give competing alternatives to classical tests.