Logarithmic Asymptotic Relations Between $p$-Values and Mutual Information

TL;DR

This paper establishes a precise asymptotic relationship between p-values and mutual information, clarifying how MI governs the tail behavior of p-values in dependence testing and enabling unified meta-analyses.

Contribution

It derives a logarithmic asymptotic relation between p-values and mutual information, connecting information theory with statistical significance in dependence testing.

Findings

Fisher's exact test p-value satisfies a specific asymptotic relation with MI.

MI acts as the exponential rate in the tail probability of p-values.

Results facilitate comparison of dependence across datasets with different sample sizes.

Abstract

We establish a precise connection between statistical significance in dependence testing and information-theoretic dependence as quantified by Shannon mutual information (MI). In the absence of prior distributional information, we consider a maximum-entropy model and show that the probability associated with the realization of a given magnitude of MI takes an exponential form, yielding a corresponding tail-probability interpretation of a $p$-value. In contingency tables with fixed marginal frequencies, we analyze Fisher's exact test and prove that its $p$-value $P_F$ satisfies a logarithmic asymptotic relation of the form $MI=-(1/N)\log P_F + O(\log(N+1)/N)$ as the sample size $N\to\infty$. These results clarify the role of MI as the exponential rate governing the asymptotic behavior of $p$-values in the settings studied here, and they enable principled comparisons of dependence across datasets with different sample sizes. We further discuss implications for combining evidence across studies via meta-analysis, allowing mutual information and its statistical significance to be integrated in a unified framework.