Cross Mutual Information

TL;DR

This paper introduces cross mutual information, a new measure to compare the dependence between two variables across different sample sets, addressing limitations of traditional MI in non-stationary distributions.

Contribution

The paper proposes the cross mutual information measure and explores its properties through simulations, extending the application of MI to non-stationary data comparisons.

Findings

Cross mutual information effectively compares dependence across different sample sets.

Simulation results demonstrate the measure's robustness in various dependency scenarios.

Potential applications in neuroimaging and model fit assessment are discussed.

Abstract

Mutual information (MI) is a useful information-theoretic measure to quantify the statistical dependence between two random variables: $X$ and $Y$. Often, we are interested in understanding how the dependence between $X$ and $Y$ in one set of samples compares to another. Although the dependence between $X$ and $Y$ in each set of samples can be measured separately using MI, these estimates cannot be compared directly if they are based on samples from a non-stationary distribution. Here, we propose an alternative measure for characterising how the dependence between $X$ and $Y$ as defined by one set of samples is expressed in another, \textit{cross mutual information}. We present a comprehensive set of simulation studies sampling data with $X$-$Y$ dependencies to explore this measure. Finally, we discuss how this relates to measures of model fit in linear regression, and some future applications in neuroimaging data analysis.