A new estimate of mutual information based measure of dependence between two variables: properties and fast implementation

TL;DR

This paper introduces MIDI, a new mutual information-based dependence measure that uses a novel histogram binning method, offering improved accuracy, efficiency, and ability to detect various relationships without assuming functional forms.

Contribution

The paper presents a new dependence measure called MIDI, with a novel histogram bin length selection method, demonstrating superior performance over existing measures like dcor and MINE.

Findings

MIDI effectively detects various types of relationships, including nonlinear.

MIDI outperforms dcor and MINE in accuracy and computational efficiency.

MIDI works well on real-world data and large datasets.

Abstract

This article proposes a new method to estimate an existing mutual information based dependence measure using histogram density estimates. Finding a suitable bin length for histogram is an open problem. We propose a new way of computing the bin length for histogram using a function of maximum separation between points. The chosen bin length leads to consistent density estimates for histogram method. The values of density thus obtained are used to calculate an estimate of an existing dependence measure. The proposed estimate is named as Mutual Information Based Dependence Index (MIDI). Some important properties of MIDI have also been stated. The performance of the proposed method has been compared to generally accepted measures like Distance Correlation (dcor), Maximal Information Coefficient (MINE) in terms of accuracy and computational complexity with the help of several artificial data sets with different amounts of noise. The proposed method is able to detect many types of relationships between variables, without making any assumption about the functional form of the relationship. The power statistics of proposed method illustrate their effectiveness in detecting non linear relationship. Thus, it is able to achieve generality without a high rate of false positive cases. MIDI is found to work better on a real life data set than competing methods. The proposed method is found to overcome some of the limitations which occur with dcor and MINE. Computationally, MIDI is found to be better than dcor and MINE, in terms of time and memory, making it suitable for large data sets.