On the Effect of Suboptimal Estimation of Mutual Information in Feature Selection and Classification

TL;DR

This paper introduces the estimator response curve, a new property for assessing mutual information estimators' performance, revealing that suboptimal estimators can significantly impair feature selection and classification tasks.

Contribution

The paper proposes the estimator response curve as a novel, distribution-agnostic metric to evaluate mutual information estimators, improving assessment over existing methods.

Findings

CIM estimator outperforms kNN, vME, AP, and H_{MI} in tests.

Suboptimal estimators lead to poorer real-world classification results.

Estimator response curve effectively identifies estimator performance issues.

Abstract

This paper introduces a new property of estimators of the strength of statistical association, which helps characterize how well an estimator will perform in scenarios where dependencies between continuous and discrete random variables need to be rank ordered. The new property, termed the estimator response curve, is easily computable and provides a marginal distribution agnostic way to assess an estimator's performance. It overcomes notable drawbacks of current metrics of assessment, including statistical power, bias, and consistency. We utilize the estimator response curve to test various measures of the strength of association that satisfy the data processing inequality (DPI), and show that the CIM estimator's performance compares favorably to kNN, vME, AP, and H_{MI} estimators of mutual information. The estimators which were identified to be suboptimal, according to the estimator response curve, perform worse than the more optimal estimators when tested with real-world data from four different areas of science, all with varying dimensionalities and sizes.