Curse of Slicing: Why Sliced Mutual Information is a Deceptive Measure of Statistical Dependence

TL;DR

Sliced Mutual Information, a popular scalable dependence measure, can be misleading and unreliable due to its susceptibility to data manipulation and counterintuitive behaviors, as shown through theoretical and empirical analysis.

Contribution

This paper reveals fundamental limitations of Sliced Mutual Information, demonstrating its vulnerabilities and counterproductive behaviors compared to traditional measures.

Findings

SMI saturates quickly and fails to detect increased dependence.

SMI can prioritize redundancy over informative content.

In some cases, SMI performs worse than correlation coefficient.

Abstract

Sliced Mutual Information (SMI) is widely used as a scalable alternative to mutual information for measuring non-linear statistical dependence. Despite its advantages, such as faster convergence, robustness to high dimensionality, and nullification only under statistical independence, we demonstrate that SMI is highly susceptible to data manipulation and exhibits counterintuitive behavior. Through extensive benchmarking and theoretical analysis, we show that SMI saturates easily, fails to detect increases in statistical dependence, prioritizes redundancy over informative content, and in some cases, performs worse than correlation coefficient.