Some new insights into information decomposition in complex systems based on common information

TL;DR

This paper critically examines existing measures of redundancy in information decomposition, revealing limitations of Gács-Körner and Wyner's common information, and proposes conditions and methods to better quantify unique information in complex systems.

Contribution

It identifies the degeneracy of Gács-Körner and Wyner's common information measures and proposes conditions and approaches for more accurate quantification of unique information.

Findings

Gács-Körner common information is degenerate for most non-trivial distributions.

Wyner's common information violates monotonicity in redundancy measures.

Conditional Gács-Körner information can serve as an ideal measure under certain conditions.

Abstract

We take a closer look at the structure of bivariate dependency induced by a pair of predictor random variables $(X_1, X_2)$ trying to synergistically, redundantly or uniquely encode a target random variable $Y$. We evaluate a recently proposed measure of redundancy based on the G\'acs-K\"{o}rner common information (Griffith et al., Entropy 2014) and show that the measure, in spite of its elegance is degenerate for most non-trivial distributions. We show that Wyner's common information also fails to capture the notion of redundancy as it violates an intuitive monotonically non-increasing property. We identify a set of conditions when a conditional version of G\'acs and K\"{o}rner's common information is an ideal measure of unique information. Finally, we show how the notions of approximately sufficient statistics and conditional information bottleneck can be used to quantify unique information.