Pooling Probability Distributions and the Partial Information Decomposition

TL;DR

This paper explores how to define and quantify synergistic, redundant, and unique information in multivariate systems by proposing a pooling-based approach to probability distributions, highlighting ambiguity and introducing a new lattice structure.

Contribution

It introduces a pooling-based framework for partial information decomposition, offering a new perspective on defining synergy and ambiguity in information measures.

Findings

Pooling leads to a new lattice structure different from redundancy-based models.

Overlap between probability distributions characterizes synergistic and unique information.

A simple pooling method illustrates the role of distribution overlap in information decomposition.

Abstract

Notwithstanding various attempts to construct a Partial Information Decomposition (PID) for multiple variables by defining synergistic, redundant, and unique information, there is no consensus on how one ought to precisely define either of these quantities. One aim here is to illustrate how that ambiguity -- or, more positively, freedom of choice -- may arise. Using the basic idea that information equals the average reduction in uncertainty when going from an initial to a final probability distribution, synergistic information will likewise be defined as a difference between two entropies. One term is uncontroversial and characterizes ``the whole'' information that source variables carry jointly about a target variable $\varT$. The other term then is meant to characterize the information carried by the ``sum of its parts.'' Here we interpret that concept as needing a suitable probability distribution aggregated (``pooled'') from multiple marginal distributions (the parts). Ambiguity arises in the definition of the optimum way to pool two (or more) probability distributions. Independent of the exact definition of optimum pooling, the concept of pooling leads to a lattice that differs from the often-used redundancy-based lattice. One can associate not just a number (an average entropy) with each node of the lattice, but (pooled) probability distributions. As an example,one simple and reasonable approach to pooling is presented, which naturally gives rise to the overlap between different probability distributions as being a crucial quantity that characterizes both synergistic and unique information.