Multivariate Partial Information Decomposition: Constructions, Inconsistencies, and Alternative Measures

TL;DR

This paper reviews the PID framework for multivariate information decomposition, resolves the two-source case, proves fundamental inconsistencies for three or more sources, and proposes alternative measures that address these issues.

Contribution

It provides explicit formulas for two-source PID, proves impossibility results for higher sources, and introduces new measures avoiding lattice-based inconsistencies.

Findings

Explicit formulas for two-source PID information atoms.

Fundamental inconsistency proofs for three or more sources.

New measures of multivariate unique and synergistic information.

Abstract

While mutual information effectively quantifies dependence between two variables, it does not by itself reveal the complex, fine-grained interactions among variables, i.e., how multiple sources contribute redundantly, uniquely, or synergistically to a target in multivariate settings. The Partial Information Decomposition (PID) framework was introduced to address this by decomposing the mutual information between a set of source variables and a target variable into fine-grained information atoms such as redundant, unique, and synergistic components. In this work, we review the axiomatic system and desired properties of the PID framework and make three main contributions. First, we resolve the two-source PID case by providing explicit closed-form formulas for all information atoms that satisfy the full set of axioms and desirable properties. Second, we prove that for three or more sources, PID suffers from fundamental inconsistencies: we review the known three-variable counterexample where the sum of atoms exceeds the total information, and extend it to a comprehensive impossibility theorem showing that no lattice-based decomposition can be consistent for all subsets when the number of sources exceeds three. Finally, we deviate from the PID lattice approach to avoid its inconsistencies, and present explicit measures of multivariate unique and synergistic information. Our proposed measures, which rely on new systems of random variables that eliminate higher-order dependencies, satisfy key axioms such as additivity and continuity, provide a robust theoretical explanation of high-order relations, and show strong numerical performance in comprehensive experiments on the Ising model. Our findings highlight the need for a new framework for studying multivariate information decomposition.