Understanding interdependency through complex information sharing

TL;DR

This paper introduces a new axiomatic framework to analyze complex interactions among multiple variables by decomposing joint entropy into redundant and synergistic information sharing modes, aiding understanding in various scientific fields.

Contribution

It proposes a novel axiomatic approach for entropy decomposition that distinguishes between redundant and synergistic information sharing, with unique formulas for key cases.

Findings

Framework effectively decomposes joint entropy into sharing modes.

Unique formulas derived for key cases of information sharing.

Applications clarify fundamental limits in network information theory.

Abstract

The interactions between three or more random variables are often nontrivial, poorly understood, and yet, are paramount for future advances in fields such as network information theory, neuroscience, genetics and many others. In this work, we propose to analyze these interactions as different modes of information sharing. Towards this end, we introduce a novel axiomatic framework for decomposing the joint entropy, which characterizes the various ways in which random variables can share information. The key contribution of our framework is to distinguish between interdependencies where the information is shared redundantly, and synergistic interdependencies where the sharing structure exists in the whole but not between the parts. We show that our axioms determine unique formulas for all the terms of the proposed decomposition for a number of cases of interest. Moreover, we show how these results can be applied to several network information theory problems, providing a more intuitive understanding of their fundamental limits.