The many faces of multivariate information

TL;DR

This paper unifies various multivariate information measures into a single hierarchical framework using a parameterized function, revealing insights into higher-order interactions, synergies, and redundancies in complex systems.

Contribution

It introduces the $ ext{Delta}^k$ function that generalizes existing measures, providing a unified hierarchy of higher-order interactions and clarifying their roles.

Findings

$ ext{Delta}^k$ recovers known measures for specific $k$ values.

Positive $ ext{Delta}^k$ indicates dominance of higher-order interactions.

Negative $ ext{Delta}^k$ indicates dominance of lower-order interactions.

Abstract

Extracting higher-order structures from multivariate data has become an area of intensive study in complex systems science, as these multipartite interactions can reveal insights into fundamental features of complex systems like emergent phenomena. Information theory provides a natural language for exploring these interactions, as it elegantly formalizes the problem of comparing ``wholes" and ``parts" using joint, conditional, and marginal entropies. A large number of distinct statistics have been developed over the years, all aiming to capture different aspects of ``higher-order" information sharing. Here, we show that three of them (the dual total correlation, S-information, and O-information) are special cases of a more general function, $\Delta^{k}$ which is parameterized by a free parameter $k$. For different values of $k$, we recover different measures: $\Delta^{0}$ is equal to the S-information, $\Delta^{1}$ is equal to the dual total correlation, and $\Delta^{2}$ is equal to the negative O-information. Generally, the $\Delta^{k}$ function is arranged into a hierarchy of increasingly high-order synergies; for a given value of $k$, if $\Delta^{k}>0$, then the system is dominated by interactions with order greater than $k$, while if $\Delta^{k}<0$, then the system is dominated by interactions with order lower than $k$. $\Delta^{k}=0$ if the system is composed entirely of synergies of order-k. Using the entropic conjugation framework, we also find that the conjugate of $\Delta^{k}$, which we term $\Gamma^{k}$ is arranged into a similar hierarchy of increasingly high-order redundancies. These results provide new insights into the nature of both higher-order redundant and synergistic interactions, and helps unify the existing zoo of measures into a more coherent structure.