Total/dual correlation/coherence, redundancy/synergy, complexity, and O-information for real and complex valued multivariate data

TL;DR

This paper derives formulas for multivariate information measures assuming Gaussianity, introduces structured group generalizations, and provides a framework for analyzing variable contributions and synergy in complex systems.

Contribution

It presents new formulas for information measures under Gaussian assumptions, generalizes these measures for structured variable groups, and offers a framework for quantifying variable contributions and synergy.

Findings

Formulas for TC, DTC, O-information, TSE complexity, and RSI derived for Gaussian data.

Structured O-information captures between-group synergy, unlike standard measures.

Framework for quantifying the impact of variable connections on information measures.

Abstract

Firstly, assuming Gaussianity, equations for the following information theory measures are presented: total correlation/coherence (TC), dual total correlation/coherence (DTC), O-information, TSE complexity, and redundancy-synergy index (RSI). Since these measures are functions of the covariance matrix "S" and its inverse "S^-1", the associated Wishart and inverse-Wishart distributions are of note. DTC is shown to be the Kullback-Leibler (KL) divergence for the inverse-Wishart pair "(S^-1)" and its diagonal matrix "D=diag(S^-1)", shedding light on its interpretation as a measure of "total partial correlation", -lndetP, with test hypothesis H0: P=I, where "P" is the standardized inverse covariance (i.e. P=(D^-1/2)(S^-1)(D^-1/2). The second aim of this paper introduces a generalization of all these measures for structured groups of variables. For instance, consider three or more groups, each consisting of three or more variables, with predominant redundancy within each group, but with synergistic interactions between groups. O-information will miss the between group synergy (since redundancy occurs more often in the system). In contrast, the structured O-information measure presented here will correctly report predominant synergy between groups. This is a relevant generalization towards structured multivariate information measures. A third aim is the presentation of a framework for quantifying the contribution of "connections" between variables, to the system's TC, DTC, O-information, and TSE complexity. A fourth aim is to present a generalization of the redundancy-synergy index for quantifying the contribution of a group of variables to the system's redundancy-synergy balance. Finally, it is shown that the expressions derived here directly apply to data from several other elliptical distributions. All program codes, data files, and executables are available (https://osf.io/jd37g/).