Redundancy and synergy in dual decompositions of mutual information gain and information loss

TL;DR

This paper extends the framework of mutual information decomposition by generalizing lattices, analyzing redundancy and synergy invariance, and proposing dual decompositions to better characterize information components.

Contribution

It introduces generalized lattice structures for information decomposition, explores their relations, and proposes dual decompositions to address asymmetry issues.

Findings

Redundancy components are invariant across decompositions.

Unique and synergy components depend on the specific decomposition.

Dual decompositions help in jointly characterizing synergy and redundancy.

Abstract

Williams and Beer (2010) proposed a nonnegative mutual information decomposition, based on the construction of information gain lattices, which allows separating the information that a set of variables contains about another into components interpretable as the unique information of one variable, or redundant and synergy components. In this work we extend the framework of Williams and Beer (2010) focusing on the lattices that underpin the decomposition. We generalize the type of constructible lattices and examine the relations between the terms in different lattices, for example relating bivariate and trivariate decompositions. We point out that, in information gain lattices, redundancy components are invariant across decompositions, but unique and synergy components are decomposition-dependent. Exploiting the connection between different lattices we propose a procedure to construct, in the general multivariate case, information decompositions from measures of synergy or unique information. We introduce an alternative type of mutual information decompositions based on information loss lattices, with the role and invariance properties of redundancy and synergy components exchanged with respect to gain lattices. We study the correspondence between information gain and information loss lattices and we define dual decompositions that allow overcoming the intrinsic asymmetry between invariant and decomposition-dependent components, which hinders the consistent joint characterization of synergy and redundancy.