Generalised Measures of Multivariate Information Content

TL;DR

This paper introduces new measures of multivariate information content that can be visualized with Venn diagrams regardless of the number of variables, addressing limitations of traditional entropy representations.

Contribution

It develops a novel algebraic framework for multivariate information measures, unifying Venn diagram visualization with the structure of information sharing among variables.

Findings

New measures accurately depict multivariate information with Venn diagrams.

The algebraic structure of information sharing is characterized by a free distributive lattice.

The redundancy lattice from partial information decomposition is derived from the algebraic framework.

Abstract

The entropy of a pair of random variables is commonly depicted using a Venn diagram. This representation is potentially misleading, however, since the multivariate mutual information can be negative. This paper presents new measures of multivariate information content that can be accurately depicted using Venn diagrams for any number of random variables. These measures complement the existing measures of multivariate mutual information and are constructed by considering the algebraic structure of information sharing. It is shown that the distinct ways in which a set of marginal observers can share their information with a non-observing third party corresponds to the elements of a free distributive lattice. The redundancy lattice from partial information decomposition is then subsequently and independently derived by combining the algebraic structures of joint and shared information content.