Multivariate information measures: a unification using M\"obius operators on subset lattices

TL;DR

This paper introduces a unifying mathematical framework using M"obius operators on subset lattices to relate and analyze various multivariate information measures, revealing their symmetries and interconnections.

Contribution

It defines M"obius operators on lattice functions, forming a group isomorphic to S3, to systematically unify and derive relationships among multivariate information measures.

Findings

Operators form a group isomorphic to S3

Derived new relationships among information measures

Connected sums of conditional log-likelihoods with dependency measures

Abstract

Information related measures are useful tools for multi variable data analysis, as measures of dependence among variables, and as descriptions of order in biological and physical systems. Information related measures, like marginal entropies, mutual / interaction / multi-information, have been used in a number of fields including descriptions of systems complexity and biological data analysis. The mathematical relationships among these measures are therefore of significant interest. Relations between common information measures include the duality relations based on M\"obius inversion on lattices. These are the direct consequence of the symmetries of the lattices of the sets of variables (subsets ordered by inclusion). While the mathematical properties and relationships among these information-related measures are of significant interest, there has been, to our knowledge, no systematic examination of the full range of relationships and no unification of this diverse range of functions into a single formalism as we do here. In this paper we define operators on functions on these lattices based on the M\"obius inversion idea that map the functions into one another (M\"obius operators.) We show that these operators form a simple group isomorphic to the symmetric group S3. Relations among the set of functions on the lattice are transparently expressed in terms of the operator algebra, and, applied to the information measures, can be used to derive a wide range of relationships among measures. We describe a direct relation between sums of conditional log-likelihoods and previously defined dependency measures. The algebra is naturally generalized which yields more extensive relationships. This formalism provides a fundamental unification of information related measures, but isomorphism of all distributive lattices with the subset lattice implies broad potential application of these results.