Information-Theoretic Measures on Lattices for High-Order Interactions

TL;DR

This paper introduces a lattice-theoretic framework for higher-order information measures in multivariate data, addressing computational and asymmetry issues of existing methods, and proposes Streitberg Information to fully characterize complex interactions.

Contribution

It develops a systematic lattice-based approach to derive comprehensive high-order information measures, including the novel Streitberg Information for complete interaction characterization.

Findings

Streitberg Information captures all interactions among variables.

Framework unifies existing measures within a lattice-theoretic context.

Validated on synthetic data and real-world applications like finance and neural decoding.

Abstract

Traditional measures based solely on pairwise associations often fail to capture the complex statistical structure of multivariate data. Existing approaches for identifying information shared among $d>3$ variables are frequently computationally intractable, asymmetric with respect to a target variable, or unable to account for all the ways in which the joint probability distribution can be factorised. Here we present a systematic framework based on lattice theory to derive higher-order information-theoretic measures for multivariate data. Our construction uses lattice and operator function pairs, whereby an operator function is applied over a lattice that represents the algebraic relationships among variables. We show that many commonly used measures can be derived within this framework, yet they fail to capture all interactions for $d>3$, either because they are defined on restricted sublattices, or because the use of the KL divergence as an operator function, a typical choice, leads to undesired disregard of groups of interactions. To fully characterise all interactions among $d$ variables, we introduce the Streitberg Information, which is defined over the full partition lattice and uses generalised divergences (beyond KL) as operator functions. We validate the Streitberg Information on synthetic data, and illustrate its application in detecting complex interactions among stocks, decoding neural signals, and performing feature selection in machine learning.