Synergy, suppression and immorality: forward differences of the entropy function

TL;DR

This paper explores the forward differences of the entropy function to understand complex interactions like synergy and suppression among variables, linking concepts in graphical models, regression, and Bayesian networks.

Contribution

It introduces a novel entropy difference expansion that unifies and generalizes concepts like synergy, suppression, and immorality across various statistical models.

Findings

Synergies are infrequent but reveal complex variable interactions.

New characterizations of suppressor variables and unshielded colliders.

Algorithm for computing entropy differences from graphical models.

Abstract

Conditional mutual information is important in the selection and interpretation of graphical models. Its empirical version is well known as a generalised likelihood ratio test and that it may be represented as a difference in entropy. We consider the forward difference expansion of the entropy function defined on all subsets of the variables under study. The elements of this expansion are invariant to permutation of their suffices and relate higher order mutual informations to lower order ones. The third order difference is expressible as an, apparently assymmetric, difference between a marginal and a conditional mutual information. Its role in the decomposition for explained information provides a technical definition for synergy between three random variables. Positive values occur when two variables provide alternative explanations for a third; negative values, termed synergies, occur when the sum of explained information is greater than the sum of its parts. Synergies tend to be infrequent; they connect the seemingly unrelated concepts of suppressor variables in regression, on the one hand, and unshielded colliders in Bayes networks (immoralities), on the other. We give novel characterizations of these phenomena that generalise to categorical variables and to higher dimensions. We propose an algorithm for systematically computing low order differences from a given graph. Examples from small scale real-life studies indicate the potential of these techniques for empirical statistical analysis.