Maximizing Multi-Information

TL;DR

This paper studies the structure of probability distributions that maximize multi-information, showing that pure pair-interactions form an exponential family containing all global maximizers in their closure.

Contribution

It characterizes the set of distributions that maximize multi-information, revealing that pure pair-interactions encompass all global maximizers within a specific exponential family.

Findings

Pure pair-interactions contain all global maximizers of multi-information.

The structure of distributions with maximal multi-information is explicitly described.

Maximizers are contained in the closure of the exponential family of pure pair-interactions.

Abstract

Stochastic interdependence of a probablility distribution on a product space is measured by its Kullback-Leibler distance from the exponential family of product distributions (called multi-information). Here we investigate low-dimensional exponential families that contain the maximizers of stochastic interdependence in their closure. Based on a detailed description of the structure of probablility distributions with globally maximal multi-information we obtain our main result: The exponential family of pure pair-interactions contains all global maximizers of the multi-information in its closure.