Factorized Mutual Information Maximization

TL;DR

This paper explores the optimization of multi-information and mutual information across variable sets, proposing computationally efficient proxies and characterizing their maximizers to improve information-theoretic analysis.

Contribution

It introduces a novel framework for maximizing multi-information using factorized mutual information proxies and characterizes their maximizers relative to traditional measures.

Findings

Maximizers of the proposed functionals are characterized.

Proxies reduce computational complexity in mutual information estimation.

Relations between maximizers of proxies and original information measures are established.

Abstract

We investigate the sets of joint probability distributions that maximize the average multi-information over a collection of margins. These functionals serve as proxies for maximizing the multi-information of a set of variables or the mutual information of two subsets of variables, at a lower computation and estimation complexity. We describe the maximizers and their relations to the maximizers of the multi-information and the mutual information.

Equations

$$D(p\|q):=\sum_{x\in{\cal X}}p(x)\log{\bigg{(}\frac{p(x)}{q(x)}\bigg{)}}.$$

$$D(p\Vert \mathcal{C}):=\underset{q\in \mathcal{C}}{\inf }D(p\Vert q).$$

$$p(X_1,\dots ,X_n)=p_1(X_1)p_2(X_2)\cdots p_n(X_n),p_i\in {\Delta }_{{\mathcal{X}}_i},i=1,\dots ,n.$$

$$I_p(X_1,X_2,\dots ,X_n):=D(p(X_1,X_2,\dots ,X_n)\Vert \mathcal{F}).$$

$${\mathrm{arginf}}_{q\in \mathcal{F}}D(p\Vert q)=p(X_1)\cdots p(X_n),$$

$$I(X_1,X_2,\dots ,X_n)=\sum _{i=1}^{n}H(X_i)-H(X_1,\dots ,X_n).$$

$$I(X_1,X_2,\dots ,X_n)\le (n-1)\log (N).$$

$$H(X_1,\dots ,X_n)=\sum _{i=1}^{n}H(X_i|X_1,\dots ,X_{i-1}).$$

$$MI(\mathbf{X},\mathbf{Y}):=D(p(\mathbf{X},\mathbf{Y})\Vert p(\mathbf{X})p(\mathbf{Y})),$$

$$p=\frac{1}{2}\left({\delta }_{(0{\pi }_2(0){\pi }_3(0)\cdots {\pi }_n(0))}+{\delta }_{(1{\pi }_2(1){\pi }_3(1)\cdots {\pi }_n(1))}\right),$$

$$\frac{1}{2}({\delta }_{0000}+{\delta }_{1111}),\frac{1}{2}({\delta }_{0101}+{\delta }_{1010}),$$

$$\frac{1}{2}({\delta }_{0001}+{\delta }_{1110}),\frac{1}{2}({\delta }_{0100}+{\delta }_{1011}),$$

$$\frac{1}{2}({\delta }_{0010}+{\delta }_{1101}),\frac{1}{2}({\delta }_{0111}+{\delta }_{1000}),$$

$$\frac{1}{2}({\delta }_{0011}+{\delta }_{1100}),\frac{1}{2}({\delta }_{0110}+{\delta }_{1001}).$$

$$\frac{1}{4}({\delta }_{0010}+{\delta }_{0100}+{\delta }_{1001}+{\delta }_{1111}),$$

$$\frac{1}{4}({\delta }_{0001}+{\delta }_{0100}+{\delta }_{1010}+{\delta }_{1111}),$$

$$\frac{1}{4}({\delta }_{0001}+{\delta }_{0110}+{\delta }_{1000}+{\delta }_{1111}),$$

$$\frac{1}{4}({\delta }_{0000}+{\delta }_{0101}+{\delta }_{1011}+{\delta }_{1110}),$$

$$\frac{1}{4}({\delta }_{0000}+{\delta }_{0111}+{\delta }_{1001}+{\delta }_{1110}),$$

$$\frac{1}{4}({\delta }_{0011}+{\delta }_{0110}+{\delta }_{1000}+{\delta }_{1101}),$$

$$\frac{1}{4}({\delta }_{0000}+{\delta }_{0110}+{\delta }_{1011}+{\delta }_{1101}),$$

$$\frac{1}{4}({\delta }_{0000}+{\delta }_{0111}+{\delta }_{1010}+{\delta }_{1101}),$$

$$\frac{1}{4}({\delta }_{0011}+{\delta }_{0101}+{\delta }_{1000}+{\delta }_{1110}),$$

$$\frac{1}{4}({\delta }_{0001}+{\delta }_{0110}+{\delta }_{1011}+{\delta }_{1100}),$$

$$\frac{1}{4}({\delta }_{0001}+{\delta }_{0111}+{\delta }_{1010}+{\delta }_{1100}),$$

$$\frac{1}{4}({\delta }_{0010}+{\delta }_{0111}+{\delta }_{1001}+{\delta }_{1100}),$$

$$I_{\Lambda }(X_1,\dots ,X_n):=\frac{1}{|\Lambda |}\sum _{\lambda \in \Lambda }I((X_i{)}_{i\in \lambda }).$$

$$FMI(X_1,\dots ,X_n):=\frac{1}{\left(\genfrac{}{}{0ex}{}{n}{2}\right)}\sum _{i\ne j}MI(X_i,X_j).$$

$$SFMI(\mathbf{X},\mathbf{Y}):=\frac{1}{n}\sum _{i=1}^{n}MI(X_i,Y_i).$$

$$p(x_1,\dots ,x_{n-1})$$

$$=\sum _{x_n}p(x_1,\dots ,x_{n-1}|x_n)p(x_n)$$

$$=\frac{1}{|{\mathcal{X}}_n|}\sum _{x_n}p(x_1,\dots ,x_{n-1}|x_n),$$

$$p(X_1,X_2)=$$

$$p(X_2,X_3)=$$

$$p(X_1,X_2,X_3)=p_{000}{\delta }_{000}+p_{001}{\delta }_{001}+\cdots +p_{111}{\delta }_{111}.$$

$$\begin{array}{rl}& p_{000}+p_{001}=\frac{1}{2},p_{110}+p_{111}=\frac{1}{2},\\ & p_{010}+p_{011}=0,p_{100}+p_{101}=0.\end{array}$$

$$\begin{array}{rl}& p_{000}+p_{010}=\frac{1}{2},p_{101}+p_{111}=\frac{1}{2},\\ & p_{001}+p_{011}=0,p_{100}+p_{110}=0.\end{array}$$

$$F(x)=(F_{\lambda ,{\tilde{x}}_{\lambda }}(x){)}_{\lambda \in \Lambda ,{\tilde{x}}_{\lambda }\in {\mathcal{X}}_{\lambda }},x\in \mathcal{X},$$

$$\begin{array}{rl}& \frac{\alpha }{2}({\delta }_{0000}+{\delta }_{1111})+\frac{\beta }{2}({\delta }_{0101}+{\delta }_{1010}),\\ & \frac{\alpha }{2}({\delta }_{0001}+{\delta }_{1110})+\frac{\beta }{2}({\delta }_{0100}+{\delta }_{1011}),\\ & \frac{\alpha }{2}({\delta }_{0010}+{\delta }_{1101})+\frac{\beta }{2}({\delta }_{0111}+{\delta }_{1000}),\\ & \frac{\alpha }{2}({\delta }_{0011}+{\delta }_{1100})+\frac{\beta }{2}({\delta }_{0110}+{\delta }_{1001}).\end{array}$$

Full text

Keywords: Multi-information, mutual information, divergence maximization, marginal specification problem, transportation polytope.

1 Introduction

Mutual information (MI) is a measure of mutual dependence between two random variables that plays a central role in information theory and machine learning. The multi-information is a generalization to composite systems with an arbitrary number of random variables. The problem of maximizing the multi-information was proposed and studied in [2], motivated by infomax principles. It was shown, in the setting of finite valued random variables, that the optimizers of the unconstrained maximization problem are attained within low dimensional exponential families of joint probability distributions. A characterization of the optimizers was obtained in [4]. Maximizing the multi-information can be regarded as a special instance of the more general problem of maximizing a divergence from an exponential family. Indeed, the multi-information of a joint probability distribution is equal to its Kullback-Leibler divergence from an independence model. The problem of maximizing the divergence from an exponential family has been advanced by Matúš [24, 25], Matúš and Ay [23], Matúš and Rauh [22], Rauh [34]. The divergence maximization problem can also be asked for more general classes of probability models, such as probabilistic graphical models with hidden variables and stochastic neural networks. Works in this direction include [31, 28] and the short overview [32].

Numerous machine learning applications involve optimizing the MI over a restricted set of joint probability distributions. Examples include the information bottleneck methods [45, 42, 1], the computation of positive information decompositions [8], the implementation of information theoretic regularizers in robotics and reinforcement learning [47, 30], the analysis of deep networks [15], and unsupervised representation learning in deep learning [17]. A parametrized set of distributions may or may not contain the unconstrained maximizers of a given functional. In some cases, the distributions are constrained by structures in addition to the chosen parametrization. For instance, in Markov decision processes the joint distributions between time consecutive states are governed not only by a parametrized policy model but also by the state transition mechanism; see, e.g., [29]. In order to optimize the mutual information in such cases, we usually need to resort to iterative parameter optimization techniques. Computing the parameter updates can quickly become intractable, even for a moderate number of variables, since the number of possible joint states grows exponentially in the number of variables. The estimation and computation of the mutual information is often a bottleneck in the implementation of algorithms that are based on it. For reference, the estimation of the MI from observations has been studied in [36, 21, 38, 16], and recently also using neural networks in [7].

We are interested in proxies that are easier to estimate from samples and easier to compute than a given functional, and which might provide a signal for optimizing it, possibly having the same optimizers, or a structured subset of the optimizers. We consider measures defined as averages of the multi-information over subsets of all random variables in a composite system. Based on previous investigations of the maximizers of multi-information [4, 2], we can expect that the maximizers of these measures will satisfy several properties. First, they may be contained within the closure of a low dimensional exponential family. Second, like many other instances of divergence maximizing distributions, they may exhibit reduced support. Last, if the measures are symmetric under the exchange of indices, the maximizing distributions are expected to display the same degree of symmetry. We characterize the sets of maximizers of various factorized measures and describe their relation to the maximizers of the multi-information and the maximizers of the mutual information of two subsets of variables.

As a motivating application, we have in mind the estimation and maximization of the mutual information between time consecutive sensor readings of a reinforcement learning agent acting in an environment. The mutual information is the $1$ time step version of a notion called predictive information, which has been considered in the literature [9, 37, 3, 48, 12]. In [30], a factorized mutual information (here called SFMI) was used in place of the mutual information as an intrinsic reward signal to encourage more robust walking behaviors for a multi-legged robot. A similar approach was also used previously in [47] for a chain of robots. Maximizing the mutual information between time consecutive measurements of a given variable should encourage behaviors that are diverse and, at the same time, predictable. This intuition comes from writing the mutual information between $X,Y$ as $I(X,Y)=H(X)-H(X|Y)$, where $H$ is the entropy. In this context, the mutual information serves as a regularizer that is added in order to ease the task-objective optimization problem and also to assign preferences to specific types of solutions. Some references to such regularizes, also known as intrinsic motivation, include [5, 27, 40, 11, 20, 9, 10].

This article is organized as follows. In Section 2 we review definitions and in Section 3 relevant existing results around the multi-information. In Section 4 we define notions of factorized multi and mutual information. The FMI of a set of random variables is defined as an average multi-information of subsets of variables. The SFMI of two random vectors is defined as an average mutual information of non-overlapping pairs with one variable from each vector. In Section 5 we study the maximizers of the FMI and characterize the cases when they coincide with the maximizers of the multi-information. In Section 6 we characterize the maximizers of the SFMI, which build a union of transportation polytopes that contain all maximizers of the multi-information and some maximizers of the mutual information. Section 7 describes partitions of sets of strings that we use for the characterization of SFMI maximizers. In Section 8 we offer a summary and discussion of our results.

2 Multi-information and mutual information

We consider discrete probability distributions supported on a finite set ${\cal X}$. The set of all such distributions is a $(|{\cal X}|-1)$-simplex denoted $\Delta_{\cal X}$. Let $D(p\|q)$ be the Kullback-Leibler divergence between the probability distributions $p$ and $q$, which is defined as

[TABLE]

If ${\cal C}$ is a family of probability distributions, then let $D(p\|{\cal C})$ denote the infimum divergence between $p$ and any distribution in ${\cal C}$,

[TABLE]

We are concerned with joint probability distributions of $n$ variables, so that $x=(x_{1},\ldots,x_{n})$ and ${\cal X}={\cal X}_{1}\times\cdots\times{\cal X}_{n}$. We denote ${\cal F}$ the set of fully factorizable joint distributions, i.e., those $p\in\Delta_{\cal X}$ which can be written as

[TABLE]

The set $\mathcal{F}$ is also referred to as the independence model of $X_{1},\ldots,X_{n}$. It is a $\sum_{i=1}^{n}(|{\cal X}_{i}|-1)$-dimensional manifold in $\Delta_{\cal X}$. The multi-information of $n$ random variables with joint distribution $p$ is then defined as

[TABLE]

It has a natural interpretation as the distance of $p$ from being independent. Following standard notation we drop the subscript $p$, and write simply $I(X_{1},\ldots,X_{n})$. The minimum Kullback-Leibler divergence from a joint distribution $p$ to the set of factorizable distributions is attained uniquely by the product of its marginals,

[TABLE]

where $p(X_{i}=x_{i})=\sum_{x_{j}\colon j\neq i}p(x_{1},\ldots,x_{n})$, $x_{i}\in\mathcal{X}_{i}$, for $i=1,\ldots,n$. The product of marginals is the maximum likelihood estimate of a joint distribution as a product distribution.

The multi-information can be written equivalently in terms of entropies, as

[TABLE]

Here $H(X)=-\sum_{x}p(x)\log{p(x)}$ denotes the entropy of $X$. If $|{\cal X}|=N$, then direct computation shows that the entropy is bounded above by $\log{(N)}$. Therefore, the maximum value of multi-information of $n$ $N$-ary variables is bounded by $n\log{(N)}$. However, as noted in [4, Lemma 4.1], this bound is never attained, with a sharp bound being

[TABLE]

The set of all distributions which maximize multi-information for $n$ variables will be denoted by ${\cal M}_{n},$ or ${\cal M}$ when $n$ is understood. We note the chain rule of entropy,

[TABLE]

From this, we see that maximizing the multi-information (6) can be interpreted as maximizing the marginal entropies while at the same time minimizing the conditional entropies. In turn, the maximizers of the multi-information are joint distributions where each individual variable is diverse (high entropy), but predictable from the values of the other variables (low conditional entropy).

The multi-information of two variables is called mutual information. We will consider the mutual information of two random vectors $\mathbf{X}=(X_{1},\ldots,X_{n})$ and $\mathbf{Y}=(Y_{1},\ldots,Y_{n})$, which is given by

[TABLE]

where $p(\mathbf{X}=\mathbf{x})=\sum_{\mathbf{y}}p(\mathbf{x},\mathbf{y})$ and $p(\mathbf{Y}=\mathbf{y})=\sum_{\mathbf{x}}p(\mathbf{x},\mathbf{y})$ are the marginal distributions. Note that here the divergence is measured to the set of distributions that factorize as a product $p(\mathbf{X})p(\mathbf{Y})$, which does not need to factorize further over the components of each random vector.

3 Maximizers of the multi-information

In this section we collect relevant existing results on the multi-information that we will use in the following sections of this article.

Proposition 1** (Maximizers of the multi-information, [4, Corollary 4.10]).**

Consider $N$-ary random variables $X_{1},\ldots,X_{n}$. Then $\max_{p\in\Delta_{\mathcal{X}}}I_{p}(X_{1},\ldots,X_{n})=(n-1)\log(N)$, and the maximizers are the uniform distributions on $N$-ary codes of length $n$, minimum Hamming distance $n$, and cardinality $N$. The number of such codes is $(N!)^{n-1}$.

The work [4] also characterizes the maximizers of the multi-information for variables with different state spaces, which is more technical. We will focus on the case where all variables are $N$-ary.

Example 2**.**

In the case of binary variables, $N=2$, Proposition 1 states that the set of maximizers of the multi-information consists of all uniform distributions over pairs of binary vectors of Hamming distance $n$. Letting $\pi_{i}:\{0,1\}\to\{0,1\}$ be one-to-one maps for $i=2,3,\dots,n$, the maximizers of the multi-information take the form

[TABLE]

where $\delta_{x}$ denotes the point measure supported on string $x$. ∎

The maximizers of the mutual information are obtained from applying Proposition 1 to the special case of two random variables.

Corollary 3**.**

For two $N$-ary variables $X$ and $Y$, the maximum mutual information is $\max_{p\in\Delta_{\mathcal{X}}}MI_{p}(X,Y)=\log(N)$, and the maximizers are the uniform distributions on $N$-ary codes of length $2$, minimum Hamming distance $2$, and cardinality $N$. The number of such codes is $N!$.

We can apply Corollary 3 not only to pairs of variables, but also to pairs of random vectors, since random vectors can be regarded as random variables with states over a product space.

Corollary 4**.**

For two $N$-ary random vectors $\mathbf{X}=(X_{1},\ldots,X_{n})$ and $\mathbf{Y}=(Y_{1},\ldots,Y_{n})$, the maximizers of $MI(\mathbf{X},\mathbf{Y})$ are the uniform distributions over $N$-ary codes of length $2n$, minimum distance $2$ when viewed as $N^{n}$-ary codes of length $2$, and cardinality $N^{n}$. The number of such codes is $N^{n}!$.

Example 5**.**

Consider four binary random variables $(X_{1},X_{2},Y_{1},Y_{2})$; that is, $n=2$ pairs of $N=2$ valued variables. The $N!^{2n-1}=8$ maximizers of the multi-information of $(X_{1},X_{2},Y_{1},Y_{2})$ are

[TABLE]

The $N^{n}!=24$ maximizers of the mutual information of $\mathbf{X}=(X_{1},X_{2})$ and $\mathbf{Y}=(Y_{1},Y_{2})$ are

[TABLE]

Some maximizers of $MI(\mathbf{X},\mathbf{Y})$ can be expressed as convex combinations of maximizers of $I(X_{1},\ldots,X_{n},Y_{1},\ldots,Y_{n})$. In the current example, those marked by $\ast$ or $\ast\ast$ can. The four distributions marked with ${\ast}$ have marginals $p(X_{1},Y_{1})$ and $p(X_{2},Y_{2})$ of maximum mutual information. The four distributions marked with ${\ast\ast}$ have margins $p(X_{1},Y_{2})$ and $p(X_{2},Y_{1})$ of maximum mutual information. We will discuss them further in Example 19. ∎

4 Factorized Measures of Multi-Information

We propose alternatives to the multi-information based on averages over subsets of random variables. After presenting the definitions in this section, we investigate the maximizers in the next Sections 5 and 6.

Definition 6**.**

For a family $\Lambda$ of subsets of $\{1,\ldots,n\}$, the $\Lambda$-factorized multi-information of $(X_{1},\ldots,X_{n})$ is defined as

[TABLE]

Here $I((X_{i})_{i\in\lambda})$ is the multi-information of the marginal distribution of variables $X_{i}$, $i\in\lambda$. In particular, $I_{\{1,\ldots,n\}}\equiv I$ and $I_{\{i\}}\equiv 0$.

We will focus on two special cases. The first is the average mutual information between all pairs of variables. We call this the average mutual information, or the pairwise factorized multi-information (FMI).

Definition 7** (Factorized multi-information, FMI).**

For $n$ random variables $X_{1},\ldots,X_{n}$, the FMI is

[TABLE]

The second case that we consider is the average mutual information between pairs of variables with one variable from each of two random vectors. We call this the separated average mutual information, or the separated factorized mutual information (SFMI).

Definition 8** (Separated factorized mutual information, SFMI).**

For two random vectors ${\bf X}=(X_{1},\dots,X_{n})$ and ${\bf Y}=(Y_{1},\dots,Y_{n})$, the SFMI is

[TABLE]

In the reinforcement learning application mentioned in the introduction, $(X_{i},Y_{i})$ corresponds to time consecutive readings of the $i$th sensor of the agent. The average over all sensors, $i=1,\ldots,n$, is used as a proxy for $MI(\mathbf{X},\mathbf{Y})$. We will investigate the properties of this choice.

An illustration of the proposed factorized measures is shown in Figure 1. In this figure, edges correspond to the pair marginals whose mutual-information is being added in the factorized measure. One natural question which arises is: how do the distributions with large multi-information compare to those with large factorized mutual information, under the various choices of margins? Since the factorized measures average interdependence between subsets of random variables, they can be less descriptive than measuring the joint interdependence of all of the variables. In the case where the joint distribution factorizes into the product of pairwise interaction terms, the multi-information naturally decomposes as the sum over mutual information terms [14]. When higher order interactions exist, the average of pairwise mutual information can serve as an approximation to the multi-information. In relation to this, we note that the exponential family with sufficient statistics computing $\Lambda$ margins is, by definition, able to attain all feasible values of these margins, and in particular it is able to express all possible values of $I_{\Lambda}$.

The trade-off between computing averages of mutual information between pairs of variables and multi-information or mutual information between groups of variables is that the former can be more easily estimated, since it involves only pair margins and not the full joint distribution. The computation of the pairwise factorized multi-information scales with $n^{2}\cdot N^{2}$, where $N=\max_{i}|{\cal X}_{i}|$, whereas the computation of multi-information scales with $N^{n}$.

5 Maximizers of the Factorized Multi-information

The next lemma shows that every marginal of a joint distribution with maximum multi-information also has maximum multi-information. A related statement, for pair margins, appeared previously in [4, Remark 4.5 (c)].

Lemma 9**.**

Consider $N$-ary variables $X_{1},\ldots,X_{n}$. If a distribution maximizes the multi-information of $X_{1},\ldots,X_{n}$, then for any subset of indices $\{j_{1},\ldots,j_{k}\}\subseteq\{1,\ldots,n\}$ the corresponding marginal maximizes the multi-information of $X_{j_{1}},\ldots,X_{j_{k}}$.

Proof.

Consider a joint distribution $p(X_{1},\ldots,X_{n})$ which attains the maximum of the multi-information. By Proposition 1, this is a uniform distribution over an $N$-ary code of length $n$, minimum Hamming distance $n$, and cardinality $N$. In this case, the marginal $p(X_{n})$ is the uniform distribution over $\mathcal{X}_{n}$. Moreover, the conditional distributions $p(X_{1},\ldots,X_{n-1}|X_{n}=x_{n})$, $x_{n}\in\mathcal{X}_{n}$, are $N$ point distributions supported on strings of Hamming distance $n-1$.

The marginal distribution of $(X_{1},\ldots,X_{n-1})$ can be written as

[TABLE]

which is a uniform distribution over $N$ strings of length $n-1$ and minimum Hamming distance $n-1$, and hence has maximum multi-information. ∎

Lemma 9 can be used to show the following theorem, which characterizes the choices of $\Lambda$ for which the maximizers of $I_{\Lambda}$ coincide with those of $I$.

Definition 10**.**

A family $\Lambda$ of subsets of $\{1,\ldots,n\}$ is a connected covering if it contains a list of sets $\lambda_{t}$, $t=1,\ldots,T$, with $\lambda_{t}\cap(\cup_{s=1}^{t-1}\lambda_{s})\neq\emptyset$ for $t=2,\ldots,T$, and $\cup_{t}\lambda_{t}=\{1,\ldots,n\}$.

Theorem 11** (Maximizers of FMI vs. I).**

Consider $N$-ary variables $X_{1},\ldots,X_{n}$ with $N>1$, and a family $\Lambda$ of subsets of $\{1,\ldots,n\}$. The set of distributions that maximize $I_{\Lambda}(X_{1},\ldots,X_{n})=\frac{1}{|\Lambda|}\sum_{\lambda\in\Lambda}I((X_{i})_{i\in\lambda})$ coincides with the set of distributions that maximize $I(X_{1},\ldots,X_{n})$ if $\Lambda$ is a connected covering of $\{1,\ldots,n\}$, and is a strict superset otherwise.

Proof of Theorem 11.

Denote $\mathcal{M}$ and $\mathcal{N}$ the sets of joint distributions that maximize the multi-information $I(X_{1},\ldots,X_{n})$ and the factorized multi-information $I_{\Lambda}(X_{1},\ldots,X_{n})$, respectively. We prove first that $\mathcal{M}\subseteq\mathcal{N}$. Since the $I_{\Lambda}$ is a sum of bounded non-negative terms, if each term is independently maximized, then $I_{\Lambda}$ will be maximized. Consider $p(X_{1},\dots,X_{n})\in{\cal M}$. Lemma 9 shows that each marginal $p(X_{\lambda})$, $\lambda\in\Lambda$, is a multi-information maximizer. Therefore, $p\in\mathcal{N}$ and $\mathcal{M}\subseteq\mathcal{N}$.

We proceed with the if statement. Assume that $\Lambda$ is a connected covering of $\{1,\ldots,n\}$. We need to show that $\mathcal{N}\subseteq\mathcal{M}$. Suppose that $p\in\mathcal{N}$. Since $\Lambda$ is a connected covering, there is a list of sets $\lambda_{t}$, $t=1,\ldots,T$, with $\lambda_{t}\cap(\cup_{s}^{t-1}\lambda_{s})\neq\emptyset$ for $t=2,\ldots,T$, and $\cup_{t}\lambda_{t}=\{1,\ldots,n\}$. By the above arguments, each $\lambda_{t}$-marginal has maximum multi-information. This means that each marginal $p(X_{\lambda_{t}})$ is one of $N!^{|\lambda_{t}|-1}$ possible distributions (this is the number of $N$-ary codes of cardinality $N$, minimum distance $|\lambda_{t}|$, and length $|\lambda_{t}|$). Each marginal fixes the joint states of $|\lambda_{t}|$ variables to one of $N$ possible choices. Marginals with overlapping variables need to be compatible on the overlap. For a given choice of the $\lambda_{s}$-marginals, $s=1,\ldots,t-1$, there is exactly one compatible choice of the $\lambda_{t}$-marginal. Since each variable appears in at least one marginal and there is a sequence of overlapping marginals that cover all variables, the joint states of all variables are fixed to one of $N$ possible strings which have distance $n$ from each other. Therefore, $p\in\mathcal{M}$ and $\mathcal{N}\subseteq\mathcal{M}$.

It remains to show the only if statement. For this, we need to show that $\mathcal{M}\neq\mathcal{N}$ if $\Lambda$ is not a connected covering. If $\Lambda$ is not a connected covering, then one of the following is true:

1. there is an $i\in\{1,\ldots,n\}$ which is not contained in any $\lambda\in\Lambda$, or
2. $\Lambda$ consists of two (or more) subfamilies of sets that are mutually disjoint.

In the first case, $I_{\Lambda}$ is independent of the marginal distribution of $X_{i}$. On the other hand, for any maximizer of the multi-information the marginal of $X_{i}$ is a uniform distribution. Hence, provided that $N>1$ (which we always assume), $\mathcal{M}\neq\mathcal{N}$.

In the second case, let $\lambda_{1}$ and $\lambda_{2}$ be two disjoint subsets of $\{1,\ldots,n\}$ which contain all sets from the family $\Lambda$. If $\lambda_{1}$ or $\lambda_{2}$ only contains one point $i\in\{1,\ldots,n\}$, then $I_{\Lambda}$ is independent of the marginal distribution of $X_{i}$ (since $I(X_{i})\equiv 0$), and we are in the same situation discussed in 1). Assume therefore that $\lambda_{1}$ and $\lambda_{2}$ each contain at least two points. By definition, $I_{\Lambda}$ only depends on the values of the marginal distributions $p_{1}$ of $(X_{i})_{i\in\lambda_{1}}$ and $p_{2}$ of $(X_{i})_{i\in\lambda_{2}}$. Each choice of the two marginals is realized by a set of joint distributions that is known as a $2$-way transportation polytope. If the margins take positive value over $m_{1}$ and $m_{2}$ entries, respectively, then the dimension of this polytope is $(m_{1}-1)(m_{2}-1)$. The dimension of such $2$-way transportation polytopes is known in the literature; see [46]. We will encounter a generalization later in this paper. For any $p\in\mathcal{M}$, Lemma 9 shows that the margins $p_{1}$ and $p_{2}$ are maximizers of the multi-information, and so, by Proposition 1, both have positive uniform value over $N$ entries. Since $p\in\mathcal{N}$, $\mathcal{N}$ also contains a polytope of dimension $(N-1)(N-1)>0$. On the other hand, $\mathcal{M}$ is a finite discrete set and has dimension zero. Therefore, $\mathcal{M}\neq\mathcal{N}$. This completes the proof. ∎

Theorem 11 states that maximizing the multi-information of sufficiently many margins imposes enough restrictions on the joint distributions to ensure that they also have maximum multi-information. Moreover, it characterizes the minimal families of marginals that are sufficient.

In particular, since the set of pairs of $\{1,\ldots,n\}$ defines an overlapping covering, we have the following corollary.

Corollary 12** (Maximizers of the FMI).**

The maximizers of the FMI (11) agree with the maximizers of the multi-information (4).

In fact, Theorem 11 shows that specifying $n-1$ pair margins suffices when the pairs form a connected covering of $\{1,\dots,n\}$. In particular, the exponential family with sufficient statistics computing $n-1$ pair margins contains in its closure all maximizers of the multi-information. This relates to the result from [4, Theorem 5.1], which shows that the closure of an exponential family of $n-1$ pure pair interactions contains all maximizers of the multi-information.

Example 13**.**

Consider the case of three binary variables, $\Lambda=\{\{1,2\},\{2,3\}\}$, and the following two mutual information maximizing marginals:

[TABLE]

These marginals fix the joint states of $(X_{1},X_{2},X_{3})$ in the following way. Write the joint distribution as a convex combination of point distributions,

[TABLE]

Eqn. (13) imposes following linear constraints on the joint distribution:

[TABLE]

Eqn. (14) imposes following linear constraints:

[TABLE]

Since probabilities are non-negative, the last two constraints in each case imply $p_{010}=p_{011}=p_{100}=p_{101}=0$ and $p_{001}=p_{011}=p_{100}=p_{110}=0$. Therefore, it must be that $p_{000}=p_{111}=\frac{1}{2}$, showing that specifying two overlapping marginal distributions to be maximizers is sufficient to conclude that the joint distribution is also a multi-information maximizer. ∎

When $\Lambda$ is not a connected covering, the set of maximizers of $I_{\Lambda}$ has dimension larger than zero. In this case, the exponential family with sufficient statistics computing $\Lambda$ margins still contains in its closure one joint distribution that expresses each choice of the margins, and in particular it contains maximizers of $I_{\Lambda}$, but in general it does not contain all of them. We will discuss the setting of not connected coverings in more detail in the next Section 6. We conclude this section with the following remark.

Remark 14**.**

We observe that, while there always exists a joint distribution each of whose marginals maximizes the multi-information, in general there are compatibility requirements among the marginals. For example, choosing $p(X_{1},X_{2})=\frac{1}{2}(\delta_{00}+\delta_{11})$ and $p(X_{2},X_{3})=\frac{1}{2}(\delta_{01}+\delta_{10})$ means that $p(X_{1},X_{3})$ must be exactly $\frac{1}{2}(\delta_{01}+\delta_{10})$.

Verifying that a certain specification of margins is indeed feasible, meaning that there exists a joint distribution with those margins, is known as the margin specification problem. Given a family $\Lambda$ of subsets of $\{1,\ldots,n\}$, the set of margins $(p_{\lambda})_{\lambda\in\Lambda}$ that are feasible, is known in the literature as the $\Lambda$-marginal polytope, which can be obtained as the convex hull of the vectors

[TABLE]

where for each $\lambda$ and $\tilde{x}_{\lambda}$ there is a coordinate taking value one if $x_{\lambda}=\tilde{x}_{\lambda}$ and value zero otherwise. These vectors form the columns of a matrix of sufficient statistics, computing the $\Lambda$-margins, for the exponential family known as the $\Lambda$-interaction model.

For a given vector $(p_{\lambda})_{\lambda}$ of $\lambda$-margins, $\lambda\in\Lambda$, the set of joint distributions that express these margins is known as a linear $\Lambda$-family at $p$ (where $p$ is a compatible joint distribution) [33], or also as the $n$-way transportation polytope $T(p_{\lambda},\lambda\in\Lambda)$ [46]. This is obtained by intersecting the simplex of joint probability distributions with the orthogonal complement of the row span of $F$ at $p$. We will encounter this type of structures in the next section. More details are provided in Appendix B. ∎

6 Maximizers of the Factorized Mutual Information

We have seen that the maximizers of any factorized multi-information measure with connected covering, like the ones shown in Figure 2, coincide with the maximizers of the multi-information. These distributions are very different from the maximizers of the mutual information of two random vectors of length $n\geq 2$. If we want to maximize $MI(\mathbf{X},\mathbf{Y})$ by maximizing a factorized measure $I_{\Lambda}$, then we should choose a $\Lambda$ that is not a connected covering.

The SFMI (12) only considers the mutual information of pairs $(X_{i},Y_{i})$, $i=1,\ldots,n$, which is not a connected covering. Theorem 11 shows that the SFMI maximizers are a strict superset of the multi-information maximizers:

Corollary 15**.**

Let $n\geq 2$. The maximizers of $I(X_{1},\ldots,X_{n},Y_{1},\ldots,Y_{n})$ are a strict subset of the maximizers of $SFMI(\mathbf{X},\mathbf{Y})$.

In the following we present a characterization of the set of maximizers of SFMI. We start with the next theorem, which describes the number of connected components, dimension, and support sets.

Theorem 16** (Maximizers of SFMI vs. I).**

The set of maximizers of the SFMI is a superset of the maximizers of the multi-information. It consists of the joint distributions with marginals $p(X_{i},Y_{i})$ maximizing $MI(X_{i},Y_{i})$, for $i=1,\ldots,n$. This is the disjoint union of $N!^{n}$ transportation polytopes of dimension $N^{n}-1-n(N-1)$ and whose interiors consist of distributions with support of cardinality $N^{n}$.

The SFMI maximizing polytopes are the sets of joint probability distributions that are compatible with the specification of $n$ pair margins as distributions with maximum mutual information. Since the pair margins have disjoint variables, the polytopes can be regarded as orthogonal linear families of an independence model of $n$ $N^{2}$-ary variables.

But since the specified margins are uniform distributions with support of cardinality $N$, the polytopes have the structure of orthogonal linear families of an independence model of $n$ $N$-ary variables at the uniform distribution. In this perspective, the elementary events of the $i$th variable are pairs $(x_{i},y_{i})$ forming an $N$-ary code of length $2$, minimum Hamming distance $2$, and cardinality $N$.

Proof of Theorem 16.

If we consider a subset $\lambda$ of variables, fixing a marginal distribution $p(X_{\lambda})$ with support $\tilde{\cal X}_{\lambda}\subseteq{\cal X}_{\lambda}$ defines a linear set in the joint probability simplex of dimension $|\tilde{\cal X}_{\lambda}|(|{\cal X}_{\lambda^{c}}|-1)$. This is namely the set of distributions of the form $p(X_{\lambda})q(X_{\lambda^{c}}|X_{\lambda})$. The dimension count comes from the fact that for each $x_{\lambda}\in\tilde{\cal X}_{\lambda}$, the conditional distribution $q(X_{\lambda^{c}}|X_{\lambda}=x_{\lambda})$ can be chosen independently as an arbitrary point in a $(|{\cal X}_{\lambda^{c}}|-1)$ dimensional simplex.

For variables $(X_{1},\ldots,X_{n},Y_{1},\ldots,Y_{n})$, the set of maximizers of the MI of a given pair $(X_{i},Y_{i})$ corresponds to a set ${\cal S}_{i}=\cup_{j=1}^{N!}{\cal S}_{i,j}$ in the joint probability simplex which is the disjoint union of $N!$ linear spaces of dimension $N(N^{2n-2}-1)$. Indeed, there are $N!$ maximizers of the marginal mutual information. Each of them corresponds to fixing a marginal distribution $p(X_{i},Y_{i})$ with support of cardinality $N$, which defines a set ${\cal S}_{i,j}$ in the joint probability simplex. In terms of the above discussion, ${\cal X}_{\lambda}={\cal X}_{i}\times{\cal Y}_{i}$ and $|\tilde{\cal X}_{\lambda}|=N$. The set $\mathcal{S}$ of maximizers of the SFMI is the intersection of $n$ such sets, ${\cal S}=\cap_{i=1}^{n}\,{\cal S}_{i}$, one corresponding to each pair marginal. Each marginal implies that $(N^{2}-N)N^{n-2}$ entries of the joint distribution vanish. Overall, only $N^{n}$ entries of the joint distribution can have nonzero value. Each marginal implies $N$ linear constraints of the form $\frac{1}{N}=p(X_{i}=x_{i},Y_{i}=y_{i})=\sum_{x_{\hat{i}},y_{\hat{i}}}p(x_{1},\ldots,x_{n},y_{1},\ldots,y_{n})$. Each marginal contributes $(N-1)$ independent constraints, and have one shared constraint, namely that the sum of entries is $1$. The number of independent constraints can be derived from the fact that $\mathbf{1}_{\lambda,x_{\lambda}}$, $\lambda\subseteq\Lambda$, $x_{\lambda}\in\prod_{i\in\lambda}(\mathcal{X}_{i}\setminus\{0\})$, including $\mathbf{1}_{\emptyset}\equiv 1$, is a basis of the set of functions $\sum_{\lambda\in\Lambda}f_{\lambda}(x)$; see [18].

We have $N^{n}$ positive entries satisfying $n(N-1)+1$ independent linear equality constraints, which gives the desired dimension $N^{n}-n(N-1)-1$.

In terms of the number of pieces, all pieces defined by one marginal intersect with all pieces defined by any other marginal. Since each pair marginal defines $N!$ pieces, the total number of pieces is $N!^{n}$.

The fact that the set of SFMI maximizers is a superset of the set of FMI maximizers follows from the fact that it is defined by a subset of the affine constraints that define the FMI maximizers. ∎

The next question is whether the transportation polytopes maximizing the SFMI contain some of the maximizers of the MI. Transportation polytopes are not very well understood in general. We obtain the following characterization.

Theorem 17** (Maximizers of SFMI vs. MI).**

Each of the $N!^{n}$ transportation polytopes maximizing the SFMI contains $N!^{n-1}$ vertices which are maximizers of the multi-information. These vertices come in $(N-1)!^{n-1}$ disjoint sets, each set consisting of $N^{n-1}$ affinely independent vertices and having the same centroid, which is a maximizer of $MI({\bf X},{\bf Y})$.

To put this in perspective, recall that $I(X_{1},Y_{1},\ldots,X_{n},Y_{n})$ has $N!^{2n-1}$ maximizers, and $MI(\mathbf{X},\mathbf{Y})$ has $N^{n}!$ maximizers. Moreover, the dimension of each of the transportation polytopes in question is $N^{n}-1-n(N-1)$. The total number of vertices of a transportation polytope is not known in general. For the special type of transportation polytopes that we consider here, the theorem gives a lower bound $N!^{n-1}$.

Proof of Theorem 17.

The proof is by construction. Consider one of the $N!^{n}$ polytopes, ${\cal S}=\cap_{i=1}^{n}{\cal S}_{i,j_{i}}$ for a fixed choice of $j_{i}\in\{1,\ldots,N!\}$, for $i=1,\ldots,n$. Let $p_{i,j_{i}}$ be the corresponding $(X_{i},Y_{i})$ margin, which is a uniform distribution on $N$ strings of length two and minimum distance two, out of the $N^{2}$ possible strings of length two. Denote these $N$ strings by $z^{k}_{i}=(x^{k}_{i},y^{k}_{i})$, $k=1,\ldots,N$. In the following we will write the joint states of all variables as $((x_{1},y_{1}),\ldots,(x_{n},y_{n}))$.

We claim that ${\cal S}$ contains $\frac{1}{N}\sum_{k=1}^{N}\delta_{z_{1}^{\pi_{1}(k)}\cdots z_{n}^{\pi_{n}(k)}}$, for each choice of an ordering $\pi_{i}$ of $\{1,\ldots,N\}$, for each $i=1,\ldots,n$. Indeed, for each $i=1,\ldots,n$ the $i$th margin is $\frac{1}{N}\sum_{k}\delta_{z_{i}^{k}}=p_{i,j_{i}}$.

Each of these points is a distinct maximizer of the multi-information of $(Z_{1},\ldots,Z_{n})$. Indeed, each of them is a uniform distribution on $N$ strings of length $n$ and minimum distance $n$ (with each $Z_{i}$ regarded as a single $N$ valued variable). They are also maximizers of the multi-information of $(X_{1},Y_{1},\ldots,X_{n},Y_{n})$, since any distinct $z_{i}^{k}$ and $z_{i}^{k^{\prime}}$ from our list have, by definition, distance two when regarded as strings $(x^{k}_{i},y^{k}_{i})$ and $(x^{k^{\prime}}_{i},y^{k^{\prime}}_{i})$.

Summarizing, ${\cal S}$ contains $N!^{n-1}$ distinct maximizers of the multi-information. These points are also vertices of the polytope. Since they have support of cardinality $N$, they cannot be expressed as a non-trivial combination of maximizers of SFMI. Indeed, $N$ is the minimum cardinality of the support of any maximizer of SFMI (because this is the minimum cardinality of any maximizer of the mutual information of any pair margin). Moreover, any maximizer of SFMI with support of cardinality $N$ must be the uniform distribution over that support (because this is true for any maximizer of the MI of any pair margin).

Next we show that ${\cal S}$ contains $(N-1)!^{n-1}$ simplices which are each the convex hull of $N^{n-1}$ distinct vertices of ${\cal S}$. Recall that $|\operatorname{supp}({\cal S})|=\max\{|\operatorname{supp}(p)|\colon p\in{\cal S}\}=N^{n}$. Because ${\cal S}$ has $N!^{n-1}$ vertices which are multi-information maximizers, it follows that there are $N!^{n-1}$ codes of length $2n$, minimum distance $2n$, and cardinality $N$. These are the support sets of the vertices. One can find $N^{n-1}$ codes of this form which are disjoint and cover all strings (see Proposition 22). For $N^{n-1}$ disjoint codes, each of cardinality $N$, the union contains $N^{n}$ distinct strings and covers $\operatorname{supp}({\cal S})$. One can find $\frac{N!^{n-1}}{N^{n-1}}=(N-1)!^{n-1}$ such collections of codes (see Proposition 23), which correspond to the different simplices.

Consider the vertices corresponding to a single collection of $N^{n-1}$ vertex support sets. Any distribution written as a convex combination of these vertices is in ${\cal S}$. Since the support sets of these vertices are disjoint, it follows that a strictly convex combination will have support on $N^{n}$ vectors. Furthermore, the convex hull of these vertices can be seen as a simplex of dimension $N^{n-1}-1$. At the center of the simplex where each vertex is given equal weight, one obtains the uniform distribution over the $N^{n}$ vectors of $\operatorname{supp}({\cal S})$. Recall from the previous that the vectors in $\operatorname{supp}({\cal S})$ are $N$-ary vectors of length $2n$ with minimum distance $2$, and therefore the uniform distribution over $N^{n}$ such vectors is a maximizer of $MI({\bf X},{\bf Y})$. Since each of the $(N-1)!^{n-1}$ simplices have the same support of $\operatorname{supp}({\cal S})$, their centroids all coincide at a maximizer of $MI({\bf X},{\bf Y})$. ∎

In particular, we note the following.

Corollary 18**.**

The maximizers of the multi-information are vertices of the polytopes of maximizers of the SFMI.

We note that the MI is independent of the ordering of the variables within the vectors $\mathbf{X}$ and $\mathbf{Y}$. The SFMI measure that we have introduced corresponds to $I_{\Lambda}$ where $\Lambda$ is a specific pairing of variables in the two vectors. Each $\Lambda$ consisting of $n$ pairs that are a perfect matching between $X_{1},\ldots,X_{n}$ and $Y_{1},\ldots,Y_{n}$ defines a different SFMI measure. The situation is illustrated in Figure 3. We expect that for each of the $n!$ possible pairings, the corresponding SFMI measure will have among its maximizers, different subsets of maximizers of the MI that can be written as convex combinations of multi-information maximizers. At this point it remains an open question how the sets of MI maximizers that maximize different SFMI measures compare to each other, and how many MI maximizes total are maximizers of some SFMI measure.

We illustrate the discussion of this section in the next example.

Example 19**.**

Consider $n=2$ pairs of $N=2$ valued variables $(X_{1},X_{2},Y_{1},Y_{2})$. The maximizers of the SFMI (12) are $N!^{n}=4$ polytopes of dimension $N^{n}-1-n(N-1)=1$. With non-negative $\alpha,\beta$ such that $\alpha+\beta=1$, the polytopes are

[TABLE]

A detailed derivation is provided in Appendix C.2. When $\alpha,\beta$ are strictly positive, these distributions have support of cardinality $4$ and do not maximize multi-information. However we see that the vertices, with $\beta=0$ or $\alpha=0$, recover all $8$ maximizers of multi-information. Additionally, setting $\alpha=\beta=\frac{1}{2}$ recovers $4$ out of the $24$ maximizers of $MI({\bf X},{\bf Y})$. These are the four points marked with ${\ast}$ in Example 5. Notice that the SFMI maximizers with highest entropy are maximizers of $MI({\bf X},{\bf Y})$.

Consider now an SFMI with a different assignment of $X$s and $Y$s, SFMI^{\prime}=\frac{1}{2}\big{(}MI(X_{1},Y_{2})+MI(X_{2},Y_{1})\big{)}. With non-negative $\alpha,\beta$ such that $\alpha+\beta=1,$ the polytopes of $SFMI^{\prime}$ maximizers are

[TABLE]

At the center of these polytopes where $\alpha=\beta$, the maximizers marked ${\ast\ast}$ in Example 5 are found, demonstrating that distinct subsets of MI maximizers are contained in each of the $n!$ possible SFMI measures. ∎

We provide more examples in Appendix C.2.

7 Codes and partitions

In the proof of Theorem 17 we used certain properties of the set of $N$-ary strings of length $n$; specifically, we used that it can be partitioned into codes of minimum distance $n$ and cardinality $N$, in a number of different ways. In this section we state and prove these facts.

Proposition 20**.**

There are $N!^{n-1}$ $N$-ary codes of length $n$, minimum distance $n$, and cardinality $N$.

Proof.

An $N$-ary code of length $n$, minimum distance $n$, and cardinality $N$ corresponds to a perfect covering of a rectangular grid of size $N\times n$ by $N$ strings of length $n$. In order to count the total number, consider the grid, and the possible choices for how to place the first string. The first coordinate is $1$, for the second coordinate there are $N$ choices, for the third coordinate there are $N$ choices, etc. For the second string the first coordinate is $2$, and there are $N-1$ possible choices for each of the remaining coordinates. For the $N$th string there is only one choice left. So in total we have $N^{n-1}\cdot(N-1)^{n-1}\cdots 1^{n-1}=N!^{n-1}$. ∎

Proposition 21**.**

The set of all $N^{2}$ edges of the full bipartite graph on $N\times N$ can be partitioned into $N$ sets of $N$ edges, each of which forms a perfect matching.

Proof.

Denote the edges by $(u,v)\in\{0,\ldots,N-1\}\times\{0,\ldots,N-1\}$. Each $t=1,\ldots,N$ defines a perfect matching $\{(u,u+t\mod(N))\colon u\in\{0,\ldots,N-1\}\}$. Each of these matchings uses one distinct edge, out of the $N$ edges that connect to the $u$-th left node, for each $u$. Hence all of these matchings have disjoint edge sets, and they, together, contain all edges from the full bipartite graph. ∎

Proposition 22**.**

The set of all $N^{n}$ $N$-ary strings of length $n$ can be partitioned into $N^{n-1}$ codes of minimum distance $n$ and cardinality $N$.

Proof.

We want to partition the set of all $N^{n}$ strings into $N^{n-1}$ sets, each of which gives a perfect covering of a grid of size $N\times n$. For any $(t_{2},\ldots,t_{n})\in\{0,\ldots,N-1\}^{n-1}$, define a code as $\{(u,u+t_{2}\mod(N),\ldots,u+t_{n}\mod(N))\colon u=0,\ldots,N-1\}$. The proof of Proposition 21 discusses how this construction gives a perfect matching between all nodes in column $i$ and column $i+1$, for all $i=1,\ldots,n-1$. There are $N^{n-1}$ of these assignments and they all build disjoint sets of strings. They, together, exhaust all $N^{n}$ strings, by the same reasoning of Proposition 21. ∎

Proposition 23**.**

There are $N!^{n-1}/N^{n-1}$ ways to partition the set of all $N^{n}$ $N$-ary strings of length $n$ in the form described in Proposition 22.

Proof.

Consider again each $N$-ary code of length $n$, minimum distance $n$, and cardinality $N$ as a perfect covering of an $N\times n$ grid by strings. Consider the partition described in Proposition 22 of the set of $N$-ary strings into $N^{n-1}$ such codes. Note that this partition is constructed by applying circular shifts, which define a subgroup of the symmetric group. Now we simply consider the cosets defined by the circular shifts. The cardinality of all cosets is equal. Two cosets are either equal or are disjoint. The number of distinct cosets is $N!/N=(N-1)!$. This gives a total of $(N-1)!^{n-1}$ partitions. ∎

8 Discussion

We formulated factorized versions of the mutual information and studied the sets of joint probability distributions that maximize them. As we observe, characterizing the maximizers of these measures translates to characterizing the sets of joint distributions that are compatible with a given specification of margins.

When a sufficient number of margins is included in the factorized measure, the maximizers correspond precisely to the maximizers of the multi-information. For this to be the case, it is necessary and sufficient that the set of margins builds a connected covering of all variables. In particular, there are families of pair margins that are sufficient, which reflects previous results showing that the multi-information can be maximized over families of joint distributions that only include pair interactions.

When the considered margins have disjoint sets of variables, the constraints that they impose on the joint distributions no longer specify the joint distributions down to the maximizers of the multi-information. Instead, they specify families of transportation polytopes some of whose vertices are multi-information maximizers. As we showed, certain choices of margins (as in the SFMI) lead to polytopes that contain some of the maximizers of the mutual information between two subsets of variables.

MI vs. I

The set of maximizers of the multi-information $I(X_{1},\ldots,X_{n},Y_{1},\ldots,Y_{n})$ is a discrete set of cardinality $(N!)^{2n-1}$. The set of maximizers of the mutual information $MI({\bf X},{\bf Y})$ is a discrete set of cardinality $(N^{n})!$. The two sets are equal if $n=1$, and are disjoint otherwise. These are well known results, collected in Proposition 1 and Corollary 4.

FMI vs. I and MI

The set of maximizers of $I_{\Lambda}(X_{1},\ldots,X_{n})$ is equal to the set of maximizers of $I(X_{1},\ldots,X_{n})$ if the considered set $\Lambda$ of margins builds a connected covering of all variables, and it is a strict superset otherwise. See Theorem 11. If $\Lambda$ is a connected covering of all variables, then the maximizers of $I_{\Lambda}(X_{1},\ldots,X_{n},Y_{1},\ldots,Y_{n})$ are disjoint from maximizers of $MI(\mathbf{X},\mathbf{Y})$, unless $n=1$ in which case the two sets are equal. If $\Lambda$ is not a connected covering, the relation is open in general. We studied in detail the special case of SFMI, summarized below.

SFMI vs. I and MI

SFMI is the special case of $I_{\Lambda}$ with $\Lambda$ being a perfect matching of $X$ and $Y$ variables. This is not a connected covering, except when $n=1$. The set of SFMI maximizers consists of $N!^{n}$ disjoint polytopes of dimension $N^{n}-1-n(N-1)$, each of which has the structure of a central $n$-way transportation polytope defined by $1$-margins of size $N$. See Theorem 16. Each of the $N!^{n}$ polytopes maximizing SFMI contains $N!^{n-1}$ maximizers of $I(X_{1},\ldots,X_{n},Y_{1},\ldots,Y_{n})$ and one maximizer of $MI({\bf X},{\bf Y})$. See Theorem 17.

Factorized multi-information as an intrinsic reward

When the MI is superimposed with another objective function, we may obtain specific types of solutions which maximize both, or simply which tend to have a larger MI.

The SFMI maximizers do not include all MI maximizers. An interpretation is that the SFMI maximizers also maximize the MI between specific pairs of variables. So for instance, the MI can be maximized if $H(Y_{1}|X_{1})\neq 0$, if in exchange one has that $H(Y_{2}|X_{1})=0$, meaning that sensor $2$ can be predicted from sensor $1$. However, the maximizers of SFMI insist that $H(Y_{1}|X_{1})=0$. In this sense, the SFMI gives a more structured objective function than the MI.

Possible generalizations

We focused on two general types of margins. In the case of families that are not connected coverings, we considered the SFMI, which consists of a non overlapping covering by edges. Extensions of our analysis to other families of margins are possible. Any connected subfamily will then specify a maximizer of the multi-information over the covered variables. This will give rise to maximizing sets defined in terms of transportation polytopes with margins given by the maximal connected subfamilies. A natural question that arises is how to choose $\Lambda$ in order to capture, as tightly as possible, a given subset of maximizers of $MI(\mathbf{X},\mathbf{Y})$.

We have focused on systems where all variables have the same number of states. The same problem can be studied in the case of inhomogeneous variables as well. One should be able to obtain a characterization as was done in [4] for the multi-information.

The case of continuous variables is another geneneralization that would be interesting to consider in the future. This has not been discussed in as much detail in the literature as the discrete case.

Acknowledgement This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement no 757983).

Appendix A Related measures of multivariate correlation

Multi-information as defined above was first introduced by McGill (1954) [26] under the name of total correlation, and later discussed in [44] under the same name. Over time as the relationship between total correlation and mutual information became clear, several authors have begun to call this quantity multi-information [4, 14, 39, 38, 6]. However, McGill’s seminal work on information transmission also introduced an entirely separate information-theoretic quantity under the name multivariate information. This quantity’s name shifted over time to also being called multi-information and multivariate mutual information, as referenced in [43, 19]. This naming confusion has been the source of false equalities between the two quantities, such as in [35, Eqn. 8]. One fundamental difference in the two quantities is that multi-information as defined here is non-negative, and multivariate mutual information can take positive and negative values.

Appendix B Distributions with fixed Margins

In various contexts one is interested in the sets joint distributions that are compatible with given values of some of the marginals. We encounter a version of this problem here. Concretely, we study whether certain collections of subsets of variables can each have maximum multi-information, and, if so, what is the set of joint distributions that are compatible with these specifications.

B.1 Transportation polytopes

An $n$-table is an $N_{1}\times\cdots\times N_{n}$ array of nonnegative real numbers. For a set of indices $\lambda\subseteq\{1,\ldots,n\}$, the $\lambda$-margin of a table $p$ is the $|\lambda|$-table obtained by summing the entries over all but the $\lambda$ indices. A multi-index transportation polytope or contingency table is the set of all $n$-tables that satisfy a set of given margins. The polytope is called central if all entries of any of the considered margins are equal. An assignment polytope is the special case where all entries across all of the considered margins are equal (which implies $N_{1}=\cdots=N_{n}=N$). The two-way assignment polytopes are also called Birkhoff polytopes or polytopes of doubly stochastic matrices (when the value of the margin entries is $1$). The $N!$ vertices of the Birkhoff polytope are the permutation matrices of size $N\times N$.

Optimal transportation problems are often formulated as linear programs with feasible set given by a transportation polytope. Hence the number of vertices and edges has been subject of study. For a 2-way transportation polytope, it is known that the dimension is equal to $(N_{1}-1)(N_{2}-1)$ (if the entry sums of the two margins are equal). A matrix $p$ in the polytope is a vertex if and only if the graph $F_{p}$ with edges $\{(x_{1},x_{2})\colon p(x_{1},x_{2})>0\}$ forms a spanning forest of the full bipartite graph with edges $\{(x_{1},x_{2})\colon 1\leq x_{1}\leq N_{1},1\leq x_{2}\leq N_{2}\}$. See [46]. Already for $3$-way transportation polytopes, the number of vertices is not known in general [13].

The set of maximizers of the SFMI is a union of polytopes, each of which has the structure of an $n$-way assignment polytope defined by $1$-margins of size $N$.

B.2 Marginal polytopes

Marginal distributions satisfy certain relations among each other. Consider the tuple of $\lambda$-marginals for all $\lambda\in\Lambda$ for some fixed $\Lambda\in 2^{\{1,\ldots,n\}}$. The set of all possible tuples is called the marginal polytope. If we consider all $q$-marginals among $n$ variables with state spaces of cardinality $N$, the dimension of the marginal polytope is $\sum_{i=1}^{q}{n\choose i}(N-1)^{i}$. In turn, a fixed feasible and generic choice of all $q$ marginals cuts out a set (a transportation polytope) in the joint probability simplex of dimension $N^{n}-1-\sum_{i=1}^{q}{n\choose i}(N-1)^{i}$. If the choice of the margins sits at the boundary of the marginal polytope, then the set that is cut out in the joint probability simplex will be smaller. Classifying the possible values of the dimension is an open problem in general. In our discussion of transportation polytopes maximizing the SFMI, we remove zeros and obtain uniform margins.

Marginalizing over $n-q$ variables can be viewed as a linear projection ${\cal L}:\Delta_{n}\to\Delta_{q}$. Each marginal distribution that we specify in $\Delta_{q}$ provides $N^{q}$ linear constraints (including normalization). In our discussion with margins being maximizers of multi-information, $N$ entries of the margin take value $1/N$ and the other $N^{q}-N$ entries take value [math]. The constraints with value [math] imply that $(N^{q}-N)\cdot N^{n-q}$ entries of the joint distributions also take value [math]. So each margin creates $N$ linear constraints for $N\cdot N^{n-q}$ entries of the joint distribution, and in addition, it equates the remaining entries of the joint distribution to zero. If we forget about the inequality constraints, the number of linearly independent constraints that arise from the specification of the marginal distributions is equal to the dimension of the of the marginal polytope.

Appendix C Examples of the sets of SFMI maximizers

C.1 The dimension of the sets of SFMI maximizers

The linear sets of SFMI maximizers come from taking the intersection of $n$ margin constraints, each of which defines a set ${\cal S}_{i,j}$. Each of these sets is defined by a number of linear equality constraints, which in general are not independent for multiple margins. When there are $n$ pairs of binary variables, each ${\cal S}_{i,j}$ is supported on $2^{2n-1}$ vectors. Only half of the support vectors between any given ${\cal S}_{i,j}$ agree, meaning ${\cal S}_{i,j}\cap{\cal S}_{i,j}^{\prime}$ has common support of size $2^{2n-2}$, and ${\cal S}_{i,j}\cap{\cal S}_{i,j}^{\prime}\cap{\cal S}_{i,j}^{\prime\prime}$ has support of size $2^{2n-3}$. Once the intersection of $n$ of these sets is performed, one is left with $2^{2n-n}=2^{n}$ support vectors. Then, the linearly independent constraints arising from the ${\cal S}$’s can be subtracted off of $2^{n}$. The result will be the dimension of the linear sets. The sets of maximizers also need to satisfy linear inequality constraints. In general, these can further reduce the dimension of the solution set.

Example 24**.**

Consider $n=1$ pair of $N=2$ valued (binary) variables. The SFMI maximizing set is exactly the MI maximizers, i.e. a zero dimensional set. The set ${\cal S}_{1,1}$ has support on $2^{1}=2$ vectors, $00$ and $11,$ and there are two constraints, $a=1/2$ and $b=1/2$ for $a\delta_{00}+b\delta_{11}.$ Written as a matrix, it would be a rank 2 matrix of constraints. Normalization is satisfied by these constraints, and the result is a zero dimensional set. ∎

Example 25**.**

Consider $n=2$ pairs of $N=2$ valued variables. Then ${\cal S}_{1,1}\cap{\cal S}_{2,1}$ has $4$ vectors in the support, and the constraints that arise are given by

[TABLE]

where $p=\frac{a}{2}\delta_{0000}+\frac{b}{2}\delta_{0101}+\frac{c}{2}\delta_{1010}+\frac{d}{2}\delta_{1111}.$ This matrix is rank 3, meaning the dimension of the resulting linear set is $4-3=1$. ∎

Example 26**.**

When $n=3$ pairs of $N=2$ valued variables. Then the intersection ${\cal S}_{1,1}\cap{\cal S}_{2,1}\cap{\cal S}_{3,1}$ has support on $2^{3}=8$ vectors, and the rank of the constraint matrix,

[TABLE]

is 4. Therefore we expect the linear sets of maximizers to be of dimension of null space, $8-4=4$. ∎

Example 27**.**

Consider $n=2$ pairs of $N=3$ valued variables. Then ${\cal S}_{1,1}\cap{\cal S}_{2,1}$ is supported on $N^{n}=9$ vectors. The constraint matrix looks like,

[TABLE]

Here, the coefficients correspond to the joint distributions probabilities on $0000,0101,02020,1010,1111,1212,2020,2121,2222$ respectively. This matrix is rank $5$ meaning the null space is $4$ dimensional. ∎

C.2 Transportation polytopes of SFMI maximizers

Example 28**.**

Consider the case of $n=2$ pairs of $N=2$ valued variables. In the following we illustrate explicitly how the SFMI maximizing distributions noted in Example 19 are all of the SFMI maximizers. Since it is known that distributions maximizing SFMI must maximize each term of the sum individually, one may consider the set of SFMI maximizers as the intersection of $n$ sets where each set contains all the distributions which maximize a single term of the sum, i.e. each maximize $D(p(x_{i},y_{i})\|{\cal F})$ for some $i$.

Consider the four binary variables $X_{1},X_{2},Y_{1},Y_{2}$. The maximizers of $I(X_{1},Y_{1})$ are $p^{1}_{1}(X_{1},Y_{1})=\frac{1}{2}(\delta_{00}+\delta_{11})$ and $p^{2}_{1}(X_{1},Y_{1})=\frac{1}{2}(\delta_{01}+\delta_{10})$. The corresponding joint distributions have the form $p(X_{1},X_{2},Y_{1},Y_{2})=p^{j}_{1}(X_{1},Y_{1})q(X_{2},Y_{2}|X_{1},Y_{1})$. Therefore, one may define ${\cal S}_{1,1}$ to be the set of distributions which maximize $I(X_{1},Y_{1})$ and have marginal $p^{1}_{1}(X_{1},Y_{1}).$ Likewise, ${\cal S}_{1,2}$ is the set of distributions which maximize $I(X_{1},Y_{1})$ and have marginal $p^{2}_{1}(X_{1},Y_{1}).$ These can be written as

[TABLE]

and

[TABLE]

respectively. Here, $q(x_{2},y_{2}|x_{1}=0,y_{1}=0)$ is the conditional probability of observing the joint state $(X_{1}=0,X_{2}=x_{2},Y_{1}=0,Y_{2}=y_{2}).$ Similarly, the two sets ${\cal S}_{2,1}$ and ${\cal S}_{2,2}$ which maximize $I(X_{2},Y_{2})$ and have marginals $p^{1}_{2}(x_{2},y_{2})=\frac{1}{2}(\delta_{00}+\delta_{11})$ and $p^{2}_{2}(x_{2},y_{2})=\frac{1}{2}(\delta_{01}+\delta_{10})$ respectively can be written as

[TABLE]

and

[TABLE]

There are 4 possibilities for how these sets may be intersected to yield maximizers of SFMI. These are, ${\cal S}_{1,1}\cap{\cal S}_{2,1},$ ${\cal S}_{1,1}\cap{\cal S}_{2,2},$ ${\cal S}_{1,2}\cap{\cal S}_{2,1},$ and ${\cal S}_{1,2}\cap{\cal S}_{2,2}.$ It will be shown that the intersection of ${\cal S}_{1,1}\cap{\cal S}_{2,1}$ corresponds to the line of distributions in Eqn. (19), and the other intersections correspond in a similar fashion. To see this, one may begin by considering the binary codes in the intersection. These are all the binary codes which match in the 1st and 3rd entry, and match in the 2nd and 4th entry. Thus, the distributions in ${\cal S}_{1,1}\cap{\cal S}_{2,1}$ can only have support on $\{0000,0101,1010,1111\}$. Thus, at most the intersection could be a 3 dimensional set, due to the normalization constraint. Next, consider the linear constraints build into ${\cal S}_{1,1}$. Letting

[TABLE]

the constraints can be written as $a+c=1$ and $b+d=1$. Similarly, the distribution in ${\cal S}_{2,1}$ must satisfy $a+b=1$ and $c+d=1.$ These constraints require that $a=d,$ and $b=c.$ The distributions in ${\cal S}_{1,1}\cap{\cal S}_{2,1}$ must take the form

[TABLE]

Letting $\alpha=\frac{a}{2}=\frac{d}{2}$ and $\beta=\frac{b}{2}=\frac{c}{2},$ the distributions in ${\cal S}_{1,1}\cap{\cal S}_{2,1}$ can be written as in Eqn. (19),

[TABLE]

where $\alpha+\beta=\frac{1}{2}$. By considering the other intersections, ${\cal S}_{1,1}\cap{\cal S}_{2,2},\;{\cal S}_{1,2}\cap{\cal S}_{2,1},\;{\cal S}_{1,2}\cap{\cal S}_{2,2},$ the other 3 transportation polytopes in Example 19 are recovered. ∎

Example 29**.**

Consider the case of $n=3$ pairs of $N=2$ valued variables $(X_{1},X_{2},X_{3},Y_{1},Y_{2},Y_{3})$. For any $(i,j)\in\{1,2,3\}\times\{1,2\}$, let ${\cal S}_{i,j}$ denote the set of joint distributions whose $(X_{i},Y_{i})$ margin is the $j$-th maximizer of the mutual information. Each ${\cal S}_{i,j}$ contains distributions supported on $2^{6-1}=32$ binary vectors. The intersections ${\cal S}_{1,j_{1}}\cap{\cal S}_{2,j_{2}}$ contain distributions supported on $2^{6-2}=16$ vectors, and when intersecting ${\cal S}_{1,j_{1}}\cap{\cal S}_{2,j_{2}}\cap{\cal S}_{3,j_{3}}$, we obtain distributions supported on $8$ binary vectors.

For concreteness, let ${\cal S}_{i,1}$ correspond to $p(X_{i},Y_{i})=\frac{1}{2}(\delta_{00}+\delta_{11})$. Then ${\cal S}_{1,1}$ is the set of joint distributions of the form

[TABLE]

where the numbers $q(x_{2},x_{3},y_{2},y_{3}|x_{1},y_{1})$ define an arbitrary conditional distribution. Each of these $p$ is supported on strings $(x_{1},x_{2},x_{3},y_{1},y_{2},y_{3})$ that satisfy $x_{1}=y_{1}$.

The distributions $p\in{\cal S}_{1,1}\cap{\cal S}_{2,1}\cap{\cal S}_{3,1}$ are supported on strings $(x_{1},x_{2},x_{3},y_{1},y_{2},y_{3})$ that satisfy $x_{i}=y_{i}$ for all $i=1,2,3$. In turn, they have the form

[TABLE]

The coefficients correspond to conditional probability distributions in three different ways. One way is as the conditional probabilities of $X_{2},X_{3},Y_{2},Y_{3}$ given $X_{1},Y_{2}$, with

[TABLE]

which corresponds to (31) and implies that $a+b+c+d=1$ and $e+f+g+h=1$. The other two ways result from conditioning on $X_{2},Y_{2}$ and $X_{3},Y_{3}$, respectively, and imply two more pairs of linear constraints. The six constraints form the linear system $M{\bf q}={\bf 1}$ with

[TABLE]

The matrix $M$ is simply the map computing the $(X_{i},Y_{i})$-margins, restricted to the support points (i.e., strings with $x_{i}=y_{i}$ for $i=1,2,3$). It can be regarded as a sufficient statistics matrix of a binary independence model of $n$ variables, including the statistic that computes the total mass. The matrix has rank $4$ and the vector $\frac{1}{4}{\bf 1}\in{\mathbb{R}}^{8}$ is a particular solution of the linear system. At this particular solution, one obtains a maximizer of $MI({\bf X},{\bf Y}).$

As discussed in Example 26, the space of solutions is 4 dimensional. To obtain all solutions, we consider the kernel of $M$, which is spanned by the rows of

[TABLE]

Any linear combination of these rows can be added to $\frac{1}{4}{\bf 1}\in{\mathbb{R}}^{8}$ to obtain another vector that satisfies the marginal equality constraints. The intersection of the $4$-dimensional affine space with the probability simplex is obtained by requiring that the solutions have non-negative entries. This results in a polytope. We compute the vertex representation using the free open-source mathematics software Sage [41], and obtain the $6$ rows

[TABLE]

Based off of the vertex representation above, the distributions which are in ${\cal S}={\cal S}_{1,1}\cap{\cal S}_{2,1}\cap{\cal S}_{3,1}$ are of the form

[TABLE]

where $\sum_{i}\alpha_{i}=1$ ensures normalization and $\alpha_{i}\geq 0$ for $i=1,\dots,6$. One can quickly check that $p(X_{i},Y_{i})=\frac{1}{2}(\delta_{00}+\delta_{11})$ for $i=1,2,3$. The vertices $\alpha_{1}=1$, $\alpha_{2}=1$, $\alpha_{3}=1$, $\alpha_{4}=1$ are the only distributions with support of cardinality two and they are also the only maximizers of multi-information within ${\cal S}$. The choices $\alpha_{1}=\alpha_{2}=\alpha_{3}=\alpha_{4}=1/4$ and $\alpha_{5}=\alpha_{6}=1/2$ give the same distribution, which is also the only maximizer of $MI({\bf X},{\bf Y})$ within ${\cal S}$. This observation motivates Theorem 17 which states that each polytope of SFMI maximizers contains one or more simplices, and at the center of each simplex lies a $MI({\bf X},{\bf Y})$ maximizer. Furthermore, when restricted to the set of SFMI maximizers, the distributions with maximum entropy are $MI({\bf X},{\bf Y})$ maximizers.

We have discussed one of the $N!^{n}=2!^{3}=8$ polytopes. The other $7$ are similar. Each of them has $4$ vertices that are maximizers of the multi-information and form a $3$-simplex with a maximizer of $MI({\bf X},{\bf Y})$ at its center. The $8$ centers are following vectors scaled by $1/8$, whereby grouped terms correspond to the vertices:

[TABLE]

One can see that the simplices are disjoint. In fact, distributions from different simplices have disjoint supports, meaning that the simplices sit in disjoint faces of the joint probability simplex. These simplices having disjoint support only occurs when dealing with binary variables. ∎

Example 30**.**

Consider the case of $n=2$ pairs of $N=3$ valued variables. For each $i=1,2$ let ${\cal S}_{i,{j_{i}}}$ be the set of joint distributions which have marginal $p(X_{i},Y_{i})$ as one of the $N!=3!=6$ multi-information maximizers, so $j_{i}\in\{1,\dots,6\}$. For concreteness, let the correspondence be

[TABLE]

There are $N!^{n}=3!^{2}=36$ intersections of the form ${\cal S}_{1,{j_{1}}}\cap{\cal S}_{2,{j_{2}}}$, each corresponding to a transportation polytope. They together contain all $N!^{2n-1}=3!^{2\cdot 2-1}=216$ maximizers of the multi-information. Consider the support sets of three to start:

[TABLE]

The probabilities assigned to the joint states are not arbitrary, since they need to satisfy the margins. One particular subset of distributions in ${\cal S}_{1,1}\cap{\cal S}_{2,1}$ can be written as

[TABLE]

where $\alpha_{1}+\alpha_{2}+\alpha_{3}=1$. This is a simplex whose $N^{n-1}=3^{2-1}=3$ vertices maximize multi-information. Another one can be written as

[TABLE]

This is another simplex whose $3$ vertices maximize multi-information. Both simplices have the same centroid, with $\alpha_{1}=\alpha_{2}=\alpha_{3}=1/3$, which is a maximizer of $MI({\bf X},{\bf Y})$.

The other transportation polytopes (intersections ${\cal S}_{1,{j_{1}}}\cap{\cal S}_{2,{j_{2}}}$) are similar. For example, the two simplices in ${\cal S}_{1,1}\cap{\cal S}_{2,2}$ are

[TABLE]

and the two simplices in ${\cal S}_{1,1}\cap{\cal S}_{2,3}$ are

[TABLE]

Observe that there are $N!^{n-1}=3!^{2-1}=6$ multi-information maximizers per transportation polytope, and there are $N!^{n}$ transportation polytopes total. This number is expected because all of the $N!^{2n-1}$ maximizers of multi-information are SFMI maximizers. ∎