Combinatorial Entropy Power Inequalities: A Preliminary Study of the Stam region

TL;DR

This paper introduces the Stam region, a new mathematical construct related to entropy powers of subset sums of independent random vectors, and explores its properties and bounds.

Contribution

It establishes that fractionally superadditive set functions bound the Stam region and proves the non-supermodularity of entropy power sums, advancing understanding of entropy inequalities.

Findings

Fractionally superadditive functions bound the Stam region.

Entropy power of sums is not supermodular in any dimension.

The closure of the Stam region is a logarithmically convex cone.

Abstract

We initiate the study of the Stam region, defined as the subset of the positive orthant in $\mathbb{R}^{2^n-1}$ that arises from considering entropy powers of subset sums of $n$ independent random vectors in a Euclidean space of finite dimension. We show that the class of fractionally superadditive set functions provides an outer bound to the Stam region, resolving a conjecture of A. R. Barron and the first author. On the other hand, the entropy power of a sum of independent random vectors is not supermodular in any dimension. We also develop some qualitative properties of the Stam region, showing for instance that its closure is a logarithmically convex cone.

Equations

$$h(X)=-\int f(x)\log f(x)dx,$$

$$f(x)=\frac{1}{x(\log x{)}^2},$$

$$\mathcal{N}(X)=e^{2h(X)/d}.$$

$$h(X)\le h(N(0,K))=\frac{1}{2}\log [(2\pi e{)}^d\det (K)]<+\infty ,$$

$$Y_s=\sum _{i\in s}{X}_i.$$

$$\begin{array}{rl}{\Gamma }_d(n)& =\{[\mathcal{N}(Y_s){]}_{s\subset [n]}:X_1,X_2,\dots ,X_n\\ & \text{are independent Rd-valued random vectors}\\ & \text{with h(X1+…+Xn)<∞}\}.\end{array}$$

$$\displaystyle\nu_{\bf f}(s)=\mathcal{N}\bigg{(}\sum_{i\in s}X_{i}\bigg{)};$$

$${\Gamma }_d(n)=\{{\nu }_{\mathbf{f}}:\mathbf{f}\in {\mathbf{F}}_{d,n}\}.$$

$$\Gamma (n)={\cup }_{d\in \mathbb{N}}{\Gamma }_d(n).$$

$$v([n])\ge \sum _{s\in \mathcal{G}}{\beta }_{s}v(s)$$

$$v(s\cup t)+v(s\cap t)\ge v(s)+v(t)$$

$$\bar{h}(X)=\underset{d\to \infty }{\lim }\frac{h(X_1,\dots ,X_d)}{d},$$

$$\displaystyle\nu_{\bf X}(s)=\exp{\bigg{\{}2\bar{h}\bigg{(}\sum_{i\in s}X^{(i)}\bigg{)}\bigg{\}}}.$$

$${\Gamma }_{\infty }(n)=\{{\nu }_{\mathbf{X}}:\mathbf{X}\in {\mathbf{F}}_n\}.$$

$${\tilde{\Gamma }}_d(n)=\{{\nu }_{\mathbf{f}}:\mathbf{f}\in {\tilde{\mathbf{F}}}_{d,n}\},$$

$$\tilde{\Gamma }(n)={\cup }_{d\in \mathbb{N}}{\tilde{\Gamma }}_d(n);$$

$$v([n])\ge \frac{1}{r}\sum _{s\in \mathcal{G}}v(s)$$

$$v([n])\ge \sum _{s\in \mathcal{G}}{\beta }_{s}v(s)$$

$$\displaystyle\mathcal{B}=\bigg{\{}\beta:2^{[n]}\setminus\phi\rightarrow{\mathbb{R}}_{+}\bigg{|}\sum_{s\subset[n]\setminus\phi}\beta_{s}\hbox{\rm{1}}_{s}=\hbox{\rm{1}}_{[n]}\bigg{\}}.$$

$$\displaystyle\mathcal{B}^{\prime}=\bigg{\{}\beta:2^{[n]}\setminus\phi\rightarrow{\mathbb{R}}\bigg{|}\sum_{s\subset[n]\setminus\phi}\beta_{s}\hbox{\rm{1}}_{s}=\hbox{\rm{1}}_{[n]}\bigg{\}}$$

$$M_{\mathcal{G}}\beta =1,$$

$$v_n([n])\ge \frac{1}{R}\sum _{s\in \mathcal{G}}{c}_s{v}_n(s),$$

$$v_n([n])\ge \frac{1}{R}\sum _{s\in {\mathcal{G}}^{\prime \prime }}{v}_n(s),$$

$$f(x)+f(y)\le f(x\vee y)+f(x\wedge y)$$

$$\frac{{\partial }^{2}f(x)}{\partial x_j\partial x_i}\ge 0$$

$$g(x_i,x_j)=f(x).$$

$$\begin{array}{rl}g(y_i,z_j)-g(y_i,y_j)& =g(y_i,z_j)-g(y_i\wedge z_i,y_j\wedge z_j)\\ & \le g(y_i\vee z_i,y_j\vee z_j)-g(z_i,y_j)\\ & =g(z_i,z_j)-g(z_i,y_j);\end{array}$$

$$\frac{{\partial }^{2}f(x)}{\partial x_j\partial x_i}\ge 0$$

$$f(a,x_i,x_j)-f(a,x_i,\widetilde{x_j})$$

Full text

I Introduction

For a ${\mathbb{R}}^{d}$-valued random vector $X$ with density $f$ with respect to the Lebesgue measure on ${\mathbb{R}}^{d}$, the differential entropy is

[TABLE]

if it exists. When $X$ is supported on a strictly lower-dimensional set than ${\mathbb{R}}^{d}$ (and hence does not have a density with respect to the $d$-dimensional Lebesgue measure), a limiting argument suggests that one should set $h(X)=-\infty$. Also, by considering for example the distribution supported on $(e,\infty)\subset{\mathbb{R}}$ with density

[TABLE]

it is seen that the entropy can take the value $h(X)=+\infty$.

Unless explicitly mentioned, we limit ourselves to random vectors $X$ with $h(X)<+\infty$. The entropy power of $X$ is defined by

[TABLE]

Then, $\mathcal{N}(X)\in{\mathbb{R}}_{+}:=[0,\infty)$ is a non-negative real number. If a random vector $X$ has a well-defined, finite covariance matrix $K$ (i.e., the variance of each of the $d$ components is finite), then it is well known that

[TABLE]

because of the fact that Gaussians maximize entropy under a constraint on the covariance matrix. Thus the class of random vectors under consideration certainly includes all those whose components have finite variances, but in fact is much larger since it also includes many heavy-tailed distributions.

There are two main motivations for considering entropy power inequalities: the first comes from the fact that it is related to probabilistic isoperimetric phenomena including the entropic central limit theorem (see, e.g., Barron [3]), and the second comes from the fact that it can be extremely useful in the study of rate and capacity regions in multi-user information theory (see, e.g., Shannon [49], Bergmans [5], Ozarow [47], Costa [12] and Oohama [46]).

Let $X_{1},X_{2},\ldots,X_{n}$ be independent random vectors. We write $[n]$ for the index set $\{1,2,\ldots,n\}$, and $\phi$ for the empty set. For any nonempty $s\subset[n]$, define the subset sum

[TABLE]

One is interested in the entropy powers $\mathcal{N}(Y_{s})$ of the subset sums, which leads naturally to the following objects of study:

[TABLE]

We now give a more precise definition of the relevant objects.

Definition 1**.**

Let ${\bf F}_{d,n}$ be the collection of all $n$-tuples ${\bf f}=(f_{1},\ldots,f_{n})$ of probability density functions $f_{i}$ on ${\mathbb{R}}^{d}$, such that if $X_{i}\sim f_{i}$ are independent, their sum has finite entropy (i.e., $h(X_{1}+\ldots+X_{n})<\infty$). Define the set function $\nu_{\bf f}:2^{[n]}\setminus\phi\rightarrow{\mathbb{R}}_{+}$ by

[TABLE]

we will also denote $\nu_{\bf f}$ by $\nu_{(X_{1},\ldots,X_{n})}$ when convenient. The $d$-dimensional Stam region is the subset of ${\mathbb{R}}^{2^{n}-1}$ given by

[TABLE]

The Stam region is defined by

[TABLE]

We name these regions after A. J. Stam in honor of his pioneering role [53] in the study of entropy power and its applications. One can extend the domain of $\nu_{\bf f}$ to the full Boolean lattice $2^{[n]}$ in a natural way by setting $\nu_{\bf f}(\phi)=0$ for every ${\bf f}\in{\bf F}_{d,n}$; this would make the Stam region a subset of ${\mathbb{R}}^{2^{n}}$ instead of ${\mathbb{R}}^{2^{n}-1}$, but we avoid such an extension since it is trivial.

Any inequality that relates entropy powers of different subset sums (usually called an “entropy power inequality” or EPI) gives a bound on the Stam region. Conversely, knowing the Stam region is equivalent, in principle, to knowing all EPI’s that hold and all that do not.

The objective of this note is to develop a better understanding of the Stam region. Our first result is the best outer bound known thus far for it. In particular, we show that for any dimension $d$, the entropy power is “fractionally superadditive”, settling a conjecture made implicitly in [33] and explicitly in [30]. We first explain what the term means.

Let $\mathcal{G}$ be a hypergraph on $[n]$, i.e., let $\mathcal{G}$ be a collection of nonempty subsets of $[n]$. Given $\mathcal{G}$, a function $\beta:\mathcal{G}\to{\mathbb{R}}_{+}$ is a fractional partition, if for each $i\in[n]$, we have $\sum_{s\in\mathcal{G}:i\in s}\beta_{s}=1$. If there exists a fractional partition $\beta$ for $\mathcal{G}$ that is $\{0,1\}$-valued, then $\beta$ is the indicator function for a partition of the set $[n]$ using a subset of $\mathcal{G}$; hence the terminology. Note that we do not need to make reference to a hypergraph $\mathcal{G}$ in order to define a fractional partition; the domain can always be extended to all nonempty subsets of $[n]$ by setting $\beta_{s}=0$ for $s\notin\mathcal{G}$.

We say that a set function $v:2^{[n]}\rightarrow{\mathbb{R}}_{+}$ is fractionally superadditive if

[TABLE]

holds for every fractional partition $\beta$ using any hypergraph $\mathcal{G}$. Write $\Gamma_{FSA}(n)$ for the class of all fractionally superadditive set functions $v$ with $v(\phi)=0$.

Theorem 1**.**

For every $n\in\mathbb{N}$, $\Gamma(n)\subset\Gamma_{FSA}(n)$.

The set function $v:2^{[n]}\rightarrow{\mathbb{R}}_{+}$ is said to be supermodular if

[TABLE]

for all sets $s,t\subset[n]$. Write $\Gamma_{SM}(n)$ for the class of all supermodular set functions $v$ with $v(\phi)=0$.

It is known [45, 41] that $\Gamma_{SM}(n)\subsetneq\Gamma_{FSA}(n)$. In [30], it was asked whether in fact the entropy power is supermodular, i.e., whether the set function $\nu_{\bf f}\in\Gamma_{SM}(n)$ for every ${\bf f}\in{\bf F}_{d,n}$. Our second result is to show that the answer to this question is no in general.

Theorem 2**.**

For any $n\geq 3$, $\Gamma(n)\nsubseteq\Gamma_{SM}(n)$.

The Stam region has some pleasing geometric properties. Unfortunately we have not yet been able to determine if the closure of the Stam region is convex; however it does have some restricted convexity properties. We say that a set $A\subset{\mathbb{R}}_{+}^{k}$ is logarithmically convex if for any two points $x=(x_{1},\ldots,x_{k})$ and $y=(y_{1},\ldots,y_{k})$ in $A$, the point $(x_{1}^{\lambda}y_{1}^{1-\lambda},\ldots,x_{k}^{\lambda}y_{k}^{1-\lambda})\in A$ for each $\lambda\in[0,1]$. We say that a set $A\subset{\mathbb{R}}^{k}$ is orthogonally convex if any line segment parallel to any of the coordinate axes connecting two points of $A$ lies totally within $A$. It is trivial to check that any subset of ${\mathbb{R}}_{+}^{k}$ that is logarithmically convex is necessarily orthogonally convex.

Theorem 3**.**

The closure $\overline{\Gamma(n)}$ of the Stam region $\Gamma(n)$ is a logarithmically convex cone in ${\mathbb{R}}^{2^{n}-1}$. Consequently, $\overline{\Gamma(n)}$ is orthogonally convex.

For the case where one has only two random variables, we can give a complete description of the Stam region.

Theorem 4**.**

$\overline{\Gamma(2)}=\Gamma_{FSA}(2)$. In particular, $\overline{\Gamma(2)}$ is a closed, convex, polyhedral cone in ${\mathbb{R}}_{+}^{3}$.

The general problem is significantly more complicated.

Theorem 5**.**

For $n\geq 3$, $\overline{\Gamma(3)}\subsetneq\Gamma_{FSA}(3)$.

The closure of the Stam region, which features in the preceding three theorems, is in fact closely related to the “Stam region” involving entropy rates of a class of stochastic processes. By a ${\mathbb{R}}$-valued stochastic process, we will mean a discrete-time stochastic process $X=(X_{1},X_{2},\ldots)$, with each $X_{i}$ taking values in ${\mathbb{R}}$. (More precisely, we may interpret this process as a measurable function from the underlying probability space to the set ${\mathbb{R}}^{\mathbb{N}}$ equipped with the cylindrical $\sigma$-algebra.) Recall that the entropy rate of a ${\mathbb{R}}$-valued stochastic process is defined by

[TABLE]

if the limit exists.

Definition 2**.**

Let ${\bf F}_{n}$ be the collection of all $n$-tuples ${\bf X}=(X^{(1)},\ldots,X^{(n)})$ of independent ${\mathbb{R}}$-valued stochastic processes, such that each $X^{(i)}$ as well as their sum has finite entropy rate (i.e., $\bar{h}(X^{(1)}+\ldots+X^{(n)})<\infty$). Define the set function $\nu_{\bf X}:2^{[n]}\setminus\phi\rightarrow{\mathbb{R}}_{+}$ by

[TABLE]

*The $\infty$-dimensional Stam region is the subset of ${\mathbb{R}}^{2^{n}-1}$ given by *

[TABLE]

Now we can relate the finite-dimensional and infinite-dimensional Stam regions.

Theorem 6**.**

For any $n\in\mathbb{N}$, $\overline{\Gamma(n)}=\Gamma_{\infty}(n)$.

This gives a pleasing “physical” interpretation of the closure of the Stam region– the closure can be thought of not just as a topological operation on a set that has intrinsic probabilistic motivation, but as an extension of the definition of the Stam region to random vectors of dimension $\infty$.

Let us note that the restriction $h(X_{1}+\ldots+X_{n})<\infty$ in the definition of the Stam region is not essential. One can define the extended $d$-dimensional Stam region as

[TABLE]

where $\tilde{\bf F}_{d,n}$ is the collection of all $n$-tuples ${\bf f}=(f_{1},\ldots,f_{n})$ of probability density functions $f_{i}$ on ${\mathbb{R}}^{d}$ such that $h(f_{i})$ exists for each $i\in[n]$. The extended Stam region would then be defined by

[TABLE]

both $\tilde{\Gamma}_{d}(n)$ and $\tilde{\Gamma}(n)$ are then subsets of $\bar{{\mathbb{R}}}_{+}^{2^{n}}$, with the extended nonnegative real numbers $\bar{{\mathbb{R}}}_{+}=[0,\infty]$ replacing ${\mathbb{R}}_{+}$. All our theorems, with straightforward minor modifications, can be stated for the extended Stam regions.

After developing some combinatorial preliminaries in Section II, we prove the above theorems in Section III. Although Theorem 2 asserts that supermodularity is false for the Stam region, the proof given in Section III does not shed light on the whether supermodularity might in fact hold in the one-dimensional case (real-valued random variables); this question is discussed in Section IV. Section V contains some additional remarks of possible interest.

II Combinatorial Preliminaries

The preliminaries we will need range from a lemma about fractional superadditivity that we could not find explicitly stated in the literature, to some facts about supermodular functions on lattices.

We now state and prove a useful lemma about fractional superadditivity that is at the heart of our proof of Theorem 1. We will use in the proof the following standard fact about a vertex or extreme point of a polytope: a point in a polytope is an extreme point if and only if it is the unique meeting point of several faces (i.e., a set of faces intersects in a singleton, containing that point). Indeed, if it were not unique, then it could not be an extreme point, since there would be a line segment in the polytope containing it.

Proposition 1**.**

[A sufficient condition for Fractional Superadditivity]* Consider a set function $v:2^{[n]}\rightarrow{\mathbb{R}}_{+}$. Let $\mathcal{G}$ be a $r$-regular multihypergraph on $[n]$, i.e., let $\mathcal{G}$ be a collection of subsets of $[n]$ (possibly repeated), such that every index $i$ lies in exactly $r$ of the elements of $\mathcal{G}$. Suppose $v$ satisfies, for any $r\in\mathbb{N}$,*

[TABLE]

where $\mathcal{G}$ is any $r$-regular multihypergraph on $[n]$. Then,

[TABLE]

holds for every fractional partition $\beta$ using any multihypergraph $\mathcal{G}$ on $[n]$.

Proof.

Denote by $\hbox{\rm{1}}_{s}$ the indicator function of the subset $s$ defined on the domain $[n]$, i.e., $\hbox{\rm{1}}_{s}(i)=1$ if $i\in s$, and $\hbox{\rm{1}}_{s}(i)=0$ otherwise. Consider the space of all fractional partitions on $[n]$, i.e.,

[TABLE]

Clearly, $\mathcal{B}$ can be viewed as a subset of the Euclidean space of dimension $2^{n}-1$ (since each point of it is defined by $2^{n}-1$ real numbers). Furthermore, $\mathcal{B}=\mathcal{B}^{\prime}\cap\mathcal{O}_{+}$, where

[TABLE]

is an affine subspace of dimension $2^{n}-1-n$ and $\mathcal{O}_{+}=\{\beta|\beta_{s}\geq 0\,\,\forall s\in 2^{[n]}\setminus\phi\}$ is the closed positive orthant. In particular, $\mathcal{B}$ is a non-empty, compact, convex set (in fact, a closed polytope), so that by the Krein-Milman theorem, $\mathcal{B}$ is the convex hull of its extreme points. Thus to prove (5) for every $\beta\in\mathcal{B}$, it is sufficient to prove (5) for every $\beta\in\text{Ex}(\mathcal{B})$, where $\text{Ex}(\mathcal{B})$ denotes the set of extreme points of $\mathcal{B}$.

For a fractional partition $\beta$, its support is defined as the collection of subsets $s$ of $[n]$ such that $\beta_{s}>0$. Given a hypergraph $\mathcal{G}$, the set of fractional partitions supported by $\mathcal{G}$ is clearly the set of positive solutions of the linear equation

[TABLE]

where $M_{\mathcal{G}}$ is the $n\times|\mathcal{G}|$ 0-1 matrix defined by $M_{i,s}=\hbox{\rm{1}}_{i\in s}$ for $i\in[n],s\in\mathcal{G}$, and 1 is the column vector in ${\mathbb{R}}^{n}$ consisting of all ones. In general, the number of fractional partitions supported by $\mathcal{G}$ could be either 0, 1 or infinite, depending on $\mathcal{G}$.

Every face of the polytope $\mathcal{B}$ corresponds to one of the inequality constraints being tight (i.e., $\beta_{s}=0$ for at least one set $s$). Given an extreme point $\beta$ of the polytope $\mathcal{B}$, we know it is the unique meeting point of several faces; let these faces correspond to setting $\beta_{s}=0$ for $s$ lying in the collection of sets $\mathcal{G}^{\prime}$. Then the complement $\mathcal{G}$ of $\mathcal{G}^{\prime}$ is the support of $\beta$, and $\beta$ must be the unique fractional partition supported by $\mathcal{G}$. By the previous paragraph, we know this means that $\beta$ is the unique, strictly positive solution of the equation (6). Consequently one must have $|\mathcal{G}|\leq n$, with precisely $|\mathcal{G}|$ of the $n$ rows of $M_{\mathcal{G}}$ being linearly independent, so that $M_{\mathcal{G}}$ has full rank. Since $M_{\mathcal{G}}$ has only integer entries, a little bit of thought shows that performing Gaussian elimination to solve (6) will result in its unique solution having only rational entries. Thus we have deduced that any fractional partition in $\text{Ex}(\mathcal{B})$ has rational111Another way of seeing this would be to observe that $\beta=M_{\mathcal{G}}^{+}\hbox{\rm{1}}$, where $M_{\mathcal{G}}^{+}$ is the Moore-Penrose pseudo-inverse of $M$. Then the rationality of entries of $M_{\mathcal{G}}$ implies that of the entries of $M_{\mathcal{G}}^{+}$, and hence the rationality of $\beta$. coefficients.

By writing all the coefficients of $\beta$ with a common denominator, one sees that (5) may be written as

[TABLE]

where $c_{s}$ is a positive integer. One may write this as

[TABLE]

where $\mathcal{G}^{\prime\prime}$ is the multihypergraph with $c_{s}$ copies of the set $s$. Note that $\mathcal{G}^{\prime\prime}$ is clearly $R$-regular.     $\Box$

Remark 1**.**

The argument we use to characterize $\text{Ex}(\mathcal{B})$ is similar to that in Gill and Grünwald [20] and seems also implicit in Friedgut and Kahn [17]; it likely goes back to the early literature on cooperative game theory (see, e.g., Shapley [50]), although we were unable to find the precise statement we wanted in these early references. Incidentally a fractional partition is interpreted in [19, 20] as a “coarsening at random” or CAR mechanism; this is a probabilistic rule that replaces any point $x$ in the ground set $[n]$ with a subset $A$ of $[n]$ containing $x$, in such a way that the probability of observing $A$ is the same for all $x$ that are contained in $A$.

The last observation we need concerns supermodular functions; first we need a definition of supermodularity for functions on ${\mathbb{R}}^{n}$ and not just set functions.

Definition 3**.**

A function $f:{\mathbb{R}}_{+}^{n}\rightarrow{\mathbb{R}}$ is supermodular if

[TABLE]

for any $x,y\in\mathbb{R}_{+}^{n}$, where $x\vee y$ denotes the componentwise maximum of $x$ and $y$ and $x\wedge y$ denotes the componentwise minimum of $x$ and $y$.

The fact that supermodular functions are closely related to functions with increasing differences is classical (see, e.g., [55], which describes more general results involving arbitrary lattices), but we give the proof for completeness.

Proposition 2**.**

Suppose a function $f:{\mathbb{R}}_{+}^{n}\rightarrow{\mathbb{R}}$ is in $C^{2}$, i.e., it is twice-differentiable with a continuous Hessian matrix. Then $f$ is supermodular if and only if

[TABLE]

for every distinct $i,j\in[n]$, and for any $x\in{\mathbb{R}}_{+}^{n}$.

Proof.

Suppose $f:{\mathbb{R}}^{n}\rightarrow{\mathbb{R}}$ is supermodular. Let $i,j\in[n]$ with $i<j$, and pick a set of coordinates $a_{k}$ for each $k\in A$, where $A=[n]\setminus\{i,j\}$. For any $x\in{\mathbb{R}}^{n}$ with $x_{k}=a_{k}$ for each $k\in A$, set

[TABLE]

Then for $y_{i}<z_{i}$ and $y_{j}<z_{j}$,

[TABLE]

in other words, differences of $g$ (and hence of $f$) in the $j$-th coordinate are increasing in the $i$-th coordinate. If $f$ is $C^{2}$, this implies that

[TABLE]

on the interior of the domain. Since the same argument can be repeated for each pair $i,j$ of distinct indices, we obtain non-negativity of all off-diagonal entries of the Hessian matrix of $f$.

Now suppose

[TABLE]

for any $x\neq 0$. By standard calculus, this implies that $f$ has increasing differences, in the sense that for any pair $i\neq j$ of distinct indices, and for every $a=(a_{k}:k\in A)$ where $A=[n]\setminus\{i,j\}$, one has that differences in the $j$-th coordinate, namely

[TABLE]

with $x_{j}>\tilde{x_{j}}$, are increasing as functions of $x_{i}$. (We have abused notation for convenience here since $f$ may not be symmetric in its arguments, but we are implicitly assuming that $x_{i}$ is the $i$-th argument and ${x_{j}}$ is the $j$-th argument when we write $f(a,x_{i},x_{j})$, etc.) Thus we have

[TABLE]

where the inequality follows from the increasing differences property of $f$. Now we can apply the trivial fact that for any function $g$ on the real line, one has $g(x)+g(y)=g(x\vee y)+g(x\wedge y)$ to deduce that the last expression equals

[TABLE]

which telescopes down to $f(y\vee z)-f(z)$. Combining the preceding two displays, we have proved that

[TABLE]

which is precisely supermodularity of $f$.     $\Box$

Our final very simple lemma connects supermodularity for functions defined on ${\mathbb{R}}_{+}^{n}$ to supermodularity for set functions. For a set $s\subset[n]$, we use $e(s)$ to denote the vector in ${\mathbb{R}}^{n}$ such that for each $i\in[n]$, the $i$-th coordinate of $e(s)$ is 1 if and only if $i\in s$ (or in other words, $e_{i}(s):=\hbox{\rm{1}}_{s}(i)$).

Lemma 1**.**

If $f:{\mathbb{R}}_{+}^{n}\rightarrow{\mathbb{R}}$ is supermodular, and we set $\bar{f}(s):=f(e(s))$ for each $s\subset[n]$, then $\bar{f}$ is a supermodular set function.

Proof.

Observe that

[TABLE]

$\Box$

III Proofs

III-A Proof of Theorem 1

In [33], the following EPI for an arbitrary multihypergraph $\mathcal{G}$ on $[n]$ was demonstrated. If $r$ is the maximum number of subsets in $\mathcal{G}$ in which any one index $i$ can appear, for $i=1,\ldots,n$, then

[TABLE]

In other words, for any ${\bf f}\in{\bf F}_{d,n}$, $\nu_{\bf f}$ satisfies the assumption in Proposition 1, and hence we have that $\nu_{\bf f}$ is fractionally superadditive.

III-B Proof of Theorem 2

Observe that if the function $\nu_{\bf f}(s)=\mathcal{N}(\sum_{i\in s}X_{i})$ were supermodular, then specializing to multivariate Gaussians would imply a similar supermodularity for the $d$-th root of the determinant of sums of positive definite matrices. To be precise, consider the set function

[TABLE]

where $M_{1},M_{2},\cdots,M_{n}$ are $d\times d$ positive definite matrices. (Here we use $\nu_{G}$ to indicate that this is the function $\nu_{\bf f}$ specialized to Gaussians, i.e., when $f_{i}$ is the density of the $N(0,M_{i})$ distribution.) We will show that $\nu_{G}$ is not supermodular for $d=2$.

First proof. We exhibit a numerical counterexample. Consider

[TABLE]

Then it is easily seen that

[TABLE]

Observe that when $\epsilon$ is small, ${\mathop{\rm det}}(A+C)^{1/2}=1+1.25\epsilon+o(\epsilon)$, so that we can arrange for ${\mathop{\rm det}}(A+B)^{1/2}+{\mathop{\rm det}}(A+C)^{1/2}=3.5+1.25\epsilon+o(\epsilon)$ to exceed ${\mathop{\rm det}}(A)^{1/2}+{\mathop{\rm det}}(A+B+C)^{1/2}=3.5+\epsilon$.

Second proof. We present a second, more complicated, proof in an attempt to give some additional insight into why supermodularity fails. To prove that $\nu_{G}$ is not supermodular, we first show that its continuous analogue is not supermodular (in the continuous sense). In other words, let $v:{\mathbb{R}}_{+}^{n}\rightarrow{\mathbb{R}}_{+}$ be defined by

[TABLE]

where $M_{1},M_{2},\cdots,M_{n}$ are $d\times d$ positive definite matrices. We will show that the function $v$ is not supermodular, i.e., there are $x,x^{\prime}\in{\mathbb{R}}_{+}^{n}$ such that the following inequality is violated

[TABLE]

where $x\vee x^{\prime}$ denotes the componentwise maximum and $x\wedge x^{\prime}$ denotes the componentwise minimum of $x$ and $x^{\prime}$.

To show that (9) is violated, it suffices to show, thanks to Proposition 2, that $v$ is $C^{2}$ (which is immediate from the fact that the determinant of a linear combination of matrices is a polynomial in the coefficients) and that

[TABLE]

for some $x\in{\mathbb{R}}_{+}^{n}$. Setting $M=\sum_{k\in[n]}x_{k}M_{k}$, we note that

[TABLE]

However, it is easy to show (see, e.g., Zhang [61], page 166) that there are ${d}\times{d}$ positive definite matrices $A$ and $B$ for which

[TABLE]

Hence, the last term in (11) can be negative. As a numerical example, consider ${d}=2$ and

[TABLE]

It then holds that $\frac{1}{2}{\rm tr\,}(A){\rm tr\,}(B)=18$ whereas ${\rm tr\,}(AB)=19$.

Finally, note that the violation of supermodularity in the discrete domain follows from this result of violation of supermodularity in continuous domain. Indeed, we have shown that there is $x\in{\mathbb{R}}_{+}^{n}$ such that (10) is true. It then follows that there are $s_{i},s_{j}>0$ such that

[TABLE]

Consider $A=s_{i}M_{i}$, $B=s_{j}M_{j}$, $C=\sum_{k\in[n]}x_{k}M_{k}$. Evidently, $A,B,C$ are positive definite. However, according to the above inequality and the definition of the function $v$, we have shown that

[TABLE]

so that neither $\nu_{G}$ nor $\nu$ is supermodular.

III-C Proof of Theorem 3

The fact that each $\Gamma_{d}(n)$ is a cone follows just from scaling. By definition, any point $z\in\Gamma_{d}(n)$ is just $\nu_{(X_{1},\ldots,X_{n})}$ for some independent random vectors $X_{1},\ldots,X_{n}$ (each of which is in ${\mathbb{R}}^{d}$). For any $\lambda>0$, the point $\lambda z$ is then simply $\nu_{(\sqrt{\lambda}X_{1},\ldots,\sqrt{\lambda}X_{n})}$ and hence also lies in $\Gamma_{d}(n)$. Since each $\Gamma_{d}(n)$ is a cone, so are $\Gamma(n)$ and $\overline{\Gamma(n)}$.

What remains is to show that $\overline{\Gamma(n)}$ is a logarithmically convex set. To do this, we will need the following lemma.

Lemma 2**.**

For any given $d,d^{\prime}\in\mathbb{N}$, the set

[TABLE]

is a dense subset of $[0,1]$.

Proof.

Letting $z(m,l)=\frac{md}{md+ld^{\prime}}$, the image $Q_{d,d^{\prime}}$ of $\mathbb{Z}_{+}\times\mathbb{Z}_{+}$ under $z$ is clearly contained in the interval $[0,1]$. Since $[0,1]$ is closed, we also have $\overline{Q_{d,d^{\prime}}}\subset[0,1]$; thus what remains to be proved is that every point of $[0,1]$ is in the closure of $Q_{d,d^{\prime}}$.

Let $x$ be an arbitrary rational number in $[0,1]$; then $x$ is of the form $\frac{p}{p+q}$, where $p,q\in\mathbb{Z}_{+}$. Writing

[TABLE]

we see that $x$ is in the image of $Q_{d,d^{\prime}}$ of $\mathbb{Z}_{+}\times\mathbb{Z}_{+}$ under $z$ by taking $l=qd$ and $m=pd^{\prime}$. In other words, we have $\mathbb{Q}\cap[0,1]\subset Q_{d,d^{\prime}}$, which– by the density of the rationals in the reals– implies that $[0,1]\subset\overline{Q_{d,d^{\prime}}}$.

Thus it is demonstrated that $\overline{Q_{d,d^{\prime}}}=[0,1]$.     $\Box$

Remark 2**.**

Lemma 2 is related to interesting problems in number theory, such as the coin problem (sometimes called the diophantine Frobenius problem), which goes back to 1882 and asks for the largest integer that cannot be written as a nonnegative linear combination of a given set of natural numbers. (A theorem of Schur ensures that there is such a largest integer.) The book of Ramírez Alfonsín [48] contains many more details.

To show that $\overline{\Gamma(n)}$ is a logarithmically convex set, let us start with two points in $\Gamma(n)$, say $\nu_{{\bf X}}:=\nu_{(X_{1},\ldots,X_{n})}\in\Gamma_{d}(n)$ and $\nu_{{\bf Y}}:=\nu_{(Y_{1},\ldots,Y_{n})}\in\Gamma_{d^{\prime}}(n)$, where our notation is as explained in Definition 1. Let ${\bf Z}^{(m,l)}$ be the vector formed by stacking $m$ independent copies of ${\bf X}$ with $l$ independent copies of ${\bf Y}$; in other words,

[TABLE]

where each $X_{i}^{(k)}\in{\bf R}^{d}$ and each $Y_{i}^{(k)}\in{\bf R}^{d^{\prime}}$. Thus, denoting the $i$-th column of ${\bf Z}^{(m,l)}$ by $Z_{i}^{(m,l)}$, we see that each $Z_{i}^{(m,l)}$ is a random vector in ${\bf R}^{md+ld^{\prime}}$. For each $s\subset[n]$, we have

[TABLE]

where the third equality follows from the assumed independence and the fact that the dimension appears in the exponent in the definition of entropy power.

Thus we have shown that if $a\in\Gamma_{d}(n)$ and $b\in\Gamma_{d^{\prime}}(n)$, then

[TABLE]

is a subset of $\Gamma(n)$. Invoking the density of $Q_{d,d^{\prime}}$ in $[0,1]$ (Lemma 2) and the continuity of the map $\lambda\mapsto a^{\lambda}b^{1-\lambda}$, we see that

[TABLE]

is a subset of $\overline{\Gamma(n)}$, which is precisely the claimed logarithmic convexity.

III-D Proof of Theorem 4

The main challenge here is to show that $\Gamma_{FSA}(2)\subset\overline{\Gamma(2)}$. Since $\Gamma_{FSA}(2)=\{(u_{\{1\}},u_{\{2\}},u_{\{1,2\}})\in{\mathbb{R}}_{+}^{3}:u_{\{1,2\}}\geq u_{\{1\}}+u_{\{2\}}\}$, it suffices to show that the ray $R_{u_{\{1\}},u_{\{2\}}}$ defined as $\{(u_{\{1\}},u_{\{2\}},u_{\{1,2\}}):u_{\{1,2\}}\geq u_{\{1\}}+u_{\{2\}}\}$ lies in $\overline{\Gamma(2)}$ for any given $u_{\{1\}},u_{\{2\}}\in{\mathbb{R}}_{+}$. In other words, we wish to show that for any given $u_{\{1\}},u_{\{2\}}\in{\mathbb{R}}_{+}$, and for a dense subset of $u_{\{1,2\}}$ such that $u_{\{1,2\}}\geq u_{\{1\}}+u_{\{2\}}$, there exist independent random vectors $X,Y$ such that $\mathcal{N}(X)=u_{\{1\}}$, $\mathcal{N}(Y)=u_{\{2\}}$, and $\mathcal{N}(X+Y)=u_{\{1,2\}}$.

We find it useful to adapt a construction of Bobkov and Chistyakov [8, Example 3] for our purposes. Consider the uniform density $\tilde{f}$ on a set

[TABLE]

where $a_{n}\geq 0$ for each $n$, and $\sum_{n\in\mathbb{N}}a_{n}=1$. Alternatively, one can write $\tilde{f}$ in the form

[TABLE]

where $\tilde{f}_{n}(x)=a_{n}^{-1}\hbox{\rm{1}}_{(2^{n},2^{n}+a_{n})}(x)$. If $X,X^{\prime}$ are i.i.d. and have density $\tilde{f}$, then $h(X)=0$ since $A$ has length 1, and [8] showed that

[TABLE]

where $s_{n}=a_{1}+\ldots+a_{n}$. By choosing the weights $a_{n}$ appropriately, one can clearly make $h(X+X^{\prime})$ larger than any constant one might pick.222Indeed, one can force $h(X+X^{\prime})$ to be infinite; note that $h(X+X^{\prime})<\infty$ if and only if

$\displaystyle\sum_{n\in\mathbb{N}}a_{n}\log\frac{1}{a_{n}}<\infty.$

This was what motivated [8] to consider this construction, since they wished to exhibit a density $\tilde{f}$ with finite differential entropy whose self-convolution has infinite differential entropy. For example, for $N$ chosen large enough, one may take $a_{n}=1/N$ for $1\leq n\leq N$, and $a_{n}=0$ for $n>N$. By scaling (i.e., considering the random variables $Y=bX$ and $Y^{\prime}=bX^{\prime}$ for $b>0$), one has examples where $h(Y)=\log b$ is any specified real number, and $h(Y+Y^{\prime})$ is larger than an arbitrary specified constant.

Now suppose $u_{\{1\}},u_{\{2\}}$ are given, and assume without loss of generality that $u_{\{1\}}\leq u_{\{2\}}$. Fix an arbitrarily large positive constant $C$. By the preceding paragraph, there exists a random vector $X$ such that $h(X)=s:=\frac{d}{2}\log u_{\{1\}}$, and $\mathcal{N}(X+X^{\prime})>C$, where $X^{\prime}$ is an independent copy of $X$. Observe that the function $u\mapsto h(X^{\prime}+\sqrt{u}Z)$, where $Z$ is an independent standard Gaussian, maps $[0,\infty)$ to $[s,\infty)$, and is also smooth and monotonically increasing (as quantified by the classical de Bruijn identity). Consequently, there must exist a $u>0$ such that $h(X^{\prime}+\sqrt{u}Z)=t:=\frac{d}{2}\log u_{\{2\}}$. Setting $Y=X^{\prime}+\sqrt{u}Z$, observe that $\mathcal{N}(X)=u_{\{1\}}$, $\mathcal{N}(Y)=u_{\{2\}}$, and $\mathcal{N}(X+Y)=\mathcal{N}(X+X^{\prime}+\sqrt{u}Z)\geq\mathcal{N}(X+X^{\prime})>C$.

From the preceding paragraph, for any given $u_{\{1\}},u_{\{2\}}>0$, and for arbitrary $C$, there exists $C^{\prime}>C$ such that $(u_{\{1\}},u_{\{2\}},C^{\prime})\in\Gamma_{1}(2)\subset\overline{\Gamma(2)}$. On the other hand, by considering one-dimensional Gaussians of appropriate variances, clearly $(u_{\{1\}},u_{\{2\}},u_{\{1\}}+u_{\{2\}})\in\Gamma_{1}(2)\subset\overline{\Gamma(2)}$. By Theorem 3, $\overline{\Gamma(2)}$ is an orthogonally convex cone, from which it follows that the ray

[TABLE]

is a subset of $\overline{\Gamma(2)}$.

We wish to now extend this subset relation to the one remaining circumstance– namely when either $u_{\{1\}}$ or $u_{\{2\}}$ (or both) is 0. However this immediately follows because of the closure of $\overline{\Gamma(2)}$. (Another way to treat this is by extending another example of Bobkov and Chistyakov [8, Example 1]. For every $\epsilon>0$, consider the density

[TABLE]

The entropy of $f_{\epsilon}$ exists for every $\epsilon>0$, and if $X$ is drawn from $f_{\epsilon}$, then $h(X)>-\infty$ if and only if $\epsilon>1$. Furthermore, if $Y$ is drawn from $f_{\epsilon^{\prime}}$ for some $\epsilon^{\prime}>0$, then the density of $X+Y$ behaves near 0 like $f_{\epsilon+\epsilon^{\prime}}$, so that $h(X+Y)>-\infty$ if and only if $\epsilon+\epsilon^{\prime}>1$. To construct examples with $u_{\{1\}}=u_{\{2\}}=0$, consider i.i.d. random variables $X,X^{\prime}$ distributed according to $f_{1}$; their convolution has finite entropy by the above reasoning. Now simply by scaling, i.e., considering $aX$ and $aX^{\prime}$ for $a>0$, one can generate examples where $u_{\{1\}}=u_{\{2\}}=0$, and $u_{\{1,2\}}$ is any positive real number.)

We have now proved that $R_{u_{\{1\}},u_{\{2\}}}\subset\overline{\Gamma(2)}$ for arbitrary $u_{\{1\}},u_{\{2\}}\geq 0$. Since

[TABLE]

we deduce that $\Gamma_{FSA}(2)\subset\overline{\Gamma(2)}$. Since the opposite inclusion follows from Theorem 1 and the fact that $\Gamma_{FSA}(2)$ is closed, we have completed the proof of the fact that $\Gamma_{FSA}(2)=\overline{\Gamma(2)}$.

III-E Proof of Theorem 5

Fix $a,b,c>0$. Consider the sets

[TABLE]

and

[TABLE]

Then $R_{a,b,c}$ is the singleton containing the number $a+b+c$; this follows from the equality conditions for the entropy power inequality, which mandate based on the defining conditions of $R_{a,b,c}$ that $X_{1},X_{2}$ and $X_{3}$ are Gaussian with proportional covariance matrices. On the other hand, $R^{\prime}_{a,b,c}$ is $\{u_{\{1,2,3\}}\in{\mathbb{R}}_{+}:u_{\{1,2,3\}}\geq a+b+c\}$. This implies that $\overline{\Gamma(3)}$ is a strict subset of $\Gamma_{FSA}(3)$.

III-F Proof of Theorem 6

It is easy to see from the definitions that $\overline{\Gamma(n)}\supset\Gamma_{\infty}(n)$.

We wish to show now that $\overline{\Gamma(n)}\subset\Gamma_{\infty}(n)$. Let $a\in\overline{\Gamma(n)}$. Then there exists a sequence $a_{m}\in\Gamma_{d_{m}}(n)$ such that $a_{m}\rightarrow a$ in ${\mathbb{R}}^{2^{n}-1}$; let us write $a_{m}(s)=e^{2h(\sum_{i\in s}X_{m,i})/d_{m}}$, where each $X_{m,i}$ is a $d_{m}$-dimensional random vector. We may always assume, without loss of generality, that the sequence $d_{m}\rightarrow\infty$ as $m\rightarrow\infty$. This is because $\Gamma_{d}(n)$ is trivially embedded in $\Gamma_{ld}(n)$ for any positive integer $l$; so we can artificially enforce the condition that $d_{m}\geq m$ for each $m$ in order to make $\infty$ the unique limit point of the sequence $(d_{m})$.

For each $k\in\mathbb{N}$, set $D_{k}=\prod_{m=1}^{k}d_{m}$, and consider the random vector $Z$ obtained by stacking $D_{0}=1$ copy of $X_{1}$, $D_{1}=d_{1}$ copies of $X_{2}$, $D_{2}=d_{1}d_{2}$ copies of $X_{3}$, and so on (to infinity). Let us call $Z_{k}$ the result of this process if we stopped at the $k$-th step. Note that $Z_{k}$ is a random matrix with $n$ columns, and $\mathcal{S}_{k}$ rows of real-valued random variables, where $\mathcal{S}_{k}:=d_{1}+D_{1}d_{2}+\ldots D_{k-1}d_{k}=\sum_{m=1}^{k}D_{m}$ and $D_{m}\geq m!$ by construction. We claim that $Z$ is a stochastic process whose point in the infinite-dimensional Stam region is $a$, which would prove our desired result, and devote the rest of this section to proving this claim.

Denote by $A_{k}$ the point in the Stam region determined by $Z_{k}$. Then

[TABLE]

so that

[TABLE]

We now need a basic lemma about limits333This lemma may be seen as a generalization of the basic theorem about Cesàro means (see, e.g., [24]). Indeed, while the Cesàro mean weights the original sequence using the weights $c_{k}=1$, and the original sequence may be seen as a limit of weighted versions of itself (using weights $c_{k}=\alpha^{k}$ with $\alpha\rightarrow\infty$), the lemma treats weights that are in between these two extremes..

Lemma 3**.**

Suppose $b_{k}\rightarrow b$ as $k\rightarrow\infty$, and

[TABLE]

with the positive coefficients $c_{k}$ satisfying, for some $\alpha>1$, $c_{k}\geq\alpha c_{k-1}$ for each $k\in\mathbb{N}$. Then $S_{n}\rightarrow b$ as $n\rightarrow\infty$.

Proof.

By considering the sequence $(b_{k}-b:k\in\mathbb{N})$, it suffices to prove the result for the case where $b=0$; so we will now assume this without loss of generality. Given that $b_{k}\rightarrow 0$, we know that for every $\epsilon>0$, there exists $m(\epsilon)$ such that $|b_{k}|<\epsilon$ for every $k>m(\epsilon)$. We wish to show that for every $\epsilon>0$, there exists $M(\epsilon)$ such that $|S_{n}|<\epsilon$ for every $n>M(\epsilon)$. We claim that

[TABLE]

serves the purpose.

Indeed, since

[TABLE]

we have that

[TABLE]

Now we can deduce

[TABLE]

as required.     $\Box$

Combining Lemma 3 (with $c_{k}=D_{k},\alpha=2$ and $b_{k}=\log a_{k}$) with the continuity of the exponential and logarithmic functions, it follows from the identity (14) and the fact that $a_{m}\rightarrow a$ as $m\rightarrow\infty$ that $A_{k}\rightarrow a$ as $k\rightarrow\infty$. In particular, since the entropy power rate of $\sum_{i\in s}Z_{i}$ is just the limit of entropy powers of $\sum_{i\in s}Z_{k,i}$ by definition, we have that the entropy power rate of $\sum_{i\in s}Z_{i}$ is $a(s)$, and consequently that the point $a$ is in $\Gamma_{\infty}(n)$.

IV Discussion of a question

Let us start by proving a general proposition about determinants of sums of positive-definite matrices.

Proposition 3**.**

Let $M_{1},M_{2},\cdots,M_{n}$ be $d\times d$ positive-definite matrices. Then, the function $v_{c}:\mathbb{R}_{+}^{n}\rightarrow\mathbb{R}_{+}$ defined by

[TABLE]

is supermodular, i.e. for any $x,y\in\mathbb{R}_{+}^{n}$ it holds that

[TABLE]

where $x\vee y$ denotes the componentwise maximum of $x$ and $y$ and $x\wedge y$ denotes the componentwise minimum of $x$ and $y$.

Proof.

Inequality (15) trivially holds when $x=0$ or $y=0$. To prove that (15) also holds otherwise, it suffices to show by Proposition 2 (since $v_{c}$ is smooth) that $\frac{\partial^{2}v_{c}(x)}{\partial x_{j}\partial x_{i}}\geq 0$ for any $x\in{\mathbb{R}}_{+}^{n}$. We note that, setting $M=\sum_{i=1}^{n}x_{i}M_{i}$,

[TABLE]

and

[TABLE]

However, for any two positive definite matrices $A$ and $B$, $\text{det}(A)\geq 0$ and $\text{trace}(A)\text{trace}(B)\geq\text{trace}(AB)$ (see e.g., Zhang [16], page 166). Hence, both terms on the right-hand side of (16) are always non-negative.     $\Box$

By Lemma 1, Proposition 3 immediately yields:

Theorem 7**.**

For any positive-definite matrices $A,B$ and $C$, we have

[TABLE]

In other words, the set function $s\mapsto e^{2h(\sum_{i\in s}X_{i})}$ is supermodular if all the random vectors $X_{i}$ are Gaussian (since in this case, the functional $e^{2h}$ is– up to irrelevant constants– the determinant of the covariance matrix).

It is natural to wonder how far this phenomenon extends. For example, one might ask if it holds for arbitrary distributions of the $X_{i}$, which– if true– would imply that $\Gamma_{1}(n)\subset\Gamma_{SM}(n)$, thus refining the inclusion $\Gamma_{1}(n)\subset\Gamma_{FSA}(n)$ that follows from Theorem 1. However, this turns out to be false. Indeed, we will see that even a much more restrictive statement is false, but to discuss it, we need to take a detour through some convex geometry.

It was recently observed in [16] (by combining the determinant inequality of Theorem 7 with some tools from optimal transportation theory) that the supermodularity property of Theorem 7 can be extended from determinants to volumes of convex sets.

Theorem 8**.**

[16]** If $B_{i}$ are convex sets in ${\mathbb{R}}^{d}$,

[TABLE]

Theorem 8 is a generalization of Theorem 7 since volumes of ellipsoids and parallelotopes (which are all convex) are given by determinants, and since Minkowski summation of these objects correspond to addition of the corresponding positive-definite matrices. Theorem 8 does not extend to arbitrary compact sets. However, [16] also poses the question of whether the inequality (18) holds when $B_{1}$ is compact and convex, while $B_{2},B_{3}$ are arbitrary compact sets; and it observes that the answer to this question is affirmative in dimension 1.

As is now well known, the entropy power inequality $\mathcal{N}(X+Y)\geq\mathcal{N}(X)+\mathcal{N}(Y)$ resembles in many ways the Brunn-Minkowski inequality, which is a very important inequality in mathematics, and a cornerstone of convex geometry in particular. This was first noticed by Costa and Cover [13] (see also [14, 54, 57] for other aspects of this connection). The analogies between inequalities in information theory and those in convex geometry have been explored quite a bit in recent years (see, e.g., the survey [39] and references therein), based loosely on the understanding that the functional $A\mapsto\text{Vol}_{d}(A)^{1/d}$ in the geometry of compact subsets of ${\bf R}^{d}$, and the functional \mbox{{\cal L}}(X)\mapsto\mathcal{N}(X) in probability are analogous to each other in many ways. In the dictionary that relates notions in convex geometry to those in probability, the natural analog of a convex set is a log-concave distribution. Recall that a log-concave distribution is one that has a density of the form $e^{-V}$, where $V:{\bf R}^{d}\rightarrow{\bf R}\cup\{\infty\}$ is a convex function.

Motivated by the afore-mentioned question of [16], it is tempting to consider the following question which would be its natural probabilistic analog:

Question 1**.**

If $X,Y,Z$ are independent ${\mathbb{R}}^{d}$-valued random vectors with $Z$ having a log-concave distribution, is it true that

[TABLE]

If the answer to Question 1 is positive, it would imply that for independent ${\mathbb{R}}^{d}$-valued random vectors $X_{1},\ldots,X_{n}$ with log-concave distributions, the set function

[TABLE]

is supermodular. Unfortunately, however, we will now show that the answer to Question 1 is negative.

Proposition 4**.**

There exist independent ${\mathbb{R}}$-valued random variables $X,Y,Z$, each with log-concave distributions, such that

[TABLE]

In particular, the answer to Question 1 is negative, already in dimension 1.

Proof.

Suppose the answer to Question 1 were yes for $d=1$. Then, given any $\epsilon>0$, taking $X_{\epsilon}$ to be Gaussian with variance $\epsilon$ would yield

[TABLE]

On rearrangement, we have

[TABLE]

Dividing by $\epsilon$ and taking the limit as $\epsilon\rightarrow 0$ gives

[TABLE]

The classical de Bruijn identity [53], which is easily proved by observing that the density of $X_{\epsilon}+U$ satisfies the heat equation when $X_{\epsilon}$ is a Gaussian of variance $\epsilon$ independent of $U$, asserts that

[TABLE]

where $I$ is the Fisher information, i.e., if $U$ has density $u$,

[TABLE]

As observed by Stam [53], the chain rule for differentiation implies that

[TABLE]

Putting together the inequality (19) and the identity (20), we conclude that444Since $\mathcal{N}(X+Y)\geq\mathcal{N}(X)$ and $I(X+Y)\leq I(X)$, if such an inequality were true, it would capture how $\mathcal{N}$ and $I$ compete in their behavior on convolution in the log-concave setting. Observe that $\mathcal{N}(X)I(X)$ is a affine-invariant functional in dimension 1, and that a limiting argument based on the entropy power inequality (see, e.g., [53]) yields the entropic isoperimetric inequality

$\displaystyle\mathcal{N}(X)I(X)\geq 2\pi e,$

with equality if and only if $X$ is Gaussian.

[TABLE]

Setting $T_{n}=\sum_{i=1}^{n}X_{i}$ and $S_{n}=\frac{1}{\sqrt{n}}T_{n}$ for i.i.d. random variables $X_{i}$ with the same distribution as $X$, we must have

[TABLE]

which upon taking the limit as $n\rightarrow\infty$ and using the entropic central limit theorem of Barron [3] and the corresponding statement for Fisher information [27, Theorem 1.6], yields the inequality $2\pi e\geq\mathcal{N}(X)I(X)$. Since this contradicts the entropic isoperimetric inequality, the assumption that we started with– namely, that the answer to Question 1 is yes for $d=1$– cannot hold.     $\Box$

Since the answer to Question 1 is negative in dimension 1, whereas the corresponding question for volumes of sets has a positive answer in dimension 1 as observed in [16], Proposition 4 adds to the examples where the analogy between the probabilistic and convex geometric statements breaks down.

Proposition 4 may also be seen as a strengthening of Theorem 2. While the proof of Theorem 2 in Section III-B shows that $\Gamma_{d}(n)\cap\Gamma_{SM}(n)^{c}\neq\phi$ for $d>1$ and $n\geq 3$, Proposition 4 shows that for $n\geq 3$, even $\Gamma_{1}(n)$ contains points outside of the supermodularity cone $\Gamma_{SM}(n)$. Indeed, by examining the proof, it shows something much stronger: even if we restrict ourselves to the class of one-dimensional random variables $X,Y,Z$ with $Z$ being Gaussian and $X,Y$ being i.i.d. and drawn from a log-concave distribution, supermodularity of entropy power fails.

V Remarks

The classical entropy power inequality of Shannon [49] and Stam [53] states that if $X_{i}$ are independent ${\mathbb{R}}^{d}$-valued random vectors,

[TABLE]

Motivated by the long-standing monotonicity conjecture for the entropic central limit theorem, Artstein, Ball, Barthe and Naor [1] proved a new entropy power inequality:

[TABLE]

Simplified proofs of inequality (22) were independently given by [32, 56, 52]. Subsequently, [33] proved the inequality (7) based on the maximum degree of the hypergraph $\mathcal{G}$; this contains the inequalities (22) and (21) since for $\mathcal{G}=\mathcal{G}_{m}$, namely the collection of all subsets of indices of size $m$, we have $r=\binom{n-1}{m-1}$. Theorem 1 says that for any fractional partition $\beta$ using a collection $\mathcal{G}$ of subsets of $[n]$,

[TABLE]

Since coefficients that are identical to the reciprocal of the degree of a regular hypergraph constitute a fractional partition, Theorem 1 subsumes and extends all the inequalities discussed above. 2. 2.

What are the basic properties of the Stam region? Apart from the theorems stated in the introduction, we do not know any other structural properties of $\Gamma_{d}(n)$ or $\Gamma(n)$. There are many natural questions, such as whether $\Gamma(n)$ or its closure is convex, that remain to be answered. 3. 3.

There are additional constraints on the Stam region for $n\geq 3$ that have not been discussed so far, but these are nonlinear. Specifically, it was observed in [31] (see [37, 38, 6, 7, 10, 29, 35, 36] for various applications and generalizations) that the differential entropy of sums of independent random vectors is submodular, i.e.,

[TABLE]

Written in terms of entropy powers, this implies

[TABLE]

which is a nonlinear constraint on the Stam region for any $n\geq 3$. 4. 4.

The study of the Stam region is analogous in some sense to the study of the entropic region defined using the joint entropy of subsets of random variables, on which there has been much progress in recent decades. Let $X_{s}$ denote $(X_{i}:i\in s)$ and $H$ denote the discrete entropy, where $(X_{i}:i\in[n])$ is a collection of dependent random variables taking values in some finite set. Fujishige [18] observed that $g(s)=H(X_{s})$ is a submodular set function. Consequent entropy inequalities obtained by Han [23] and Shearer [11] became influential in information theory and in combinatorics respectively. These inequalities were unified and generalized in [45, 41], where it was shown that for any submodular set function $g:2^{[n]}\rightarrow{\mathbb{R}}_{+}$ (and in particular for the joint entropy), one has

[TABLE]

as well as corresponding lower bounds that we do not state here, for any fractional partition $\beta$ using any collection of sets $\mathcal{G}$. Motivated by the problem of characterizing the entropic region (or equivalently, the class of all entropy inequalities for the joint distributions of a collection of dependent random variables), Zhang and Yeung [62] observed in 1998 that there exist so-called non-Shannon inequalities that do not follow from submodularity of joint entropy; these have seen much active investigation recently (see, e.g., Matú$\check{\text{s}}$ [44], who showed the remarkable fact that the entropic region is not polyhedral if $n\geq 4$). While Theorem 1 is analogous in some sense to the inequality (25), Theorem 2 shows that the analogue of Fujishige’s submodularity is not true. In particular, the question of whether there exist entropy inequalities for sums that are formally analogous to non-Shannon inequalities is ill-posed, since we do not have supermodularity of entropy power to start with. 5. 5.

Theorem 1 is a more informative statement than its predecessors such as (7), as pointed out in [30]. Recall that entropy power inequalities have been key to the determination of some capacity and rate regions, and that rate regions for several multi-user problems (such as the $m$-user Slepian-Wolf problem) involve subset sum constraints. Vaguely motivated by this, one may consider the “region” of all $(R_{1},\ldots,R_{n})\in{\mathbb{R}}_{+}^{n}$ satisfying $\sum_{j\in s}R_{j}\geq\mathcal{N}(T^{s})$ for each $s\subset[n]$. Then Theorem 1 is equivalent to the existence of a point in this region such that the total sum $\sum_{j\in[n]}R_{j}=\mathcal{N}(T^{[n]})$. Although we are not yet aware of a specific multiuser capacity problem with precisely this rate region, this fact appears intriguing. 6. 6.

As discussed in Section IV, there are extensive analogies between inequalities in convex geometry and entropy inequalities. The volume analog of the fractional entropy power inequalities proved in this paper, namely

[TABLE]

for fractional partitions $\beta$, was observed to hold for Minkowski sums of compact convex sets in [9], and conjectured to hold more generally for Minkowski sums of compact sets. However, recently it was shown in [15] that even the weaker inequality

[TABLE]

corresponding to the leave-one-out subsets, fails for general compact sets in dimension 12 or greater. On the other hand, it was shown in [16] that the inequality

[TABLE]

without the exponents applied to the volumes, holds for general compact sets $A_{i}$ in ${\mathbb{R}}^{d}$, while [4] recently showed that the inequality (26) in fact holds in dimension 1. 7. 7.

Shannon (differential) entropy is merely an instance of the more general family of Rényi entropies $h_{p}$ for $p\in[0,\infty]$. Indeed, the Brunn-Minkowski inequality as well as other inequalities discussed in the preceding remark may be seen as inequalities for the Rényi entropy of order 0, while the Shannon entropy power inequalities are inequalities for the Rényi entropy of order 1. One can thus seek to understand the Rényi analog of the Stam region– where, say, one replaces Shannon entropy powers by Rényi entropy powers. However, what we know about Rényi entropy power inequalities is so sparse that we are far from being able to address this question at all meaningfully. Indeed, it is not even clear what the best definition of Rényi entropy power is– whether it should be defined as $e^{2h_{p}(X)/d}$ or with an exponent other than 2, for instance. The only (at least asymptotically) sharp Rényi entropy power inequalities known for $p\neq 0,1$ are for $p=\infty$ and can be found in [40]. 8. 8.

While this paper focused on characterization of possible inequalities for the entropies of sums of independent random vectors in ${\mathbb{R}}^{d}$, the same question also makes sense (and is interesting both as a basic mathematical question and in view of applications to communication theory) for random variables taking values in any group. A priori, it is natural to first try groups that have particularly simple structure– such as finite cyclic groups or the integers. However, even for these seemingly staid examples, there is little that can currently be said. Indeed, we do not even know a fully satisfactory analogue of the entropy power inequality on the integers (some partial results are available in [25, 51, 22, 58, 59, 43, 21, 43, 42]). As in the Euclidean setting, such inequalities for integer-valued random variables also have connections to probabilistic limit theorems (see, e.g., [2, 60, 28]). The only discrete groups for which there exist sharp entropy power inequalities are groups of order that is a power of 2; these were recently proved in [26].

Acknowledgment

The first author thanks Young-Han Kim for suggesting that this note, which was largely written in 2009 but set aside, may be of broader interest and for the encouragement to complete and submit it for publication. Parts of this work were completed while the first author was at the Institute of Mathematics and its Applications (IMA) in Minneapolis during spring 2015, and he is grateful to the IMA for its hospitality and providing a conducive research environment. The authors also thank the Associate Editor and two anonymous reviewers for comments that improved the exposition.