A new approach to mutual information

TL;DR

This paper introduces a novel asymptotic expression for mutual information using discrete micro-states of permutations, applicable to both continuous and discrete random variables.

Contribution

It presents a new theoretical approach to mutual information based on asymptotic limits and micro-states, expanding the analytical tools available.

Findings

Provides a new formula for mutual information

Applicable to both continuous and discrete variables

Enhances understanding of mutual information structure

Abstract

A new expression as a certain asymptotic limit via "discrete micro-states" of permutations is provided to the mutual information of both continuous and discrete random variables.

Equations

$$I(X\wedge Y)=-H(X,Y)+H(X)+H(Y)$$

$$I(X\wedge Y):=S({\mu }_{(X,Y)},{\mu }_X\otimes {\mu }_Y),$$

$${1\over N}\log\gamma_{S_{N}}^{\otimes n}\bigl{(}\Delta_{\mathrm{sym}}(X_{1},\dots,X_{n};N,m,\delta)\bigr{)}$$

$$-H(X_1,\dots ,X_n)+\sum _{i=1}^{n}H(X_i)$$

$$H(X_1,\dots ,X_n):=-\int \cdots {\int }_{{\mathbb{R}}^n}p(x_1,\dots ,x_n)\log p(x_1,\dots ,x_n)d{x}_1\cdots d{x}_n$$

$${\kappa }_N(\mathbf{x}):=\frac{1}{N}\sum _{j=1}^N{x}_j.$$

$$|{\kappa }_N({\mathbf{x}}_{i_1}\cdots {\mathbf{x}}_{i_k})-\mathbb{E}(X_{i_1}\cdots X_{i_k})|<\delta$$

$${\mathbf{x}}_{i_1}\cdots {\mathbf{x}}_{i_k}:=(x_{i_{1}1}\cdots x_{i_{k}1},x_{i_{1}2}\cdots x_{i_{k}2},\dots ,x_{i_{1}N}\cdots x_{i_{k}N})\in {\mathbb{R}}^N$$

$$\lim_{N\to\infty}{1\over N}\log\lambda_{N}^{\otimes n}\bigl{(}\Delta_{R}(X_{1},\dots,X_{n};N,m,\delta)\bigr{)}$$

$$H(X_{1},\dots,X_{n})=\lim_{m\rightarrow\infty,\delta\searrow 0}\lim_{N\to\infty}{1\over N}\log\lambda_{N}^{\otimes n}\bigl{(}\Delta_{R}(X_{1},\dots,X_{n};N,m,\delta)\bigr{)}$$

$$\sigma (\mathbf{x}):=(x_{{\sigma }^{-1}(1)},x_{{\sigma }^{-1}(2)},\dots ,x_{{\sigma }^{-1}(N)})$$

$$({\sigma }_1(x_1),\dots ,{\sigma }_n(x_n))\in {\Delta }_R(X_1,\dots ,X_n;N,m,\delta )$$

$$I_{{\mathrm{sym}},R}(X_{1},\dots,X_{n}):=-\lim_{m\to\infty,\delta\searrow 0}\limsup_{N\to\infty}{1\over N}\log\gamma_{S_{N}}^{\otimes n}\bigl{(}\Delta_{{\mathrm{sym}},R}(X_{1},\dots,X_{n};N,m,\delta)\bigr{)},$$

$$0\le I_{\mathrm{sym},R}(X_1,\dots ,X_n)\le {\overline{I}}_{\mathrm{sym},R}(X_1,\dots ,X_n).$$

$$({\sigma }_1({\xi }_1(N)),\dots ,{\sigma }_n({\xi }_n(N)))\in \Delta (X_1,\dots ,X_n;N,m,\delta ).$$

$$I_{\mathrm{sym}}(X_1,\dots ,X_n:{\xi }_1,\dots ,{\xi }_n)$$

$$\displaystyle\quad:=-\lim_{m\rightarrow\infty,\delta\searrow 0}\limsup_{N\to\infty}{1\over N}\log\gamma_{S_{N}}^{\otimes n}\bigl{(}\Delta_{\mathrm{sym}}(X_{1},\dots,X_{n}:\xi_{1}(N),\dots,\xi_{n}(N);N,m,\delta)\bigr{)}$$

$$I_{\mathrm{sym},\infty }(X_1,\dots ,X_n)$$

$${\overline{I}}_{\mathrm{sym},\infty }(X_1,\dots ,X_n)$$

$${\Delta }_{\mathrm{sym},\infty }(X_1,\dots ,X_n;N,m^{\prime },{\delta }^{\prime })\subset {\Delta }_{\mathrm{sym}}(X_1,\dots ,X_n:{\xi }_1(N),\dots ,{\xi }_n(N);N,m,\delta )$$

$$\Vert {\mathbf{x}}_i{\Vert }_m$$

$$<(\mathbb{E}(X_i^{2m})+1{)}^{1/2m}$$

$$\le (R_0^{2m}+1{)}^{1/2m}\le R_0+1.$$

$$|{\kappa }_N({\sigma }_{i_1}({\xi }_{i_1}(N))\cdots {\sigma }_{i_k}({\xi }_{i_k}(N)))-\mathbb{E}(X_{i_1}\cdots X_{i_k})|$$

$$\le |{\kappa }_N({\sigma }_{i_1}({\xi }_{i_1}(N))\cdots {\sigma }_{i_k}({\xi }_{i_k}(N)))-{\kappa }_N({\sigma }_{i_1}({\mathbf{x}}_{i_1})\cdots {\sigma }_{i_k}({\mathbf{x}}_{i_k}))|$$

$$+|{\kappa }_N({\sigma }_{i_1}({\mathbf{x}}_{i_1})\cdots {\sigma }_{i_k}({\mathbf{x}}_{i_k}))-\mathbb{E}(X_{i_1}\cdots X_{i_k})|$$

$$\le m(R_0+1{)}^{m-1}\rho +{\delta }^{\prime }<\delta$$

$$H(X_1,\dots ,X_n)$$

$$=-{\overline{I}}_{\mathrm{sym}}(X_1,\dots ,X_n)+\sum _{i=1}^{n}H(X_i).$$

$$\mathbf{s}\in {\mathbb{R}}^N⟷(\mathbf{x},\sigma )\in {\mathbb{R}}_{\le }^N\times S_N,\mathbf{s}=\sigma (\mathbf{x})$$

$$({\mathbf{x}}_1,\dots ,{\mathbf{x}}_n;{\sigma }_1,\dots ,{\sigma }_n)$$

$$\displaystyle\qquad\in\Biggl{(}\prod_{i=1}^{n}\bigl{(}\Delta_{R}(X_{i};N,m,\delta)\cap{\mathbb{R}}_{\leq}^{N}\bigr{)}\Biggr{)}\times\Delta_{{\mathrm{sym}},R}(X_{1},\dots,X_{n};N,m,\delta).$$

$$H(X_1,\dots ,X_n)$$

$$\displaystyle\leq\liminf_{N\to\infty}{1\over N}\Biggl{(}\sum_{i=1}^{n}\log\lambda_{N}\bigl{(}\Delta_{R}(X_{i};N,m,\delta)\cap{\mathbb{R}}_{\leq}^{N}\bigr{)}$$

$$\displaystyle\qquad\qquad\qquad+\log\#\Delta_{{\mathrm{sym}},R}(X_{1},\dots,X_{n};N,m,\delta)\Biggr{)}$$

$$\displaystyle=\liminf_{N\to\infty}{1\over N}\Biggl{(}\sum_{i=1}^{n}\log\lambda_{N}\bigl{(}\Delta_{R}(X_{i};N,m,\delta)\bigr{)}-n\log N!$$

Full text

${}^{1}\,$Supported in part by Grant-in-Aid for Scientific Research (B)17340043.

${}^{2}\,$Supported in part by the Hungarian Research Grant OTKA T068258.

AMS subject classification: Primary: 62B10, 94A17.

Introduction

One of the important quantities in information theory is the mutual information of two random variables $X$ and $Y$ which is expressed in terms of the Boltzmann-Gibbs entropy $H(\cdot)$ as follows:

[TABLE]

when $X,Y$ are continuous variables. For the expression of $I(X\wedge Y)$ of discrete variables $X,Y$, the above $H(\cdot)$ is replaced by the Shannon entropy. A more practical and rigorous definition via the relative entropy is

[TABLE]

where $\mu_{(X,Y)}$ denotes the joint distribution measure of $(X,Y)$ and $\mu_{X}\otimes\mu_{Y}$ the product of the respective distribution measures of $X,Y$.

The aim of this paper is to show that the mutual information $I(X\wedge Y)$ is gained as a certain asymptotic limit of the volume of “discrete micro-states” consisting of permutations approximating joint moments of $(X,Y)$ in some way. In Section 1, more generally we consider an $n$-tuple of real bounded random variables $(X_{1},\dots,X_{n})$. Denote by $\Delta(X_{1},\dots,X_{n};N,m,\delta)$ the set of $({\mathbf{x}}_{1},\dots,{\mathbf{x}}_{n})$ of ${\mathbf{x}}_{i}\in{\mathbb{R}}^{N}$ whose joint moments (on the uniform distributed $N$-point set) of order up to $m$ approximate those of $(X_{1},\dots,X_{n})$ up to an error $\delta$. Furthermore, denote by $\Delta_{\mathrm{sym}}(X_{1},\dots,X_{n};N,m,\delta)$ the set of $(\sigma_{1},\dots,\sigma_{n})$ of permutations $\sigma_{i}\in S_{N}$ such that $(\sigma_{1}({\mathbf{x}}_{1}),\dots,\sigma_{n}({\mathbf{x}}_{n}))\in\Delta(X_{1},\dots,X_{n};N,m,\delta)$ for some ${\mathbf{x}}_{1},\dots,{\mathbf{x}}_{n}\in{\mathbb{R}}_{\leq}^{N}$, where ${\mathbb{R}}_{\leq}^{N}$ is the ${\mathbb{R}}^{N}$-vectors arranged in increasing order. Then, the asymptotic volume

[TABLE]

under the uniform probability measure $\gamma_{S_{N}}$ on $S_{N}$ is shown to converge as $\limsup_{N\to\infty}$ (also $\liminf_{N\to\infty}$) and then $\lim_{m\to\infty,\delta\searrow 0}$ to

[TABLE]

as long as $H(X_{i})>-\infty$ for $1\leq i\leq n$. Thus, we obtain a kind of discretization of the mutual information via symmetric group (or permutations).

The approach can be applied to an $n$-tuple of discrete random variables $(X_{1},\dots,X_{n})$ as well. But the definition of the $\Delta_{\mathrm{sym}}$-set of micro-states for discrete variables is somewhat different from the continuous variable case mentioned above, and we discuss the discrete variable case in Section 2 separately.

The idea comes from the paper [3]. Motivated by theory of mutual free information in [6], a similar approach to Voiculescu’s free entropy is provided there. The free entropy is the free probability counterpart of the Boltzmann-Gibbs entropy, and ${\mathbb{R}}^{N}$-vectors and the symmetric group $S_{N}$ here are replaced by Hermitian $N\times N$ matrices and the unitary group $\mathrm{U}(N)$, respectively. In this way, the “discretization approach” here is in some sense a classical analog of the “orbital approach” in [3].

1. The continuous case

For $N\in{\mathbb{N}}$ let ${\mathbb{R}}_{\leq}^{N}$ be the convex cone of the $N$-dimensional Euclidean space ${\mathbb{R}}^{N}$ consisting of ${\mathbf{x}}=(x_{1},\dots,x_{N})$ such that $x_{1}\leq x_{2}\leq\dots\leq x_{N}$. The space ${\mathbb{R}}^{N}$ is naturally regarded as the real function algebra on the $N$-point set. Let $S_{N}$ be the symmetric group of order $N$ (i.e., the permutations on $\{1,2,\dots,n\}$). Throughout this section let $(X_{1},\dots,X_{n})$ be an $n$-tuple of real random variables on a probability space $(\Omega,{\mathbb{P}})$, and assume that the $X_{i}$’s are bounded (i.e., $X_{i}\in L^{\infty}(\Omega;{\mathbb{P}})$). The Boltzmann-Gibbs entropy of $(X_{1},\dots,X_{n})$ is defined to be

[TABLE]

if the joint density $p(x_{1},\dots,x_{n})$ of $(X_{1},\dots,X_{n})$ exists; otherwise $H(X_{1},\dots,X_{n})=-\infty$. Note that the above integral is well defined in $[-\infty,\infty)$ since the density $p$ is compactly supported.

Definition 1.1**.**

The mean value of ${\mathbf{x}}=(x_{1},\dots,x_{N})$ in ${\mathbb{R}}^{N}$ is given by

[TABLE]

For each $N,m\in{\mathbb{N}}$ and $\delta>0$ we define $\Delta(X_{1},\dots,X_{n};N,m,\delta)$ to be the set of all $n$-tuples $({\mathbf{x}}_{1},\dots,{\mathbf{x}}_{n})$ of ${\mathbf{x}}_{i}=(x_{i1},\dots,x_{iN})\in{\mathbb{R}}^{N}$, $1\leq i\leq n$, such that

[TABLE]

for all $1\leq i_{1},\dots,i_{k}\leq n$ with $1\leq k\leq m$, where ${\mathbf{x}}_{i_{1}}\cdots{\mathbf{x}}_{i_{k}}$ means the pointwise product, i.e.,

[TABLE]

and ${\mathbb{E}}(\cdot)$ denotes the expectation on $(\Omega,{\mathbb{P}})$. For each $R>0$, define $\Delta_{R}(X_{1},\dots,X_{n};\allowbreak N,m,\delta)$ to be the set of all $({\mathbf{x}}_{1},\dots,{\mathbf{x}}_{n})\in\Delta(X_{1},\dots,X_{n};N,m,\delta)$ such that ${\mathbf{x}}_{i}\in[-R,R]^{N}$ for all $1\leq i\leq n$.

Heuristically, $\Delta(X_{1},\dots,X_{n};N,m,\delta)$ is the set of “micro-states” consisting of $n$-tuples of discrete random variables on the $N$-point set with the uniform probability such that all joint moments of order up to $m$ give the corresponding joint moments of $X_{1},\dots,X_{n}$ up to an error $\delta$.

For ${\mathbf{x}}\in{\mathbb{R}}^{N}$ write $\|{\mathbf{x}}\|_{p}:=(N^{-1}\sum_{j=1}^{N}|x_{j}|^{p})^{1/p}$ for $1\leq p<\infty$ and $\|{\mathbf{x}}\|_{\infty}:=\max_{1\leq j\leq N}|x_{j}|$ while $\|X\|_{p}$ denotes the $L^{p}$-norm of a real random variable $X$ on $(\Omega,{\mathbb{P}})$.

The next lemma is seen from [4, 5.1.1] based on the Sanov large deviation theorem, which says that the Boltzmann-Gibbs entropy is gained as an asymptotic limit of the volume of the approximating micro-states.

Lemma 1.2**.**

For every $m\in{\mathbb{N}}$ and $\delta>0$ and for any choice of $R\geq\max_{1\leq i\leq n}\|X_{i}\|_{\infty}$, the limit

[TABLE]

exists, where $\lambda_{N}$ is the Lebesgue measure on ${\mathbb{R}}^{N}$. Furthermore, one has

[TABLE]

independently of the choice of $R\geq\max_{1\leq i\leq n}\|X_{i}\|_{\infty}$.

In the following let us introduce some kinds of mutual information in the discretization approach using micro-states of permutations.

Definition 1.3**.**

The action of $S_{N}$ on ${\mathbb{R}}^{N}$ is given by

[TABLE]

for $\sigma\in S_{N}$ and ${\mathbf{x}}=(x_{1},\dots,x_{N})\in{\mathbb{R}}^{N}$. For each $N,m\in{\mathbb{N}}$, $\delta>0$ and $R>0$ we denote by $\Delta_{{\mathrm{sym}},R}(X_{1},\dots,X_{n};N,m,\delta)$ the set of all $(\sigma_{1},\dots,\sigma_{n})\in S_{N}^{n}$ such that

[TABLE]

for some $({\mathbf{x}}_{1},\dots,{\mathbf{x}}_{n})\in({\mathbb{R}}_{\leq}^{N})^{n}$. For each $R>0$ define

[TABLE]

where $\gamma_{S_{N}}$ is the uniform probability measure on $S_{N}$. Define also $\overline{I}_{{\mathrm{sym}},R}(X_{1},\dots,X_{n})$ by replacing $\limsup$ by $\liminf$. Obviously,

[TABLE]

Moreover, $\Delta_{{\mathrm{sym}},\infty}(X_{1},\dots,X_{n};N,m,\delta)$ is defined by replacing $\Delta_{R}(X_{1},\dots,X_{n};N,m,\delta)$ in the above by $\Delta(X_{1},\dots,X_{n};N,m,\delta)$ without cut-off by the parameter $R$. Then $I_{{\mathrm{sym}},\infty}(X_{1},\dots,X_{n})$ and $\overline{I}_{{\mathrm{sym}},\infty}(X_{1},\dots,X_{n})$ are also defined as above.

Definition 1.4**.**

For each $1\leq i\leq n$ we choose and fix a sequence $\xi_{i}=\{\xi_{i}(N)\}$ of $\xi_{i}(N)\in{\mathbb{R}}_{\leq}^{N}$, $N\in{\mathbb{N}}$, such that $\kappa_{N}(\xi_{i}(N)^{k})\to{\mathbb{E}}(X_{i}^{k})$ as $N\to\infty$ for all $k\in{\mathbb{N}}$, i.e., $\xi_{i}(N)\to X_{i}$ in moments. For each $N,m\in{\mathbb{N}}$ and $\delta>0$ we define $\Delta_{\mathrm{sym}}(X_{1},\dots,X_{n}:\xi_{1}(N),\dots,\xi_{n}(N);N,m,\delta)$ to be the set of all $(\sigma_{1},\dots,\sigma_{n})\in S_{N}^{n}$ such that

[TABLE]

Define

[TABLE]

and $\overline{I}_{\mathrm{sym}}(X_{1},\dots,X_{n}:\xi_{1},\dots\xi_{n})$ by replacing $\limsup$ by $\liminf$.

The next proposition asserts that the quantities in Definitions 1.3 and 1.4 are all equivalent.

Lemma 1.5**.**

For any choice of $R\geq\max_{1\leq i\leq n}\|X_{i}\|_{\infty}$ and for any choices of approximating sequences $\xi_{1},\dots,\xi_{n}$ one has

[TABLE]

Proof.

It is obvious that $\Delta_{\mathrm{sym}}(X_{1},\dots,X_{n}:\xi_{1}(N),\dots,\xi_{n}(N);N,m,\delta)$ is included in $\Delta_{{\mathrm{sym}},\infty}(X_{1},\dots,X_{n};N,m,\delta)$ for any approximating sequences $\xi_{i}$. Moreover, for each $1\leq i\leq n$ an approximating sequence $\xi_{i}$ can be chosen so that $\|\xi_{i}(N)\|_{\infty}\leq\|X_{i}\|_{\infty}$ for all $N$; then $\Delta_{\mathrm{sym}}(X_{1},\dots,X_{n}:\xi_{1}(N),\dots,\xi_{n}(N);N,m,\delta)\subset\Delta_{{\mathrm{sym}},R}(X_{1},\dots,X_{n};\allowbreak N,m,\delta)$ for any $R\geq R_{0}:=\max_{1\leq i\leq n}\|X_{i}\|_{\infty}$. Hence it suffices to prove that for any approximating sequences $\xi_{i}$ and for every $m\in{\mathbb{N}}$ and $\delta>0$, there are an $m^{\prime}\in{\mathbb{N}}$, a $\delta^{\prime}>0$ and an $N_{0}\in{\mathbb{N}}$ so that

[TABLE]

for all $N\geq N_{0}$. Choose a $\rho\in(0,1)$ with $m(R_{0}+1)^{m-1}\rho<\delta/2$. By [5, Lemma 4.3] (also [4, 4.3.4]) there exist an $m^{\prime}\in{\mathbb{N}}$ with $m^{\prime}\geq 2m$, a $\delta^{\prime}>0$ with $\delta^{\prime}\leq\min\{1,\delta/2\}$ and an $N_{0}\in{\mathbb{N}}$ such that for every $1\leq i\leq n$ and every ${\mathbf{x}}\in{\mathbb{R}}_{\leq}^{N}$ with $N\geq N_{0}$, if $|\kappa_{N}({\mathbf{x}}^{k})-{\mathbb{E}}(X_{i}^{k})|<\delta^{\prime}$ for all $1\leq k\leq m^{\prime}$, then $\|{\mathbf{x}}-\xi_{i}(N)\|_{m}<\rho$. Suppose $N\geq N_{0}$ and $(\sigma_{1},\dots,\sigma_{n})\in\Delta_{{\mathrm{sym}},\infty}(X_{1},\dots,X_{n};N,m^{\prime},\delta^{\prime})$; then $(\sigma_{1}({\mathbf{x}}_{1}),\dots,\sigma_{n}({\mathbf{x}}_{n}))\in\Delta(X_{1},\dots,X_{n};N,m^{\prime},\delta^{\prime})$ for some $({\mathbf{x}}_{1},\dots,{\mathbf{x}}_{n})\in({\mathbb{R}}_{\leq}^{N})^{n}$. Since $|\kappa_{N}({\mathbf{x}}_{i}^{k})-{\mathbb{E}}(X_{i}^{k})|<\delta^{\prime}$ for all $1\leq k\leq m^{\prime}$, we get $\|{\mathbf{x}}_{i}-\xi_{i}(N)\|_{m}\leq\rho$ and

[TABLE]

Therefore,

[TABLE]

for all $1\leq i_{1},\dots,i_{k}\leq n$ with $1\leq k\leq m$. The above latter inequality follows from the Hölder inequality. Hence $(\sigma_{1},\dots,\sigma_{n})\in\Delta_{\mathrm{sym}}(X_{1},\dots,X_{n}:\xi_{1}(N),\dots,\xi_{n}(N);N,m,\delta)$, and the result follows. ∎

Consequently, we denote all the quantities in (1.1) by the same $I_{\mathrm{sym}}(X_{1},\dots,X_{n})$ and those in (1.2) by $\overline{I}_{\mathrm{sym}}(X_{1},\dots,X_{n})$. We call $I_{\mathrm{sym}}(X_{1},\dots,X_{n})$ and $\overline{I}_{\mathrm{sym}}(X_{1},\dots,X_{n})$ the mutual information and upper mutual information of $(X_{1},\dots,X_{n})$, respectively. The terminology “mutual information” will be justified after the next theorem.

In the continuous variable case, our main result is the following exact relation of $I_{\mathrm{sym}}$ and $\overline{I}_{\mathrm{sym}}$ with the Boltzmann-Gibbs entropy $H(\cdot)$, which says that $I_{\mathrm{sym}}(X_{1},\dots,X_{n})$ is formally the sum of the separate entropies $H(X_{i})$’s minus the compound $H(X_{1},\dots,X_{n})$. Thus, a naive meaning of $I_{\mathrm{sym}}(X_{1},\dots,X_{n})$ is the entropy (or information) overlapping among the $X_{i}$’s.

Theorem 1.6**.**

[TABLE]

Proof.

If the coordinates $s_{i}$ of ${\mathbf{s}}\in{\mathbb{R}}^{N}$ are all distinct, then ${\mathbf{s}}$ is uniquely written as ${\mathbf{s}}=\sigma({\mathbf{x}})$ with ${\mathbf{x}}\in{\mathbb{R}}_{\leq}^{N}$ and $\sigma\in S_{N}$. Note that the set of ${\mathbf{s}}\in{\mathbb{R}}^{N}$ with $s_{i}=s_{j}$ for some $i\neq j$ is a closed subset of $\lambda_{N}$-measure zero. Under the correspondence

[TABLE]

(well defined on a co-negligible subset of ${\mathbb{R}}^{N}$), the measure $\lambda_{N}$ is transformed into the product of $\lambda_{N}|_{{\mathbb{R}}_{\leq}^{N}}$ and the counting measure on $S_{N}$.

In the following proof we adopt, due to Lemma 1.5, the description of $I_{\mathrm{sym}}$ and $\overline{I}_{\mathrm{sym}}$ as $I_{{\mathrm{sym}},R}(X_{1},\dots,X_{n})$ and $\overline{I}_{{\mathrm{sym}},R}(X_{1},\dots,X_{n})$ with $R:=\max_{1\leq i\leq n}\|X_{i}\|_{\infty}$. For each $N,m\in{\mathbb{N}}$ and $\delta>0$, suppose $({\mathbf{s}}_{1},\dots,{\mathbf{s}}_{n})\in\Delta_{R}(X_{1},\dots,X_{n};N,m,\delta)$ and write ${\mathbf{s}}_{i}=\sigma_{i}({\mathbf{x}}_{i})$ with ${\mathbf{x}}_{i}\in{\mathbb{R}}_{\leq}^{N}$ and $\sigma_{i}\in S_{N}$. Then it is obvious that

[TABLE]

By Lemma 1.2 and the fact stated at the beginning of the proof, we obtain

[TABLE]

This implies that

[TABLE]

Conversely, for each $m\in{\mathbb{N}}$ and $\delta>0$, by [5, Lemma 4.3] (also [4, 4.3.4]) there are an $m^{\prime}\in{\mathbb{N}}$ with $m^{\prime}\geq m$, a $\delta^{\prime}>0$ with $\delta^{\prime}\leq\delta/2$ and an $N_{0}\in{\mathbb{N}}$ such that for every $N\in{\mathbb{N}}$ and for every ${\mathbf{x}},{\mathbf{y}}\in{\mathbb{R}}_{\leq}^{N}$, if $\|{\mathbf{x}}\|_{\infty}\leq R$ and $|\kappa_{N}({\mathbf{x}}^{k})-\kappa_{N}({\mathbf{y}}^{k})|<2\delta^{\prime}$ for all $1\leq k\leq m^{\prime}$, then $\|{\mathbf{x}}-{\mathbf{y}}\|_{1}<\delta/2m(R+1)^{m-1}$. Suppose $N\geq N_{0}$ and

[TABLE]

so that $(\sigma_{1}({\mathbf{y}}_{1}),\dots,\sigma_{n}({\mathbf{y}}_{n}))\in\Delta_{R}(X_{1},\dots,X_{n};N,m^{\prime},\delta^{\prime})$ for some $({\mathbf{y}}_{1},\dots,{\mathbf{y}}_{n})\in({\mathbb{R}}_{\leq}^{N})^{n}$. Since

[TABLE]

for all $1\leq k\leq m^{\prime}$, we get $\|{\mathbf{x}}_{i}-{\mathbf{y}}_{i}\|_{1}<\delta/2m(R+1)^{m-1}$ for $1\leq i\leq n$. Therefore,

[TABLE]

for all $1\leq i_{1},\dots,i_{k}\leq n$ with $1\leq k\leq m$. This implies that $(\sigma_{1}({\mathbf{x}}_{1}),\dots,\sigma_{n}({\mathbf{x}}_{n}))\in\Delta_{R}(X_{1},\dots,X_{n};N,m,\delta)$. By Lemma 1.2 we obtain

[TABLE]

This implies by Lemma 1.2 once again that

[TABLE]

The result follows from (1.3) and (1.4). ∎

Let $\mu_{(X_{1},\dots,X_{n})}$ be the joint distribution measure on ${\mathbb{R}}^{n}$ of $(X_{1},\dots,X_{n})$ while $\mu_{X_{i}}$ is that of $X_{i}$ for $1\leq i\leq n$. Let $S(\mu_{(X_{1},\dots,X_{n})},\mu_{X_{1}}\otimes\dots\otimes\mu_{X_{n}})$ denote the relative entropy (or the Kullback-Leibler divergence) of $\mu_{(X_{1},\dots,X_{n})}$ with respect to the product measure $\mu_{X_{1}}\otimes\dots\otimes\mu_{X_{n}}$, i.e.,

[TABLE]

if $\mu_{(X_{1},\dots,X_{n})}$ is absolutely continuous with respect to $\mu_{X_{1}}\otimes\,\cdots\,\otimes\mu_{X_{n}}$; otherwise $S(\mu_{(X_{1},\dots,X_{n})},\mu_{X_{1}}\otimes\dots\otimes\mu_{X_{n}}):=+\infty$. When $H(X_{i})>-\infty$ for all $1\leq i\leq n$, one can easily verify that

[TABLE]

Thus, the above theorem yields the following:

Corollary 1.7**.**

If $H(X_{i})>-\infty$ for all $1\leq i\leq n$, then

[TABLE]

Corollary 1.8**.**

Under the same assumption as the above corollary, $I_{\mathrm{sym}}(X_{1},\dots,X_{n})=0$ if and only if $X_{1},\dots,X_{n}$ are independent.

In particular, the original mutual information $I(X_{1}\wedge X_{2})$ of two real random variables $X_{1},X_{2}$ is normally defined as

[TABLE]

Hence we have

[TABLE]

as long as $H(X_{1})>-\infty$ and $H(X_{2})>-\infty$ (and $X_{1},X_{2}$ are bounded). For this reason, we gave the term “mutual information” to $I_{\mathrm{sym}}$.

Finally, some open problems are in order:

• (1)

Without the assumption $H(X_{i})>-\infty$ for $1\leq i\leq n$, does $I_{\mathrm{sym}}(X_{1},\dots,X_{n})=\overline{I}_{\mathrm{sym}}(X_{1},\dots,X_{n})$ hold true?

• (2)

More strongly, does the limit such as

[TABLE]

or

[TABLE]

exist as in Lemma 1.2?

• (3)

Without the assumption $H(X_{i})>-\infty$ for $1\leq i\leq n$, does $I_{\mathrm{sym}}(X_{1},\dots,X_{n})=S(\mu_{(X_{1},\dots,X_{n})},\mu_{X_{1}}\otimes\cdots\otimes\mu_{X_{n}})$ hold true? Also, is $I_{\mathrm{sym}}(X_{1},\dots,X_{n})=0$ equivalent to the independence of $X_{1},\dots,X_{n}$?

• (4)

Although the boundedness assumption for $X_{1},\dots,X_{n}$ is rather essential in the above discussions, it is desirable to extend the results in this section to $X_{1},\dots,X_{n}$ not necessarily bounded but having all moments.

2. The discrete case

Let ${\mathcal{Y}}$ be a finite set with a probability measure $p$. The Shannon entropy of $p$ is

[TABLE]

For each sequence ${\mathbf{y}}=(y_{1},\dots,y_{N})\in{\mathcal{Y}}^{N}$, the type of ${\mathbf{y}}$ is a probability measure on ${\mathcal{Y}}$ given by

[TABLE]

The number of possible types is smaller than $(N+1)^{\#{\mathcal{Y}}}$. If $\nu$ is a type and ${\mathcal{T}}_{N}(\nu)$ denotes the set of all sequences of type $\nu$ from ${\mathcal{Y}}^{N}$, then the cardinality of ${\mathcal{T}}_{N}(\nu)$ is estimated as follows:

[TABLE]

(see [1, 12.1.3] and [2, Lemma 2.2]).

Let $p$ be a probability meausre on ${\mathcal{Y}}$. For each $N\in{\mathbb{N}}$ and $\delta>0$ we define $\Delta(p;N,\delta)$ to be the set of all sequences ${\mathbf{y}}\in{\mathcal{Y}}^{N}$ such that $|\nu_{\mathbf{y}}(t)-p(t)|<\delta$ for all $t\in{\mathcal{Y}}$. In other words, $\Delta(p;N,\delta)$ is the set of all $\delta$-typical sequeces (with respect to the measure $p$). Then the next lemma is well known.

Lemma 2.1**.**

[TABLE]

In fact, this easily follows from (2.1). Let $P_{N,\delta}$ be the maximizer of the Shannon entropy on the set of all types $\nu_{\mathbf{y}}$, ${\mathbf{y}}\in{\mathcal{Y}}^{N}$, such that $|\nu_{\mathbf{y}}(t)-p(t)|<\delta$ for all $t\in{\mathcal{Y}}$. We can use the Shannon entropy of the type class corresponding to $P_{N,\delta}$ to estimate the cardinality of $\Delta(p;N,\delta)$:

[TABLE]

It follows that

[TABLE]

and the lemma follows.

We consider the case where $p$ is the joint distribution of an $n$-tuple $(X_{1},\dots,X_{n})$ of discrete random variables on $(\Omega,{\mathbb{P}})$. Throughout this section we assume that the random variables $X_{1},\dots,X_{n}$ have their values in a finite set ${\mathcal{X}}=\{t_{1},\dots,t_{d}\}$.

Definition 2.2**.**

Let $p_{(X_{1},\dots,X_{n})}$ denote the joint distribution of $(X_{1},\dots,X_{n})$, which is a measure on ${\mathcal{X}}^{n}$ while the distribution $p_{X_{i}}$ of $X_{i}$ is a measure on ${\mathcal{X}}$, $1\leq i\leq n$. We write $\Delta(X_{i};N,\delta)$ for $\Delta(p_{X_{i}};N,\delta)$ and $\Delta(X_{1},\dots,X_{n};N,\delta)$ for $\Delta(p_{(X_{1},\dots,X_{n})};N,\delta)$.

Next, we introduce the counterparts of Definitions 1.3 and 1.4 in the discrete variable case.

Definition 2.3**.**

The action of $S_{N}$ on ${\mathcal{X}}^{N}$ is similar to that on ${\mathbb{R}}^{N}$ given in Defintion 1.3. For $N\in{\mathbb{N}}$ let ${\mathcal{X}}_{\leq}^{N}$ denote the set of all sequences of length $N$ of the form

[TABLE]

Oviously, such a sequence ${\mathbf{x}}$ is uniquely determined by $(N_{\mathbf{x}}(t_{1}),\dots,N_{\mathbf{x}}(t_{d}))$ or the type of ${\mathbf{x}}$. That is, ${\mathcal{X}}_{\leq}^{N}$ is regarded as the set of all types from ${\mathcal{X}}^{N}$. For each $N\in{\mathbb{N}}$ and $\delta>0$ we denote by $\Delta_{\mathrm{sym}}(X_{1},\dots,X_{n};N,\delta)$ the set of all $(\sigma_{1},\dots,\sigma_{n})\in S_{N}^{n}$ such that

[TABLE]

for some $({\mathbf{x}}_{1},\dots,{\mathbf{x}}_{n})\in({\mathcal{X}}_{\leq}^{N})^{n}$. Define

[TABLE]

and $\overline{I}_{\mathrm{sym}}(X_{1},\dots,X_{n})$ by replacing $\limsup$ by $\liminf$. Moreover, for each $1\leq i\leq n$, choose a sequence $\xi_{i}=\{\xi_{i}(N)\}$ of $\xi_{i}(N)=(\xi_{i}(N)_{1},\dots,\xi_{i}(N)_{N})\in{\mathcal{X}}_{\leq}^{N}$ such that $\nu_{\xi_{i}(N)}\to p_{X_{i}}$ as $N\to\infty$. We then define $\Delta_{\mathrm{sym}}(X_{1},\dots,X_{n}:\xi_{1}(N),\dots,\xi_{n}(N);N,\delta)$, $I_{\mathrm{sym}}(X_{1},\dots,X_{n}:\xi_{1},\dots,\xi_{n})$ and $\overline{I}_{\mathrm{sym}}(X_{1},\dots,X_{n}:\xi_{1},\dots,\xi_{n})$ as in Definition 1.4.

Lemma 2.4**.**

For any choices of approximating sequences $\xi_{1},\dots,\xi_{n}$ one has

[TABLE]

Proof.

It suffices to show that for each $\delta>0$ there are a $\delta^{\prime}>0$ and an $N_{0}\in{\mathbb{N}}$ such that

[TABLE]

for all $N\geq N_{0}$. Choose $\delta^{\prime}>0$ so that $3nd^{n+1}\delta^{\prime}\leq\delta$, where $d=\#{\mathcal{X}}$. Suppose $(\sigma_{1},\dots,\sigma_{n})$ is in the left-hand side of (2.2) so that $(\sigma_{1}({\mathbf{x}}_{1}),\dots,\sigma_{n}({\mathbf{x}}_{n}))\in\Delta(X_{1},\dots,X_{n};\allowbreak N,\delta^{\prime})$ for some $({\mathbf{x}}_{1},\dots,{\mathbf{x}}_{n})$, ${\mathbf{x}}_{i}=(x_{i1},\dots,x_{iN})\in{\mathcal{X}}_{\leq}^{N}$. Since

[TABLE]

[TABLE]

[TABLE]

it follows that

[TABLE]

for any $1\leq i\leq n$ and $t\in{\mathcal{X}}$. Now, choose an $N_{0}\in{\mathbb{N}}$ so that $|\nu_{\xi_{i}(N)}(t)-p_{X_{i}}(t)|<\delta^{\prime}$ and hence

[TABLE]

for any $1\leq i\leq n$ and $t\in{\mathcal{X}}$ and for all $N\geq N_{0}$. Since

[TABLE]

for every $1\leq l\leq d$ thanks to (2.5), it is easily seen that

[TABLE]

for any $1\leq i\leq n$. Hence we get

[TABLE]

so that thanks to (2.3)

[TABLE]

for every $(z_{1},\dots,z_{n})\in{\mathcal{X}}^{n}$. Therefore, $(\sigma_{1},\dots,\sigma_{n})$ is in the right-hand side of (2.2), as required. ∎

The next theorem is the discrete variable version of Theorem 1.6.

Theorem 2.5**.**

[TABLE]

Proof.

For each sequence $(N_{1},\dots,N_{d})$ of integers $N_{l}\geq 0$ with $\sum_{l=1}^{d}N_{l}=N$, let $S(N_{1},\dots,N_{d})$ denote the subgroup of $S_{N}$ consisting of products of permutations of $\{1,\dots,N_{1}\}$, $\{N_{1}+1,\dots,N_{1}+N_{2}\}$, $\dots$, $\{N_{1}+\dots+N_{d-1}+1,\dots,N\}$, and let

[TABLE]

be the set of left cosets of $S(N_{1},\dots,N_{d})$. For each ${\mathbf{x}}\in{\mathcal{X}}_{\leq}^{N}$ and $\sigma\in S_{N}$ we write $[\sigma]_{\mathbf{x}}$ for the left coset of $S(N_{\mathbf{x}}(t_{1}),\dots,N_{\mathbf{x}}(t_{d}))$ containing $\sigma$. Then it is clear that every ${\mathbf{s}}\in{\mathcal{X}}^{N}$ is represented as ${\mathbf{s}}=\sigma({\mathbf{x}})$ with a unique pair $({\mathbf{x}},[\sigma]_{\mathbf{x}})$ of ${\mathbf{x}}\in{\mathcal{X}}_{\leq}^{N}$ and $[\sigma]_{\mathbf{x}}\in S_{N}/S(N_{\mathbf{x}}(t_{1}),\dots,N_{\mathbf{x}}(t_{d}))$.

For any $\varepsilon>0$ one can choose a $\delta>0$ such that for every $1\leq i\leq n$ and every probability measure $p$ on ${\mathcal{X}}$, if $|p(t)-p_{X_{i}}(t)|<\delta$ for all $t\in{\mathcal{X}}$, then $|S(p)-S(p_{X_{i}})|<\varepsilon$. This implies that for each $N\in{\mathbb{N}}$ and $1\leq i\leq n$, one has $|S(\nu_{\mathbf{x}})-S(p_{X_{i}})|<\varepsilon$ whenever ${\mathbf{x}}\in\Delta(X_{i};N,\delta)$. Notice that $\Delta_{\mathrm{sym}}(X_{1},\dots,X_{n};N,\delta/d^{n-1})$ is the union of $[\sigma_{1}]_{{\mathbf{x}}_{1}}\times\dots\times[\sigma_{n}]_{{\mathbf{x}}_{n}}$ for all $({\mathbf{x}}_{1},\dots,{\mathbf{x}}_{n};[\sigma_{1}]_{{\mathbf{x}}_{1}},\dots,[\sigma_{n}]_{{\mathbf{x}}_{n}})$ of ${\mathbf{x}}_{i}\in{\mathcal{X}}_{\leq}^{N}$ and $[\sigma_{i}]_{{\mathbf{x}}_{i}}\in S_{N}/S(N_{{\mathbf{x}}_{i}}(t_{1}),\dots,N_{{\mathbf{x}}_{i}}(t_{d}))$ such that $(\sigma_{1}({\mathbf{x}}_{1}),\dots,\sigma_{n}({\mathbf{x}}_{n}))\in\Delta(X_{1},\dots,X_{n};N,\delta/d^{n-1})$. Now, suppose $({\mathbf{x}}_{1},\dots,{\mathbf{x}}_{n})\in({\mathcal{X}}_{\leq}^{N})^{n}$, $(\sigma_{1},\dots,\sigma_{n})\in S_{N}^{n}$ and $(\sigma_{1}({\mathbf{x}}_{1}),\dots,\sigma_{n}({\mathbf{x}}_{n}))\in\Delta(X_{1},\dots,X_{n};N,\delta/d^{n-1})$. Then, for each $1\leq i\leq n$ we get ${\mathbf{x}}_{i}\in\Delta(X_{i};N,\delta)$, i.e., $|\nu_{{\mathbf{x}}_{i}}(t)-p_{X_{i}}(t)|<\delta$ for all $t\in{\mathcal{X}}$ as (2.4). Hence we have

[TABLE]

so that

[TABLE]

Therefore,

[TABLE]

For each $1\leq i\leq n$ and for any ${\mathbf{x}}\in\Delta(X_{i};N,,\delta)$, the Stirling formula yields

[TABLE]

thanks to the above choice of $\delta>0$. Here, note that the $o(1)$ in the above estimate is uniform for ${\mathbf{x}}\in\Delta(X_{i};N,\delta)$. Hence, by (2), (2) and by Lemma 2.1 applied to $p_{(X_{1},\dots,X_{n})}$ on ${\mathcal{X}}^{n}$, we obtain

[TABLE]

and hence

[TABLE]

Next, we prove the converse direction. For any $\varepsilon>0$ choose a $\delta>0$ as above. For $N\in{\mathbb{N}}$ let $\Xi(N,\delta/d^{n-1})$ be the set of all $({\mathbf{x}}_{1},\dots,{\mathbf{x}}_{n})\in({\mathcal{X}}_{\leq}^{N})^{n}$ such that

[TABLE]

for some $(\sigma_{1},\dots,\sigma_{n})\in S_{N}^{n}$. Furthermore, for each $({\mathbf{x}}_{1},\dots,{\mathbf{x}}_{n})\in\Xi(N,\delta/d^{n-1})$, let $\Sigma({\mathbf{x}}_{1},\dots,{\mathbf{x}}_{n};N,\delta/d^{n-1})$ be the set of all

[TABLE]

such that $(\sigma_{1}({\mathbf{x}}_{1}),\dots,\sigma_{n}({\mathbf{x}}_{n}))\in\Delta(X_{1},\dots,X_{n};N,\delta/d^{n-1})$. Then it is obvious that

[TABLE]

When $({\mathbf{x}}_{1},\dots,{\mathbf{x}}_{n})\in\Xi(N,\delta/d^{n-1})$, we get ${\mathbf{x}}_{i}\in\Delta(X_{i};N,\delta)$ as (2.4) for $1\leq i\leq n$. Hence it is seen that

[TABLE]

For any fixed $({\mathbf{x}}_{1},\dots,{\mathbf{x}}_{n})\in\Xi(N,\delta/d^{n-1})$, suppose $([\sigma_{1}]_{{\mathbf{x}}_{1}},\dots,[\sigma_{n}]_{{\mathbf{x}}_{n}})\in\Sigma({\mathbf{x}}_{1},\dots,{\mathbf{x}}_{n};\allowbreak N,\delta/d^{n-1})$; then we get

[TABLE]

similarly to (2.6). Therefore,

[TABLE]

By (2.10)–(2) we obtain

[TABLE]

so that

[TABLE]

Since it follows similarly to (2) that

[TABLE]

with uniform $o(1)$ for all ${\mathbf{x}}\in\Delta(X_{i};N,\delta)$, we obtain

[TABLE]

by Lemma 2.1 again, and hence

[TABLE]

The conclusion follows from (2.9) and (2.13). ∎

In particular, the mutual information $I(X_{1}\wedge X_{2})$ of $X_{1}$ and $X_{2}$ is equivalently expressed as

[TABLE]

Similarly to the problem (2) mentioned in the last of Section 1, it is unknown whether the limit

[TABLE]

exists or not.