Renyi partial orders for BISO channels

TL;DR

This paper extends the understanding of partial orders in information theory by analyzing Renyi mutual information for BISO channels, establishing extremality of BSC and BEC with respect to generalized Renyi capacity.

Contribution

It introduces Renyi partial orders for BISO channels and proves the extremality of BSC and BEC in this new setting, generalizing previous results.

Findings

BSC and BEC are extremal in Renyi partial orders for BISO channels.

Introduction of $\\alpha$-Lorenz curves as a new analytical tool.

Generalization of capacity comparison tools to Renyi mutual information.

Abstract

A fundamental question in information theory is to quantify the loss of information under a noisy channel. Partial orders are typical tools to that end, however, they are often also challenging to evaluate. For the special class of binary input symmetric output (BISO) channels, Geng et al. showed that among channels with the same capacity, the binary symmetric channel (BSC) and binary erasure channel (BEC) are extremal with respect to the more capable order. Here we extend on this result by considering partial orders based on Renyi mutual information. We establish the extremality of the BSC and BEC in this setting with respect to the generalized Renyi capacity. In the process, we also generalize the needed tools and introduce $\alpha$-Lorenz curves.

Equations

$$D_{\alpha }(P\Vert Q)=\frac{1}{\alpha -1}\log \left(\sum _{a\in \mathcal{A}}{P}^{\alpha }(a)Q^{1-\alpha }(a)\right).$$

$$D_{\alpha }(P_{Y|X}\Vert Q_{Y|X}|P_X)=D_{\alpha }(P_{Y|X}{P}_X\Vert Q_{Y|X}{P}_X).$$

$$I_{\alpha }^S(X:Y)$$

$$=\frac{\alpha }{\alpha -1}\log \sum _{y\in \mathcal{Y}}{\left(\sum _{x\in \mathcal{X}}{P}_X(x)P_{Y|X=x}^{\alpha }(y)\right)}^{\frac{1}{\alpha }}.$$

$$C_{\alpha }(P_{Y|X})=\underset{P_X}{\sup }{I}_{\alpha }^S(X:Y).$$

$$I_{\alpha }^A(X:Y)$$

$$=\frac{\alpha }{\alpha -1}\log \sum _{y\in \mathcal{Y}}{\left(\sum _{x\in \mathcal{X}}{P}_{X_{\alpha }}(x)P_{Y|X=x}^{\alpha }(y)\right)}^{\frac{1}{\alpha }},$$

$$I(X:Y_1)\ge I(X:Y_2)\forall P_X.$$

$$I_{\alpha }^S(X:Y_1)\ge I_{\alpha }^S(X:Y_2)\forall P_X.$$

$$t_k=\sum _{i=1}^k(p_{{\pi }_i}^{\alpha }+p_{-{\pi }_i}^{\alpha }{)}^{\frac{1}{\alpha }}$$

$$f_{\alpha }(t)=k_{\frac{1}{\alpha }}\left(\frac{p_{{\pi }_y}^{\alpha }}{p_{{\pi }_y}^{\alpha }+p_{-{\pi }_y}^{\alpha }}\right)$$

$$F_{\alpha }(t)={\int }_0^t{f}_{\alpha }(\tau )\text{d}\tau .$$

$$F_{\alpha }(t_i)=\sum _{k=1}^i{\xi }_k{f}_{\alpha }(t_k)\le \sum _{k=1}^i{\xi }_k{g}_{\alpha }(t_k)=G_{\alpha }(t_i),i=1,\dots ,N$$

$$I_{\alpha }^S(X:Y)$$

$$=\frac{\alpha }{\alpha -1}\log {\int }_0^{d_C}{k}_{\frac{1}{\alpha }}\left(x\ast k_{\frac{1}{\alpha }}^{-1}(f(\tau ))\right)\text{d}\tau ,$$

$$C_{\alpha }(P_{Y|X})=\log 2-\frac{\alpha }{\alpha -1}\log \sum _{y>0}(p_y^{\alpha }+p_{-y}^{\alpha }{)}^{\frac{1}{\alpha }},$$

$$k_{\alpha }\left(k_{\alpha }^{-1}(x)\ast k_{\alpha }^{-1}(y)\right)$$

$$\sum _{j=k}^l{\xi }_j{x}_j\ge \sum _{j=k}^l{\xi }_j{y}_j,1\le k\le l,$$

$$\sum _{j=1}^l{\xi }_j\Lambda (x_j)\ge \sum _{j=1}^l{\xi }_j\Lambda (y_j).$$

$$F_{\alpha ,1}(t_i)=\sum _{k=1}^i{\xi }_k{f}_{\alpha ,1}(t_k)\le \sum _{k=1}^i{\xi }_k{f}_{\alpha ,2}(t_k)=F_{\alpha ,2}(t_i),i=1,\dots ,N.$$

$$F_{\alpha ,1}(d_C)={\int }_0^{d_C}{f}_{\alpha ,1}(\tau )\text{d}\tau$$

$$=\sum _{i=1}^N(p_{{\pi }_i}+p_{-{\pi }_i})=1.$$

$$\sum _{j=1}^N{\xi }_j{k}_{\frac{1}{\alpha }}\left(x\ast k_{\frac{1}{\alpha }}^{-1}(f_{\alpha ,1}(t_j))\right)\ge \sum _{j=1}^N{\xi }_j{k}_{\frac{1}{\alpha }}\left(x\ast k_{\frac{1}{\alpha }}^{-1}(f_{\alpha ,2}(t_j))\right),$$

$$\sum _{j=1}^N{\xi }_j{k}_{\frac{1}{\alpha }}\left(x\ast k_{\frac{1}{\alpha }}^{-1}(f_{\alpha ,1}(t_j))\right)\le \sum _{j=1}^N{\xi }_j{k}_{\frac{1}{\alpha }}\left(x\ast k_{\frac{1}{\alpha }}^{-1}(f_{\alpha ,2}(t_j))\right).$$

$$I_{\alpha }^S(X:Y_1)\ge I_{\alpha }^S(X:Y_2),\forall p(x),$$

$$I_{\frac{1}{3}}^S(X:Y)$$

$$=-\frac{1}{2}\log \left[1-x(1-x)\left[4-d_{C_{\frac{1}{3}}}\right]\right]$$

$$I_{\frac{1}{2}}^S(X:Y)$$

$$=-\log \left[1-2x(1-x)\left[2-d_{C_{\frac{1}{2}}}\right]\right].$$

$$W_1{⪰}_{m.c.}^{\frac{1}{3}}{W}_2,W_2{⪰}_{m.c.}^{\frac{1}{3}}{W}_1,$$

$$W_1{⪰}_{m.c.}^{\frac{1}{2}}{W}_2,W_2{⪰}_{m.c.}^{\frac{1}{2}}{W}_1.$$

$$BEC{⪰}_{m.c.}^{\alpha }W{⪰}_{m.c.}^{\alpha }BSC,$$

$$BSC{⪰}_{m.c.}^{\alpha }W{⪰}_{m.c.}^{\alpha }BEC.$$

$$I_{\alpha }^A(X:Y)$$

Full text

1 Introduction

Data processing inequalities state that information measures are monotone under the application of a noisy channel, i.e. information can only decrease along the transmission, qualifying them as distinguishability measures [13]. However, often we need to make more quantitative statements. To that end, information measure based partial orders allow to compare channels with respect to their data processing properties. A particular example is the more capable partial order [12]. With any such order it is an interesting question whether one can establish extremal channels among certain subclasses of channels. Geng et al. [4] established that among all binary input symmetric output (BISO) channels with the same capacity the binary erasure channel (BEC) and the binary symmetric channel (BSC) form the two extremes with respect to the more capable partial order. This leads to their main result that the inner and outer bounds for the corresponding BISO broadcast channel differ if and only if the two BISO channels are more capable comparable. Recent work in [11] extended on these results by considering the less noisy and degradable orders. In this case, capacity is not the natural common classifier, but several contraction coefficients play that role. Beyond those results, partial orders and contraction coefficients have found numerous further applications [2, 9, 17]. In this work, we take a different approach and consider generalizations of the more capable partial order using Rényi mutual information.

2 Preliminaries and technical tools

2.1 Notation

In this work, we mainly consider binary input symmetric output (BISO) channels with input alphabet ${\cal X}=\{0,1\}$ and output alphabet ${\cal Y}=\{0,\pm 1,\pm 2,\dots,\pm l\}$ for some integer $l\geq 1$, those are the channels for which $P_{Y|X}(y|0)=P_{Y|X}(-y|1)\coloneqq p_{y}$. We can always assume that the output alphabet ${\cal Y}$ has even number of elements because else we can split $Y=0$ into two outputs, $Y=0_{+}$ and $Y=0_{-}$, with $P_{Y|X}(0_{-}|0)=P_{Y|X}(0_{+}|0)=\frac{p_{0}}{2}$. The binary convolution is denoted by $a\ast b=a(1-b)+(1-a)b$ and the binary Rényi entropy is $h_{\alpha}(p)=\frac{1}{1-\alpha}\log\left(p^{\alpha}+(1-p)^{\alpha}\right)$, with the usual binary entropy as special case, $h(p)=\lim_{\alpha\to 1}h_{\alpha}(p)=-p\log(p)-(1-p)\log(1-p)$. We denote the Bernoulli distribution with probability $P(X=0)=p$ by $Ber(p)$.

2.2 Rényi mutual information

The mutual information can be expressed as a Kulback-Leibler divergence in several ways. It is hence natural to seek a Rényi generalization by instead using the Rényi divergence [14],

[TABLE]

Mostly for notational convenience, we also define the conditional Rényi divergence as

[TABLE]

Sibson [15] proposed the definition of an information radius which succinctly was generalized to that of a Rényi mutual information by Verdu [16]. That definition is

[TABLE]

The closed form expression can be found in [16, Equation (53)].

For a given $P_{Y|X}$, the maximum mutual information over all input distributions $P_{X}$ is sometimes called the capacity of order $\alpha$, see [1], and we denote it here by

[TABLE]

For simplicity we simply call it the $\alpha$-capacity. In a slight abuse of notation, we denote by $BISO(C_{\alpha})$ the set of all BISO channels with fixed capacity of order $\alpha$.

Another definition of Rényi mutual information was given by Arimoto [1] as

[TABLE]

where $P_{X_{\alpha}}(x)=\frac{P^{\alpha}_{X}(x)}{\sum_{x}P^{\alpha}_{X}(x)}$. Although this is generally a very different quantity, the $\alpha$-capacity does not change using this definition [3].

2.3 Rényi partial orders

Körner and Marton [12] defined a channel $W_{1}:X\mapsto Y_{1}$ to be more capable than a channel $W_{2}:X\mapsto Y_{2}$, denoted $W_{1}\succeq_{\operatorname{m.c.}}W_{2}$, if

[TABLE]

Here, we are interested in the generalization of this concept to Rényi entropies and make the following definition.

Definition 1

A channel $W_{1}:X\mapsto Y_{1}$ is said to be $\alpha$-more capable than the channel $W_{2}:X\mapsto Y_{2}$, denoted $W_{1}\succeq_{\operatorname{m.c.}}^{\alpha}W_{2}$, if

[TABLE]

We will later see that for the purpose of this work we could have also used Arimotos Rényi mutual information.

2.4 $\alpha$-Lorenz curves

In [4], Geng et. al. gave a definition of the Lorenz curve for BISO channels. Here, we give an extension inspired by Rényi entropies that we call the $\alpha$-Lorenz curve. We set $k_{\alpha}(p)=e^{(1-\alpha)h_{\alpha}(p)}$ and make the following definitions.

Definition 2 ($\alpha$-BISO partition and $\alpha$-BISO curve)

For a BISO channel with transition probabilities $\{p_{y},p_{-y}\}_{y}$, rearrange $k_{\frac{1}{\alpha}}\left(\frac{p_{y}^{\alpha}}{p_{y}^{\alpha}+p_{-y}^{\alpha}}\right)$ in ascending order and denote the permutation by $\pi$. The $\alpha$-BISO partition is defined as the partition of $[0,d_{C}]$ with points

[TABLE]

and $d_{C}=\sum_{y>0}(p_{y}^{\alpha}+p_{-y}^{\alpha})^{\frac{1}{\alpha}}$. We set $t_{0}=0$. The $\alpha$-BISO curve is defined as the stepwise function $f_{\alpha}(t)$ such that

[TABLE]

on $(t_{k-1},t_{k}]$, and $f_{\alpha}(0)=0$.

Note that the interval $[0,d_{C}]$ on which we define the partition depends on the channel. However, we will see later that the $d_{C}$ is solely determined by the capacity of the channel and therefore the same for all BISO channels with identical capacity. Equipped with this, we can make the following definition.

Definition 3 ($\alpha$-Lorenz curve of a BISO channel)

For a BISO channel with $\alpha$-BISO curve $f_{\alpha}(t)$, the $\alpha$-Lorenz curve $F_{\alpha}(t)$ is defined as

[TABLE]

The following result holds for $\alpha$-BISO curves.

Lemma 1

Given BISO channels $X\to Y$ and $X\to Z$ with equal capacity of order $\alpha$ and $\alpha$-BISO curves $f_{\alpha}(t)$ and $g_{\alpha}(t)$, respectively. Let the common refinement of these two BISO partitions be $\{t_{k}:k=0,\dots,N\}$, and $\xi_{k}=t_{k}-t_{k-1}$. Then

[TABLE]

if and only if the $\alpha$-Lorenz curve satisfies $F_{\alpha}(t)\leq G_{\alpha}(t)$ for all $t\in[0,d_{C}]$.

Proof

The proof is essentially the same as that of [4, Lemma 1] and essentially uses the observation that $F_{\alpha}(t)$ is a piecewise linear function.

We are now in the position to combine all concepts of this section and make statements about Rényi partial orders for BISO channels.

3 Main results

In this section we will only consider BISO channels. After some derivation, one can find that for a BISO channel $P_{Y|X}$, Sibson’s Rényi mutual information becomes

[TABLE]

where $x:=P(X=0)$ and we recall $k_{\alpha}(p)=e^{(1-\alpha)h_{\alpha}(p)}$. From here one can find that the $\alpha$-capacity is given by

[TABLE]

that is, it is optimized by $x=\frac{1}{2}$. This follows because $k_{\frac{1}{\alpha}}(x)$ is maximal at $x=\frac{1}{2}$ and $\frac{1}{2}\ast x=\frac{1}{2}$. This implies $C_{\alpha}(P_{Y|X})=\log 2-\frac{\alpha}{\alpha-1}\log d_{C}$, showing our earlier claim that $d_{C}$ depends solely on the $\alpha$-capacity.

We will need the following lemma, previously proven in the context of information combining.

Lemma 2 (Lemma IV.10 in [7])

The function

[TABLE]

is convex in both $x$ and $y$ for $0<\alpha<1$ and $2<\alpha\leq 3$ and concave for $1<\alpha\leq 2$ and $\alpha\geq 3$.

In particular, note that for $\alpha\in\{2,3\}$ the function is indeed linear in both $x$ and $y$. Again similar to [4], we will also need the following lemma.

Lemma 3 (Lemma 1 in [6])

Let $x_{1},\dots,x_{l}$ and $y_{1},\dots,y_{l}$ be nondecreasing sequences of real numbers. Let $\xi_{1},\dots,\xi_{l}$ be a sequence of real numbers such that

[TABLE]

with equality for $k=1$. Then for any convex function $\Lambda$,

[TABLE]

It is easy to check that in the above lemma if $\Lambda$ is concave instead, then Equation (3.6) holds with the direction of the inequality swapped.

With these result, we can establish the following theorem.

Theorem 3.1 (A sufficient condition)

Given BISO channels $W_{1}:X\mapsto Y_{1}$ and $W_{2}:X\mapsto Y_{2}$ with same $\alpha$-capacity $C_{\alpha}$ and $\alpha$-Lorenz curves $F_{\alpha,1}(t)$ and $F_{\alpha,2}(t)$, respectively. If $F_{\alpha,1}(t)\leq F_{\alpha,2}(t)$ then $W_{1}$ is $\alpha$-more capable than $W_{2}$, i.e. $W_{1}\succeq_{\operatorname{m.c.}}^{\alpha}W_{2}$, for all $\alpha>1$, $\frac{1}{2}\leq\alpha<1$ and $0<\alpha\leq\frac{1}{3}$. For $\frac{1}{3}\leq\alpha\leq\frac{1}{2}$ we have instead $W_{2}\succeq_{\operatorname{m.c.}}^{\alpha}W_{1}$.

Proof

Since both channels have the same capacity of order alpha, their Lorenz curves are defined on the same interval $[0,d_{C}]$. By Lemma 1, $F_{\alpha,1}(t)\leq F_{\alpha,2}(t)$ implies

[TABLE]

Next, we have,

[TABLE]

This holds indeed for all BISO channels, and therefore we have $F_{\alpha,1}(d_{C})=F_{\alpha,2}(d_{C})=1$. Furthermore, $f_{\alpha,1}(t_{k})$ and $f_{\alpha,2}(t_{k})$ are both nondecreasing and we can apply Lemma 3 with the function $\Lambda(y)=k_{\alpha}\left(x\ast k^{-1}_{\alpha}(y)\right)$. Due to the convexity properties in Lemma 2, this gives for $\alpha>1$ and $\frac{1}{3}\leq\alpha\leq\frac{1}{2}$,

[TABLE]

and for $\frac{1}{2}\leq\alpha<1$ and $0<\alpha\leq\frac{1}{3}$,

[TABLE]

Taking $\frac{\alpha}{\alpha-1}\log(\cdot)$ on both sides, noting the negative prefactor for $\alpha<1$, we have for $\alpha>1$, $\frac{1}{2}\leq\alpha<1$ and $0<\alpha\leq\frac{1}{3}$, that

[TABLE]

and for $\frac{1}{3}\leq\alpha\leq\frac{1}{2}$ with the inequality exchanged. This implies the claimed result.

For large ranges of $\alpha$ this shows the same behavior as the $\alpha=1$ case discussed in [4]. The notable exception is $\frac{1}{3}\leq\alpha\leq\frac{1}{2}$, where the implication is turned around. However, special attention should also be given to the settings where $\alpha\in\{\frac{1}{3},\frac{1}{2}\}$. As here both directions hold, ordered $\alpha$-Lorenz curves imply that the channels are equivalent in the $\alpha$-more capable ordering. Of course, then one might expext this to hold for any two channels, independent of their Lorenz curves. To that end, a calculation can show that,

[TABLE]

Hence, in terms of channel properties, the mutual information only depends on the $\alpha$-capacity of the channel. This gives immediately the following corollary.

Corollary 1

For any two BISO channels $W_{1}:X\mapsto Y_{1}$ and $W_{2}:X\mapsto Y_{2}$ with same $\alpha$-capacity $C_{\alpha}$, we have

[TABLE]

It follows also more generally that for these values of $\alpha\in\{\frac{1}{2},\frac{1}{3}\}$ the $\alpha$-more capable order is equivalent to comparing the $\alpha$-capacity between channels. Beyond these values, we find that, similar to the $\alpha=1$ case in [4], the binary erasure channel and the binary symmetric channel are the extrema with respect to the $\alpha$-more capable order.

Corollary 2

Let, $W$, $BSC$ and $BEC$ be channels with the same $\alpha$-capacity. Then, we have for $\alpha>1$, $\frac{1}{2}\leq\alpha<1$ and $0<\alpha\leq\frac{1}{3}$

[TABLE]

and for $\frac{1}{3}\leq\alpha\leq\frac{1}{2}$,

[TABLE]

Proof

By Theorem 3.1 it is sufficient to prove that the $\alpha$-Lorenz curves are ordered. To that end, recall that the $\alpha$-Lorenz curve is always a convex function. For the BSC, the only BISO channel with output dimension $2$, the curve is simply the straight line from [math] to $1$ on the interval $[0,d_{C}]$ and therefore an upper bound on any convex function with the same endpoints. The BEC is slightly more complicated. Consider $\alpha\geq 1$, then the curve starts with a derivative of $1$ on the interval $(0,1-\epsilon]$, which is indeed the smallest possible derivative in this case. Then, on the interval $(1-\epsilon,1-\epsilon+\epsilon 2^{\frac{1-\alpha}{\alpha}}]$, it has the maximal derivative $2^{\frac{\alpha-1}{\alpha}}$. Hence, the BEC curve always gives a lower bound. A similar argument holds for $\alpha<1$ but now the maximum derivative is $1$ and the minimum derivative is $2^{\frac{\alpha-1}{\alpha}}$.

3.1 Arimoto mutual information

So far, all derived results have been proven for the Sibson mutual information definition. Here we briefly point out that all of the above results also hold for Arimoto’s mutual information. Note that for BISO channels, we get

[TABLE]

with $x_{\alpha}=\frac{x^{\alpha}}{x^{\alpha}+(1-x)^{\alpha}}$. Hence, we can use the same definition for $\alpha$-Lorenz curves and now use the convexity or concavity of

[TABLE]

which still follows from Lemma 2 because we keep $\alpha$ fixed.

The Arimoto mutual information can be of particular interest, because it obeys a chain rule, allowing to translate the results also to conditional entropies,

[TABLE]

A similar result does not directly holds for the Rényi mutual information of Sibson.

4 Conclusions

In this work, we establish extremality result for a partial order based on Rényi mutual information. The results are closely connected to Rényi bounds on information combining [7, 8]. A natural direction for further investigation would be to go beyond BISO channels, include auxiliary random variables akin to the less noisy ordering or to consider quantum settings, where it might help to solve open problems in information combining [10]. One might expect that such results will have implications for contraction coefficients of Rényi divergences [5].

{credits}

4.0.1 Acknowledgements

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 550206990.

4.0.2 \discintname

The authors have no competing interests to declare that are relevant to the content of this article.