Sharp Bounds on Arimoto's Conditional R\'{e}nyi Entropies Between Two Distinct Orders

TL;DR

This paper derives precise bounds on Arimoto's conditional R'enyi entropy for different orders, identifying distributions that achieve these bounds and linking them to various information measures.

Contribution

It provides the first sharp bounds on Arimoto's conditional R'enyi entropy for distinct orders and characterizes the distributions that attain these bounds.

Findings

Identified distributions achieving the bounds.

Derived bounds applicable to other information measures.

Connected bounds to error probability and mutual information.

Abstract

This study examines sharp bounds on Arimoto's conditional R\'enyi entropy of order $\beta$ with a fixed another one of distinct order $\alpha \neq \beta$. Arimoto inspired the relation between the R\'enyi entropy and the $\ell_{r}$-norm of probability distributions, and he introduced a conditional version of the R\'enyi entropy. From this perspective, we analyze the $\ell_{r}$-norms of particular distributions. As results, we identify specific probability distributions whose achieve our sharp bounds on the conditional R\'enyi entropy. The sharp bounds derived in this study can be applicable to other information measures, e.g., the minimum average probability of error, the Bhattacharyya parameter, Gallager's reliability function $E_{0}$, and Sibson's $\alpha$-mutual information, whose are strictly monotone functions of the conditional R\'enyi entropy.

Equations

$$H_{\alpha }(X)=H_{\alpha }(P):=\underset{r\to \alpha }{\lim }\frac{r}{1-r}\ln \Vert P{\Vert }_r,$$

$$\displaystyle\|P\|_{r}\coloneqq\bigg{(}\sum_{x\in\operatorname{supp}(P)}P(x)^{r}\bigg{)}^{1/r}$$

$$H_{\alpha }(X)$$

$$H_0(X)$$

$$H_1(X)$$

$$H_{\infty }(X)$$

$$H_{\alpha }(X\mid Y):=\underset{r\to \alpha }{\lim }\frac{r}{1-r}\ln N_r(X\mid Y),$$

$$\displaystyle N_{r}(X\mid Y)\coloneqq\mathbb{E}\big{[}\|P_{X|Y}(\cdot\mid Y)\|_{r}\big{]}$$

$$H_{\alpha }(X\mid Y)$$

$$H_0(X\mid Y)$$

$$H_1(X\mid Y)$$

$$H_{\infty }(X\mid Y)$$

$$P_{\mathrm{e}}(X\mid Y):=\underset{f}{\min }\mathrm{Pr}(X\ne f(Y)),$$

$${\mathbf{v}}_n(p):=(v_0,v_1,v_2,\dots ,v_{n-1}),$$

$$v_i:=\{\begin{array}{ll}p& \mathrm{if}i=0,\\ \frac{1-p}{n-1}& \mathrm{otherwise}\end{array}$$

$$\mathbf{w}(p):=(w_0,w_1,w_2,\dots ),$$

$$w_i:=\{\begin{array}{ll}p& \mathrm{if}0\le i<\lfloor 1/p\rfloor ,\\ 1-\lfloor 1/p\rfloor p& \mathrm{if}i=\lfloor 1/p\rfloor ,\\ 0& \mathrm{if}i>\lfloor 1/p\rfloor \end{array}$$

$$\Vert {\mathbf{v}}_n(p){\Vert }_r$$

$$\Vert \mathbf{w}(p){\Vert }_r$$

$$H_{\alpha }({\mathbf{v}}_n(p))$$

$$H_{\alpha }(\mathbf{w}(p))$$

$$\frac{\partial \Vert {\mathbf{v}}_n(p){\Vert }_r}{\partial p}$$

$$\displaystyle=\frac{1}{r}\Big{(}p^{r}+(n-1)^{1-r}\,(1-p)^{r}\Big{)}^{(1/r)-1}\bigg{(}\frac{\partial}{\partial p}\Big{(}p^{r}+(n-1)^{1-r}\,(1-p)^{r}\Big{)}\bigg{)}$$

$$\displaystyle=\frac{1}{r}\Big{(}p^{r}+(n-1)^{1-r}\,(1-p)^{r}\Big{)}^{(1/r)-1}\Big{(}r\,p^{r-1}-r\,(n-1)^{1-r}\,(1-p)^{r-1}\Big{)}$$

$$\displaystyle=\Big{(}p^{r}+(n-1)^{1-r}\,(1-p)^{r}\Big{)}^{(1/r)-1}\Big{(}p^{r-1}-(n-1)^{1-r}\,(1-p)^{r-1}\Big{)}.$$

$$\mathrm{sgn}(x):=\{\begin{array}{ll}-1& \mathrm{if}x<0,\\ 0& \mathrm{if}x=0,\\ 1& \mathrm{if}x>0\end{array}$$

$$\displaystyle\operatorname{sgn}\bigg{(}\frac{\partial\|\boldsymbol{v}_{n}(p)\|_{r}}{\partial p}\bigg{)}$$

$$\displaystyle=\operatorname{sgn}\Big{(}p^{r-1}-(n-1)^{1-r}\,(1-p)^{r-1}\Big{)}$$

$$=\{\begin{array}{ll}-1& \mathrm{if}r<1,\\ 0& \mathrm{if}r=1,\\ 1& \mathrm{if}r>1\end{array}$$

$$\frac{\partial \Vert \mathbf{w}(p){\Vert }_r}{\partial p}$$

$$\displaystyle=\frac{1}{r}\Big{(}m\,p^{r}+\big{(}1-m\,p\big{)}^{r}\Big{)}^{(1/r)-1}\bigg{(}\frac{\partial}{\partial p}\Big{(}m\,p^{r}+\big{(}1-m\,p\big{)}^{r}\Big{)}\bigg{)}$$

$$\displaystyle=\frac{1}{r}\Big{(}m\,p^{r}+\big{(}1-m\,p\big{)}^{r}\Big{)}^{(1/r)-1}\Big{(}r\,m\,p^{r-1}-r\,m\big{(}1-m\,p\big{)}^{r-1}\Big{)}$$

$$\displaystyle=m\,\Big{(}m\,p^{r}+\big{(}1-m\,p\big{)}^{r}\Big{)}^{(1/r)-1}\Big{(}p^{r-1}-\big{(}1-m\,p\big{)}^{r-1}\Big{)}$$

$$\displaystyle\operatorname{sgn}\bigg{(}\frac{\partial\|\boldsymbol{w}(p)\|_{r}}{\partial p}\bigg{)}$$

$$\displaystyle=\Big{(}p^{r-1}-\big{(}1-m\,p\big{)}^{r-1}\Big{)}$$

$$=\{\begin{array}{ll}-1& \mathrm{if}r<1,\\ 0& \mathrm{if}r=1,\\ 1& \mathrm{if}r>1\end{array}$$

$$\underset{p\to (1/m{)}^{+}}{\lim }\Vert \mathbf{w}(p){\Vert }_r$$

$$\displaystyle=\Big{(}(m-1)\,\big{(}1/m\big{)}^{r}+\Big{(}1-(m-1)\,\big{(}1/m\big{)}\Big{)}^{r}\Big{)}^{1/r}$$

$$\displaystyle=\Big{(}(m-1)\,m^{-r}+m^{-r}\Big{)}^{1/r}$$

$$=m^{(1-r)/r}$$

$$=\Vert \mathbf{w}(1/m){\Vert }_r$$

Full text

I Introduction

In information theory, the Shannon entropy $H(X)$ and the conditional Shannon entropy $H(X\mid Y)$ [34] are traditional information measures of random variables (RVs) $X$ and $Y$, whose characterize several theoretical limits for information transmission. Later, the Rényi entropy $H_{\alpha}(X)$ [26] was axiomatically proposed as a generalized Shannon entropy with order $\alpha$. For a discrete RV111An RV $X$ with distribution $P$ is denoted by $X\sim P$. $X\sim P$, the Rényi entropy of order $\alpha\in[0,\infty]$ is defined by

[TABLE]

where $\ln$ denotes the natural logarithm, the $\ell_{r}$-norm of a discrete probability distribution $P$ is defined by

[TABLE]

for $r\in\mathbb{R}$, and $\operatorname{supp}(P)\coloneqq\{x\in\mathcal{X}\mid P(x)>0\}$ denotes the support of a distribution $P$ on a countable alphabet $\mathcal{X}$. Note that (1) is well-defined since the limiting value exists for each $\alpha\in[0,\infty]$ as follows:

[TABLE]

where $|\cdot|$ denotes the cardinality222In this study, suppose that $|\mathcal{S}|=\infty$ if $\mathcal{S}$ is a countably infinite set; thus, note in (4) that $H_{0}(X)=\infty$ if $\operatorname{supp}(P)$ is countably infinite. of the countable set, $\mathbb{E}[\cdot]$ denotes the expectation of the RV, and $\|P\|_{\infty}\coloneqq\lim_{r\to\infty}\|P\|_{r}=\max_{x\in\operatorname{supp}(P)}P(x)$ denotes the $\ell_{\infty}$-norm of $P$. Moreover, Arimoto [2] proposed a conditional version333There are many definition of conditional Rényi entropy (cf. [13, 37, 38]). In this paper, the conditional Rényi entropy means Arimoto’s definition unless otherwise noted. of the Rényi entropy $H_{\alpha}(X\mid Y)$ as a generalized conditional Shannon entropy with order $\alpha$. For a pair of RVs $(X,Y)\sim P_{X|Y}P_{Y}$, the conditional Rényi entropy [2] of order $\alpha\in[0,\infty]$ is defined by

[TABLE]

where the expectation of $\ell_{r}$-norm is denoted by

[TABLE]

for $r\in(0,\infty]$, and note in (8) that $Y\sim P_{Y}$. In this study, the RV $Y$ can be considered to be either discrete or continuous. By convention, we write $H_{\alpha}(X\mid Y=y)\coloneqq H_{\alpha}(P_{X|Y}(\cdot\mid y))$ for $y\in\operatorname{supp}(P_{Y})$. As with the unconditional Rényi entropy (1), note that (7) is also well-defined since the limiting value also exists for each $\alpha\in[0,\infty]$ as follows:444Proofs of (10)–(12) can be found in, e.g., [13, Propositions 1 and 2].

[TABLE]

Note that Arimoto [2] proposed $H_{\alpha}(X\mid Y)$ in terms of the relation between the $\ell_{r}$-norm and the unconditional Rényi entropy, shown in (1) (see also [41, Section II-A]). As shown in (5) and (11), Rényi’s information measures can be reduced to Shannon’s information measures as $\alpha\to 1$. In many situations, Rényi’s information measures derive stronger results than Shannon’s information measures (cf. [1, 6, 7, 9, 38]). In addition, the quantity $H_{\alpha}(X\mid Y)$ is closely related to Gallager’s reliability function $E_{0}$ [14, Eq. (5.6.14)] and Sibson’s $\alpha$-mutual information [35, 21]; and thus, coding theorems with them can be written by $H_{\alpha}(X\mid Y)$ (cf. [2, 41]). Many basic properties of $H_{\alpha}(X\mid Y)$ were studied by Fehr and Berens [13].

Bounds on information measures are crucial tools in several engineering fields, e.g., information theory, coding theory, cryptology, machine learning, statistics, etc. In this paper, a bound is said to be sharp if there is no tighter bound than it in the same situation. One of well-known sharp bounds is Fano’s inequality [11], which bounds the conditional Shannon entropy $H(X\mid Y)$ from above for fixed (i) average probability of error $\Pr(X\neq f(Y))$ and (ii) size of support $|\!\operatorname{supp}(P_{X})|$, where the function $f$ is an estimator of $X$ given $Y$. As related bounds, the reverse of Fano’s inequality, i.e., sharp lower bounds on $H(X\mid Y)$ with a fixed minimum average probability of error

[TABLE]

were established by Kovalevsky [23] and Tebbe and Dwyer [36] (see also [12]). Ho and Verdú [20] generalized Fano’s inequality by relaxing its constraints from fixed number $|\!\operatorname{supp}(P_{X})|$ to fixed distribution $P_{X}$. Very recently, Sason and Verdú [33] generalized Fano’s inequality and the reverse of it to sharp bounds on the conditional Rényi entropy $H_{\alpha}(X\mid Y)$. In their study [33], interplay between $H_{\alpha}(X\mid Y)$ and $P_{\mathrm{e}}(X\mid Y)$ was investigated with broad applications and comparisons to related works. On the other hand, we [29] derived sharp bounds on $H(X\mid Y)$ with a fixed $H_{\alpha}(X\mid Y)$, and vice versa, by analyzing interplay between $H(X\mid Y)$ and $N_{r}(X\mid Y)$ (cf. (7)). Unconditional versions of its results [29] were also examined in [30].

In this study, we further generalize interplay between Shannon’s information measure and Rényi’s information measure of our results [30, 29] to interplay between two Rényi’s information measures with distinct orders $\alpha\neq\beta$. We start to analyze extremal probability distributions $\boldsymbol{v}_{n}(\cdot)$ and $\boldsymbol{w}(\cdot)$ defined in Section II, where the extremal distributions means that our sharp bounds on $H_{\alpha}(X)$ can be achieved by them (cf. Section III). To utilize the nature of the expectation $N_{r}(X\mid Y)$ of $\ell_{r}$-norm, our analyses of this study are concentrated on the $\ell_{r}$-norm of extremal distributions. Main results of this study are shown in Section IV, which show sharp bounds on $H_{\beta}(X\mid Y)$ with a fixed another one $H_{\alpha}(X\mid Y)$, $\alpha\neq\beta$, in several situation. In this study, we represent our bounds via specific distributions to ensure sharpnesses of the bounds. The main results of this study are organized as follows:

• •

Section IV-A shows sharp bounds on $H_{\beta}(X\mid Y)$ with fixed $H_{\alpha}(X\mid Y)$ and the cardinality $|\!\operatorname{supp}(P_{X})|<\infty$ for distinct orders $\alpha\neq\beta$ as follows:

• –

Theorem 5 gives bounds on $H_{\alpha}(X\mid Y)$ with a fixed $H_{\infty}(X\mid Y)$ for $\alpha\in(0,\infty)$, and vice versa.

• –

Theorem 6 gives bounds on $H_{\beta}(X\mid Y)$ with a fixed $H_{\alpha}(X\mid Y)$ for $\alpha,\beta\in[1/2,\infty]$ and $|\!\operatorname{supp}(P_{X})|\leq 2$.

• –

Theorem 7 gives bounds on $H_{\beta}(X\mid Y)$ with a fixed $H_{\alpha}(X\mid Y)$ for $\alpha,\beta\in[1/2,\infty]$ and $|\!\operatorname{supp}(P_{X})|\geq 3$.

• •

Section IV-B shows sharp bounds on $H_{\beta}(X\mid Y)$ with a fixed $H_{\alpha}(X\mid Y)$ for two orders $\alpha\in(0,1)\cup(1,\infty]$ and $\beta\in(0,\infty]$, as shown in Theorem 8. Note that unlike Theorems 5–7, Theorem 8 has no constraint of the support $\operatorname{supp}(P_{X})$.

Finally, Section V shows some applications of sharp bounds on the conditional Rényi entropy to other related information measures, whose are strictly monotone functions of the conditional Rényi entropy.

II Extremal Distributions $\boldsymbol{v}_{n}(\cdot)$ and $\boldsymbol{w}(\cdot)$ and Their Properties

In this subsection, we introduce the probability distributions $\boldsymbol{v}_{n}(\cdot)$ and $\boldsymbol{w}(\cdot)$, which play significant roles in this study. In addition, the $\ell_{r}$-norms and Rényi entropy of them are investigated. Until Section III, we defer to show extremality of these distributions $\boldsymbol{v}_{n}(\cdot)$ and $\boldsymbol{w}(\cdot)$ in terms of the $\ell_{r}$-norm and Rényi entropy.

For each $n\in\mathbb{N}$ and $p\in[1/n,1]$, we define the $n$-dimensional probability vector555Note in [30, Eq. (3)] that the probability vector $\boldsymbol{v}_{n}(\cdot)$ is defined by another form; however, a simple change of variables immediately shows that these are essentially equavalent.

[TABLE]

where $\mathbb{N}$ denotes the set of positive integers and $v_{i}$ is chosen so that

[TABLE]

for each $i\in\{0,1,2,\dots,n-1\}$. In addition, for $p\in(0,1]$, we define the infinite-dimensional probability vector

[TABLE]

where $w_{i}$ is chosen so that

[TABLE]

for each $i\in\{0,1,2\dots\}$, and $\lfloor x\rfloor\coloneqq\max\{z\in\mathbb{Z}\mid z\leq x\}$ denotes the floor function of $x\in\mathbb{R}$. Note that $|\!\operatorname{supp}(\boldsymbol{v}_{n}(p))|=n$ for every $p\in[1/n,1)$, and $|\!\operatorname{supp}(\boldsymbol{w}(p))|=m+1$ for every $m\in\mathbb{N}$ and $p\in[1/(m+1),1/m)$, i.e., these are discrete probability distributions with finite supports. Since $\boldsymbol{v}_{1}(1)$ has only one probability mass $1$ whenever $n=1$, we omit its trivial case in our analyses; and assume that $n\in\mathbb{N}_{\geq 2}$ in this study, where $\mathbb{N}_{\geq k}$ denotes the set of integers $n$ satisfying $n\geq k$. By the definition (2) of $\ell_{r}$-norm, for each $r\in(0,\infty)$, the $\ell_{r}$-norms of these distributions $\boldsymbol{v}_{n}(\cdot)$ and $\boldsymbol{w}(\cdot)$ can be calculated as follows:

[TABLE]

respectively. In particular, the $\ell_{\infty}$-norms are $\|\boldsymbol{v}_{n}(p)\|_{\infty}=p$ for $p\in[1/n,1]$ and $\|\boldsymbol{w}(p)\|_{\infty}=p$ for $p\in(0,1]$. Substituting (18) and (19) into (1), the Rényi entropies of the distributions $\boldsymbol{v}_{n}(\cdot)$ and $\boldsymbol{w}(\cdot)$, respectively, can also be calculated as follows:

[TABLE]

respectively. We first show the monotonicities of $\ell_{r}$-norm of the distributions $\boldsymbol{v}_{n}(\cdot)$ and $\boldsymbol{w}(\cdot)$ in the following lemma.

Lemma 1**.**

Let $r\in(0,1)\cup(1,\infty]$ and $n\in\mathbb{N}_{\geq 2}$ be fixed numbers. If $r\in(0,1)$, then both $\ell_{r}$-norms $p_{v}\mapsto\|\boldsymbol{v}_{n}(p_{v})\|_{r}$ and $p_{w}\mapsto\|\boldsymbol{w}(p_{w})\|_{r}$ are strictly decreasing functions of $p_{v}\in[1/n,1]$ and $p_{w}\in(0,1]$, respectively. Conversely, if $r\in(1,\infty]$, then both $\ell_{r}$-norms $p_{v}\mapsto\|\boldsymbol{v}_{n}(p_{v})\|_{r}$ and $p_{w}\mapsto\|\boldsymbol{w}(p_{w})\|_{r}$ are strictly increasing functions of $p_{v}\in[1/n,1]$ and $p_{w}\in(0,1]$, respectively.

Proof:

Since $\|\boldsymbol{v}_{n}(p_{v})\|_{\infty}=p_{v}$ and $\|\boldsymbol{w}(p_{w})\|_{\infty}=p_{w}$ for $p_{v}\in[1/n,1]$ and $p_{w}\in(0,1]$, respectively, Lemma 1 is trivial if $r=\infty$. Hence, it suffices to consider the $\ell_{r}$-norm for $r\in(0,1)\cup(1,\infty)$.

We first verify the monotonicity of the function $p\mapsto\|\boldsymbol{v}_{n}(p)\|_{r}$. A direct calculation shows

[TABLE]

If we define the sign function as

[TABLE]

for $x\in\mathbb{R}$, then it follows that

[TABLE]

for every $n\in\mathbb{N}_{\geq 2}$, $p\in(1/n,1)$, and $r\in(0,\infty)$. This implies that for any fixed $n\in\mathbb{N}_{\geq 2}$,

• •

if $r\in(0,1)$, then $p\mapsto\|\boldsymbol{v}_{n}(p)\|_{r}$ is strictly decreasing for $p\in[1/n,1]$,

• •

if $r\in(1,\infty)$, then $p\mapsto\|\boldsymbol{v}_{n}(p)\|_{r}$ is strictly increasing for $p\in[1/n,1]$;

and therefore, the assertion of Lemma 1 holds for $p\mapsto\|\boldsymbol{v}_{n}(p)\|_{r}$.

We next verify the monotonicity of the function $p\mapsto\|\boldsymbol{w}(p)\|_{r}$. Since $\lfloor 1/p\rfloor=m$ for each $p\in(1/(m+1),1/m]$ and $m\in\mathbb{N}$, we readily see that

[TABLE]

for every $m\in\mathbb{N}$, $p\in(1/(m+1),1/m)$, and $r\in(0,\infty)$. Hence, we obtain

[TABLE]

for every $m\in\mathbb{N}$, $p\in(1/(m+1),1/m)$, and $r\in(0,\infty)$. This implies that for each fixed $m\in\mathbb{N}$ and $r\in(0,1)\cup(1,\infty)$,

• •

if $r\in(0,1)$, then $p\mapsto\|\boldsymbol{w}(p)\|_{r}$ is strictly decreasing for $p\in(1/(m+1),1/m]$,

• •

if $r\in(1,\infty)$, then $p\mapsto\|\boldsymbol{w}(p)\|_{r}$ is strictly increasing for $p\in(1/(m+1),1/m]$.

Finally, it follows that

[TABLE]

for each $m\in\mathbb{N}_{\geq 2}$ and $r\in(0,1)$, which implies that $p\mapsto\|\boldsymbol{w}(p)\|_{r}$ is continuous on $p\in(0,1]$; therefore, the monotonicity of $p\mapsto\|\boldsymbol{w}(p)\|_{r}$ come from (36) can be improved as follows:

• •

if $r\in(0,1)$, then $p\mapsto\|\boldsymbol{w}(p)\|_{r}$ is strictly decreasing for $p\in(0,1]$,

• •

if $r\in(1,\infty)$, then $p\mapsto\|\boldsymbol{w}(p)\|_{r}$ is strictly increasing for $p\in(0,1]$.

This completes the proof of Lemma 1. ∎

Lemma 1 implies the existences of inverse functions. Let666Eq. (42) is defined to fulfill $\theta(\infty)=-1$.

[TABLE]

and let $\mathcal{I}_{n}(r)$ and $\mathcal{J}(r)$ be real intervals defined by

[TABLE]

for each $n\in\mathbb{N}_{\geq 2}$ and $r\in(0,1)\cup(1,\infty]$, respectively. For each $r\in(0,1)\cup(1,\infty]$ and $n\in\mathbb{N}_{\geq 2}$, we denote by

[TABLE]

inverse functions of $p_{v}\mapsto\|\boldsymbol{v}_{n}(p_{v})\|_{r}$ and $p_{w}\mapsto\|\boldsymbol{w}(p_{w})\|_{r}$, respectively. As simple instances of them, if $r=\infty$, then $N_{\infty}^{-1}(\boldsymbol{v}_{n}:t_{v})=t_{v}$ and $N_{\infty}^{-1}(\boldsymbol{w}:t_{w})=t_{w}$ for $t_{v}\in[1/n,1]$ and $t_{w}\in(0,1]$, respectively, because $\|\boldsymbol{v}_{n}(p_{v})\|_{\infty}=p_{v}$ and $\|\boldsymbol{w}(p_{w})\|_{\infty}=p_{w}$ for $p_{v}\in[1/n,1]$ and $p_{w}\in(0,1]$, respectively.

Since logarithm functions are strictly monotone, it also follows from (1) and Lemma 1 that both Rényi entropies $p_{v}\mapsto H_{\alpha}(\boldsymbol{v}_{n}(p_{v}))$ and $p_{w}\mapsto H_{\alpha}(\boldsymbol{w}(p_{w}))$ also have inverse functions for every777If $\alpha=1$, i.e., if these are Shannon entropies, these inverse functions also exist due to [30, Lemma 1]. $\alpha\in(0,\infty]$, as with (45) and (46). For each $n\in\mathbb{N}_{\geq 2}$ and $\alpha\in(0,\infty]$, we denote by

[TABLE]

inverse functions of $p_{v}\mapsto H_{\alpha}(\boldsymbol{v}_{n}(p_{v}))$ and $p_{w}\mapsto H_{\alpha}(\boldsymbol{w}(p_{w}))$, respectively. By convention of the Shannon entropy, we write $H^{-1}(\boldsymbol{v}_{n}:\cdot)$ and $H^{-1}(\boldsymbol{w}:\cdot)$ as the inverse functions $H_{1}^{-1}(\boldsymbol{v}_{n}:\cdot)$ and $H_{1}^{-1}(\boldsymbol{w}:\cdot)$ with $\alpha=1$, respectively. In general, these inverse functions are hard-to-express in closed-forms, as with the inverse function of the binary entropy function $h_{2}:t\mapsto-t\ln t-(1-t)\ln(1-t)$. As special cases of them, we give the following specific closed-forms.

Fact 1**.**

If $\alpha=1/2$, $\alpha=2$, or $\alpha=\infty$, then the inverse functions (47) and (48) can be expressed in the following closed-forms:

[TABLE]

where $\mathrm{e}$ denotes the base of natural logarithm.

Fact 1 can be verified by the quadratic formula in the case of888If $\alpha=\infty$, then Fact 1 is almost trivial from the definition (6). $\alpha=1/2$ and $\alpha=2$. In Fig. 1, we illustrate instances of the inverse functions of Fact 1, along with the inverse functions $H^{-1}(\boldsymbol{v}_{n}:\cdot)$ and $H^{-1}(\boldsymbol{w}:\cdot)$ of the Shannon entropies. As with Fact 1, from the relation between the Rényi entropy and $\ell_{r}$-norm (cf. (1)), the inverse functions $N_{r}^{-1}(\boldsymbol{v}_{n}:\cdot)$ of (45) and $N_{r}^{-1}(\boldsymbol{w}:\cdot)$ of (46) can also be expressed in closed-forms if $r=1/2$, $r=2$, or $r=\infty$. By Fact 1, sharp bounds established in this paper can be expressed in closed-forms in some situations.

Using the inverse functions $H^{-1}(\boldsymbol{v}_{n}:\cdot)$ and $H^{-1}(\boldsymbol{w}:\cdot)$ of the Shannon entropies, we introduce relations of the convexity/concavity of the $\ell_{r}$-norm with respect to the Shannon entropy of distributions $\boldsymbol{v}_{n}(\cdot)$ and $\boldsymbol{w}(\cdot)$ in Lemmas 2 and 3, respectively.

Lemma 2** ([29, Lemma 2]).**

If $n=2$, for each $r\in(0,1)\cup(1,\infty)$, the $\ell_{r}$-norm $\mu\mapsto\|\boldsymbol{v}_{2}(H^{-1}(\boldsymbol{v}_{2}:\mu))\|_{r}$ is strictly concave in $\mu\in[0,\ln 2]$. In addition999This concavity is shown in not [29, Lemma 2] but the below paragraph of [29, Lemma 2]., for each $n\in\mathbb{N}_{\geq 2}$, the $\ell_{\infty}$-norm $\mu\mapsto\|\boldsymbol{v}_{n}(H^{-1}(\boldsymbol{v}_{n}:\mu))\|_{\infty}$ is strictly concave in $\mu\in[0,\ln n]$. Moreover, for each $n\in\mathbb{N}_{\geq 3}$ and $r\in[1/2,1)\cup(1,\infty)$, there exists an inflection point $\chi_{n}(r)\in(0,\ln n)$ such that satisfies the following:

• •

the $\ell_{r}$-norm $\mu\mapsto\|\boldsymbol{v}_{n}(H^{-1}(\boldsymbol{v}_{n}:\mu))\|_{r}$ is strictly concave in $\mu\in[0,\chi_{n}(r)]$,

• •

the $\ell_{r}$-norm $\mu\mapsto\|\boldsymbol{v}_{n}(H^{-1}(\boldsymbol{v}_{n}:\mu))\|_{r}$ is strictly convex in $\mu\in[\chi_{n}(r),\ln n]$.

Lemma 3** ([29, Lemma 3]101010In [29, Lemma 3], the case $r=\infty$ is not considered; however, it can also be proved by the fact that $\|\boldsymbol{w}(p)\|_{\infty}=p$ for $p\in(0,1]$, as with Lemma 2.).**

For each $m\in\mathbb{N}$ and $r\in(0,1)\cup(1,\infty]$, the $\ell_{r}$-norm $\mu\mapsto\|\boldsymbol{w}(H^{-1}(\boldsymbol{w}:\mu))\|_{r}$ is strictly concave in $\mu\in[\ln m,\ln(m+1)]$.

In [29], Lemmas 2 and 3 were used to derive sharp bounds on the conditional Shannon entropy $H(X\mid Y)$ with a fixed conditional Rényi entropy $H_{\alpha}(X\mid Y)$, and vice versa, from perspectives of the expectation of (8) and (11). In this study, we establish sharp bounds on $H_{\beta}(X\mid Y)$ with a fixed $H_{\alpha}(X\mid Y)$ for distinct orders $\alpha\neq\beta$ in a similar manner to [29], i.e., the property of expectation (8) are employed. To this end, we further examine the convexity/concavity of $\ell_{r}$-norms with respect to $\ell_{s}$-norm for distributions $\boldsymbol{v}_{n}(\cdot)$ and $\boldsymbol{w}(\cdot)$, as with Lemmas 2 and 3, respectively. To derive such convexity/concavity lemmas, we now give the following Lemma 4.

Lemma 4**.**

We define the function

[TABLE]

for each $n\in\mathbb{N}_{\geq 2}$, $z\in(0,\infty)$, and $r,s\in(0,\infty)$, where the $q$-logarithm function111111Note that the limiting value $\lim_{q\to 1}(x^{1-q}-1)/(1-q)=\ln x$ can be verified by L’Hôpital’s rule. [40] is defined by

[TABLE]

for $x>0$ and $q\in\mathbb{R}$. Then, the following three assertions hold:

• •

For any $n\in\mathbb{N}_{\geq 2}$, any $z\in(0,1)$, and any $0<r<s<\infty$, it holds that

[TABLE]

• •

if $n=2$, then for any $z\in(1,\infty)$ and $1/2\leq r<s<\infty$, it holds that

[TABLE]

• •

for any $n\geq\mathbb{N}_{\geq 3}$ and any $1/2\leq r<s<\infty$, there exists $\zeta(n;r,s)\in(1,\infty)$ such that

[TABLE]

for every $z\in(1,\infty)$.

Lemma 4 is proved in Appendix A. Defining121212In (60), suppose that $\gamma(\infty,\infty)=1$.

[TABLE]

we present the convexity/concavity of $\ell_{s}$-norms of $\boldsymbol{v}_{n}(\cdot)$ and $\boldsymbol{w}(\cdot)$ with respect to $\ell_{r}$-norms of them, $r\neq s$, in Lemmas 5 and 6, respectively. We emphasize that Lemma 4 is a key lemma for deriving Lemmas 5 and 6.

Lemma 5**.**

For each $n\in\mathbb{N}_{\geq 2}$ and $r,s\in(0,1)\cup(1,\infty)$, it holds that

• •

the $\ell_{\infty}$-norm $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{\infty}$ is strictly concave in $t\in\mathcal{I}_{n}(r)$,

• •

if $s\in(0,1)$, then $t\mapsto\|\boldsymbol{v}_{n}(N_{\infty}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ is strictly concave in $t\in[1/n,1]$,

• •

if $s\in(1,\infty)$, then $t\mapsto\|\boldsymbol{v}_{n}(N_{\infty}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ is strictly convex in $t\in[1/n,1]$.

Moreover, if $n=2$, then it holds that for any distinct $r,s\in[1/2,1)\cup(1,\infty)$,

• •

if $\gamma(r,s)>1$, then $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ is strictly convex in $t\in\mathcal{I}_{n}(r)$,

• •

if $\gamma(r,s)<1$, then $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ is strictly concave in $t\in\mathcal{I}_{n}(r)$.

Furthermore, for each $n\in\mathbb{N}_{\geq 3}$ and distinct $r,s\in[1/2,1)\cup(1,\infty)$, there exists an inflection point $\tau(n;r,s)\in\mathcal{I}_{n}(r)\setminus\{1,n^{\theta(r)}\}$ such that

• •

if $\gamma(r,s)>1$, then

• –

the $\ell_{s}$-norm $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ is strictly convex in $t\in\mathcal{I}_{n}^{(1)}(r,s)$,

• –

the $\ell_{s}$-norm $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ is strictly concave in $t\in\mathcal{I}_{n}^{(2)}(r,s)$,

• •

if $\gamma(r,s)<1$, then

• –

the $\ell_{s}$-norm $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ is strictly convex in $t\in\mathcal{I}_{n}^{(2)}(r,s)$,

• –

the $\ell_{s}$-norm $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ is strictly concave in $t\in\mathcal{I}_{n}^{(1)}(r,s)$,

where real intervals $\mathcal{I}_{n}^{(1)}(r,s)$ and $\mathcal{I}_{n}^{(2)}(r,s)$ are defined by

[TABLE]

respectively.

Note that the convexity and the concavity of Lemma 5 are switched each other according to either $\gamma(r,s)>1$ or $\gamma(r,s)<1$. We illustrate two regions of pairs $(r,s)$ which fulfill $\gamma(r,s)>1$ and $\gamma(r,s)<1$, respectively, in Fig. 2.

Proof:

In a similar way to the proofs of [15, Lemma 1] and [29, Lemma 2], we prove this lemma by verifying signs of derivatives. A simple calculation yields

[TABLE]

where (a) follows by the definition

[TABLE]

and (b) follows from the fact that

[TABLE]

Then, we obtain

[TABLE]

for every $n\in\mathbb{N}_{\geq 2}$, $p\in(1/n,1)$, and $r\in(0,\infty)$. By the inverse function theorem, we have

[TABLE]

for every $n\in\mathbb{N}_{\geq 2}$, $r\in(0,1)\cup(1,\infty)$, and $t\in\mathcal{I}_{n}(r)\setminus\{1,n^{\theta(r)}\}$, where $\mathcal{I}_{n}(\cdot)$ is defined in (43), and the variables $t$ and $p$ are chosen to satisfy $\|\boldsymbol{v}_{n}(p)\|_{r}=t$ (cf. the definition (45) of $N_{r}^{-1}(\boldsymbol{v}_{n}:\cdot)$), i.e.,

[TABLE]

In particular, since $\|\boldsymbol{v}_{n}(p)\|_{\infty}=p$ for $p\in[1/n,1]$, it follows from (29) and (79) that

[TABLE]

for every $n\in\mathbb{N}_{\geq 2}$, $r\in(0,1)\cup(1,\infty)$, and $t\in\mathcal{I}_{n}(r)\setminus\{1,n^{\theta(r)}\}$. Moreover, since $N_{\infty}^{-1}(\boldsymbol{v}_{n}:t)=t$ for $t\in[1/n,1]$, we also get

[TABLE]

for every $n\in\mathbb{N}_{\geq 2}$, $t\in[1/n,1]$, and $s\in(0,\infty)$. Therefore, it follows from (86) and (88) that

• •

for each $n\in\mathbb{N}_{\geq 2}$ and $r\in(0,1)\cup(1,\infty)$, the $\ell_{\infty}$-norm $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{\infty}$ is strictly concave in $t\in\mathcal{I}_{n}(r)$,

• •

for each $n\in\mathbb{N}_{\geq 2}$ and $s\in(0,1)$, the $\ell_{s}$-norm $t\mapsto\|\boldsymbol{v}_{n}(N_{\infty}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ is strictly concave in $t\in[1/n,1]$,

• •

for each $n\in\mathbb{N}_{\geq 2}$ and $s\in(1,\infty)$, the $\ell_{s}$-norm $t\mapsto\|\boldsymbol{v}_{n}(N_{\infty}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ is strictly convex in $t\in[1/n,1]$.

Henceforth, we consider the convexity/concavity of $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ with respect to $t\in\mathcal{I}_{n}(r)$ for each distinct $r,s\in(0,1)\cup(1,\infty)$. By the chain rule of derivatives, we have

[TABLE]

where (a) follows from

• •

the change of variables as

[TABLE]

• •

and the fact that

[TABLE]

Since $p\in(1/n,1)$ for $t\in\mathcal{I}_{n}(r)\setminus\{1,n^{\theta(r)}\}$ (cf. (82)), it suffices to consider the range of variable $z$ of (95) on $z\in(1,\infty)$. A further calculation derives

[TABLE]

for every $n\in\mathbb{N}_{\geq 2}$, distinct $r,s\in(0,1)\cup(1,\infty)$, and $t\in\mathcal{I}_{n}(r)\setminus\{1,n^{\theta(r)}\}$. That is, the convexity/concavity of $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ with respect to $t\in\mathcal{I}_{n}(r)$ depend on the sign of $g(n,z;r,s)$. If $n=2$, we have from (58) of Lemma 4 and (116) that

[TABLE]

for every distinct $r,s\in[1/2,1)\cup(1,\infty)$ and $t\in\mathcal{I}_{2}(r)\setminus\{1,n^{\theta(r)}\}$, where $\gamma(r,s)$ is defined in (60). This implies the assertion of Lemma 5 for $n=2$.

Furthermore, we verify the assertion of Lemma 5 for $n\in\mathbb{N}_{\geq 3}$. It is clear from (95) that $p\mapsto z(n,p)$ is strictly increasing for $p\in[1/n,1)$. Moreover, it follows from (29) that

• •

if $r\in(0,1)$, then $p\mapsto\|\boldsymbol{v}_{n}(p)\|_{r}$ is strictly decreasing for $p\in[1/n,1]$,

• •

if $r\in(1,\infty)$, then $p\mapsto\|\boldsymbol{v}_{n}(p)\|_{r}$ is strictly increasing for $p\in[1/n,1]$.

Hence, we observe from the relation $N_{r}^{-1}(\boldsymbol{v}_{n}:t)=p$ that

• •

it holds that $\lim_{t\to 1}z(n,N_{r}^{-1}(\boldsymbol{v}_{n}:t))=\lim_{p\to 1}z(n,p)=\infty$,

• •

it holds that $z(n,N_{r}^{-1}(\boldsymbol{v}_{n}:n^{\theta(r)}))=z(n,1/n)=1$,

• •

if $r\in(0,1)$, then $t\mapsto z(n,N_{r}^{-1}(\boldsymbol{v}_{n}:t))$ is strictly decreasing for $t\in\mathcal{I}_{n}(r)\setminus\{1\}$,

• •

if $r\in(1,\infty)$, then $t\mapsto z(n,N_{r}^{-1}(\boldsymbol{v}_{n}:t))$ is strictly increasing for $t\in\mathcal{I}_{n}(r)\setminus\{1\}$.

Therefore, it follows from (59) of Lemma 4 and (116) that for any $n\in\mathbb{N}_{\geq 3}$ and distinct $r,s\in[1/2,1)\cup(1,\infty)$, there exists $\tau(n;r,s)\in\mathcal{I}_{n}(r)\setminus\{1,n^{\theta(r)}\}$ such that satisfies the following:

• •

if $r<s<1$, then

[TABLE]

for every $t\in\mathcal{I}_{n}(r)\setminus\{1,n^{\theta(r)}\}$,

• •

if $r<1<s$, then

[TABLE]

for every $t\in\mathcal{I}_{n}(r)\setminus\{1,n^{\theta(r)}\}$,

• •

if $1<r<s$, then

[TABLE]

for every $t\in\mathcal{I}_{n}(r)\setminus\{1,n^{\theta(r)}\}$,

• •

if $s<r<1$, then

[TABLE]

for every $t\in\mathcal{I}_{n}(r)\setminus\{1,n^{\theta(r)}\}$,

• •

if $s<1<r$, then

[TABLE]

for every $t\in\mathcal{I}_{n}(r)\setminus\{1,n^{\theta(r)}\}$,

• •

if $1<s<r$, then

[TABLE]

for every $t\in\mathcal{I}_{n}(r)\setminus\{1,n^{\theta(r)}\}$,

Combining (119)–(124), we obtain that for every $n\in\mathbb{N}_{\geq 3}$, distinct $r,s\in[1/2,1)\cup(1,\infty)$, and $t\in\mathcal{I}_{n}(r)$,

• •

if $\gamma(r,s)>1$, then

[TABLE]

• •

if $\gamma(r,s)<1$, then

[TABLE]

where $\mathcal{I}_{n}^{(1)}(r,s)$ and $\mathcal{I}_{n}^{(2)}(r,s)$ are defined in (61) and (62), respectively. This completes the proof of Lemma 5. ∎

Lemma 6**.**

Define the real interval $\mathcal{J}_{m}(r)$ by131313Note in (127) that for every $m\in\mathbb{N}$, it holds that $m^{\theta(r)}<(m+1)^{\theta(r)}$ if $r\in(0,1)$, and $(m+1)^{\theta(r)}<m^{\theta(r)}$ if $r\in(1,\infty]$.

[TABLE]

For each $m\in\mathbb{N}$ and distinct $r,s\in(0,1)\cup(1,\infty]$, the following convexity/concavity holds:

• •

if $\gamma(r,s)>1$, then $t\mapsto\|\boldsymbol{w}(N_{r}^{-1}(\boldsymbol{w}:t))\|_{s}$ is strictly convex in $t\in\mathcal{J}_{m}(r)$,

• •

if $\gamma(r,s)<1$, then $t\mapsto\|\boldsymbol{w}(N_{r}^{-1}(\boldsymbol{w}:t))\|_{s}$ is strictly concave in $t\in\mathcal{J}_{m}(r)$.

Proof:

In a similar manner to the proof of [29, Lemma 3], we also prove this lemma by verifying signs of derivatives, as with the proof of Lemma 5. A simple calculation yields

[TABLE]

for every $m\in\mathbb{N}$, $r\in(0,\infty)$, and $p\in(1/(m+1),1/m)$, where (a) follows by the definition

[TABLE]

and (b) follows from the fact that

[TABLE]

Then, we obtain

[TABLE]

for every $m\in\mathbb{N}$, $p\in(1/(m+1),1/m)$, and $r\in(0,\infty)$. By the inverse function theorem, we have

[TABLE]

for every $m\in\mathbb{N}$, $r\in(0,1)\cup(1,\infty)$, and $t\in\mathcal{J}_{m}(r)\setminus\{m^{\theta(r)},(m+1)^{\theta(r)}\}$, where $\mathcal{J}_{m}(\cdot)$ is defined in (127), and the variables $t$ and $p$ are chosen to satisfy $\|\boldsymbol{w}(p)\|_{r}=t$ (cf. the definition (46) of $N_{r}^{-1}(\boldsymbol{w}:\cdot)$), i.e.,

[TABLE]

In particular, since $\|\boldsymbol{w}(p)\|_{\infty}=p$ for $p\in(0,1]$, it follows from (36) and (144) that

[TABLE]

for every $m\in\mathbb{N}$, $r\in(0,1)\cup(1,\infty)$, and $t\in\mathcal{J}_{m}(r)\setminus\{m^{\theta(r)},(m+1)^{\theta(r)}\}$. Moreover, since $N_{\infty}^{-1}(\boldsymbol{w}:t)=t$ for $t\in(0,1]$, we also get

[TABLE]

for every $m\in\mathbb{N}$, $t\in(1/(m+1),1/m)$, and $s\in(0,1)\cup(1,\infty)$. Therefore, it follows from (151) and (153) that

• •

for each $m\in\mathbb{N}$ and $r\in(0,1)\cup(1,\infty)$, the $\ell_{\infty}$-norm $t\mapsto\|\boldsymbol{w}(N_{r}^{-1}(\boldsymbol{w}:t))\|_{\infty}$ is strictly concave in $t\in\mathcal{J}_{m}(r)$,

• •

for each $m\in\mathbb{N}$ and $s\in(0,1)$, the $\ell_{s}$-norm $t\mapsto\|\boldsymbol{w}(N_{\infty}^{-1}(\boldsymbol{w}:t))\|_{s}$ is strictly concave in $t\in[1/(m+1),1/m]$,

• •

for each $m\in\mathbb{N}$ and $s\in(1,\infty)$, the $\ell_{s}$-norm $t\mapsto\|\boldsymbol{w}(N_{\infty}^{-1}(\boldsymbol{w}:t))\|_{s}$ is strictly convex in $t\in[1/(m+1),1/m]$.

Henceforth, we consider the convexity/concavity of $t\mapsto\|\boldsymbol{w}(N_{r}^{-1}(\boldsymbol{w}:t))\|_{s}$ with respect to $t\in\mathcal{J}_{m}(r)$ for each distinct $r,s\in(0,1)\cup(1,\infty)$. By the chain rule of derivatives, we have

[TABLE]

where (a) follows from

• •

the change of variables as

[TABLE]

• •

the fact that

[TABLE]

Since $p\in(1/(m+1),1/m)$ for $t\in\mathcal{J}_{m}(r)\setminus\{m^{\theta(r)},(m+1)^{\theta(r)}\}$ (cf. (147)), it suffices to consider the range of variable $z$ of (160) on $z\in(0,1)$. A further calculation derives

[TABLE]

for $m\in\mathbb{N}$, distinct $r,s\in(0,1)\cup(1,\infty)$, and $t\in\mathcal{J}_{m}(r)\setminus\{m^{\theta(r)},(m+1)^{\theta(r)}\}$. That is, the convexity/concavity of $t\mapsto\|\boldsymbol{w}(N_{r}^{-1}(\boldsymbol{w}:t))\|_{s}$ with respect to $t\in\mathcal{J}_{m}(r)$ depend on the sign of $g(m+1,z;r,s)$. Combining (57) of Lemma 4 and (182), we have

[TABLE]

for every $m\in\mathbb{N}$, distinct $r,s\in(0,1)\cup(1,\infty)$, and $t\in\mathcal{J}_{m}(r)\setminus\{m^{\theta(r)},(m+1)^{\theta(r)}\}$, where $\gamma(r,s)$ is defined in (60). This completes the proof of Lemma 6. ∎

III Sharp Bounds on Unconditional Rényi Entropy

In this section, we introduce sharp bounds on the Rényi entropy $H_{\beta}(X)$ with a fixed another one $H_{\alpha}(X)$, studied in [28]. We first show extremality of the distribution $\boldsymbol{v}_{n}(\cdot)$ defined in (14) in terms of the relation between $\ell_{r}$-norm and $\ell_{s}$-norm in the following theorem.

Theorem 1** ([28, Lemma 2]).**

Let $P$ be a discrete probability distribution with finite support, and let $n=|\!\operatorname{supp}(P)|$. For any $r,s\in(0,1)\cup(1,\infty]$, it holds that

[TABLE]

with $p=N_{r}^{-1}(\boldsymbol{v}_{n}:\|P\|_{r})$, where $N_{r}^{-1}(\boldsymbol{v}_{n}:\cdot)$ and $\gamma(r,s)$ are defined in (45) and (60), respectively.

Proof:

In the original version [28, Lemma 2], we wrote this proposition as follows: Let $P$ be a discrete probability distributions with finite support, i.e., $|\!\operatorname{supp}(P)|=n$ for some $n\in\mathbb{N}_{\geq 2}$. For any $r\in(0,1)\cup(1,\infty]$, there exists $p\in[1/n,1]$ such that

[TABLE]

It is obvious that the value $p$ satisfying (187) is determined as $p=N_{r}^{-1}(\boldsymbol{v}_{n}:\|P\|_{r})$. It follows from the definition (60) of $\gamma(r,s)$ that

[TABLE]

Moreover, the case $\gamma(r,s)=1$ implies $r=s$, i.e., it is a trivial case. Therefore, the statements of (187)–(189) shown in [28, Lemma 2] can be rewritten as Theorem 1. ∎

We second show extremality of the distribution $\boldsymbol{w}(\cdot)$ defined in (16) in terms of the relation between $\ell_{r}$-norm and $\ell_{s}$-norm in the following theorem.

Theorem 2** ([28, Lemma 3]).**

Let $P$ be a discrete probability distribution with possibly countably infinite support. For any $r,s\in(0,1)\cup(1,\infty]$, it holds that

[TABLE]

with $p=N_{r}^{-1}(\boldsymbol{w}:\|P\|_{r})$, where $N_{r}^{-1}(\boldsymbol{w}:\cdot)$ and $\gamma(r,s)$ are defined in (46) and (60), respectively.

Proof:

In the proof of [28, Lemma 2], we considered only for finite-dimensional probability vectors as follows: Let $\boldsymbol{p}=(p_{1},p_{2},\dots,p_{n})$ be an $n$-dimensional probability vector satisfying

[TABLE]

Since the equiprobable distribution is a trivial case, suppose that $\boldsymbol{p}=(1/n,1/n,\dots,1/n)$ is omitted. Let $k\in\{2,3,\dots,n-1\}$ and $l\in\{k+1,k+2,\dots,n\}$ be positive integers chosen so that

[TABLE]

where

[TABLE]

denotes the components of $\boldsymbol{p}$ in decreasing order141414We used this notation by following the book of Marshall and Olkin [24].. Then, total derivatives of the probability vector $\boldsymbol{p}$ was considered in the following assumptions:

[TABLE]

and the parameter $r\in(0,1)\cup(1,\infty)$ is fixed. Due to the hypothesis of (196), the support size of $\boldsymbol{p}$ is $l\in\mathbb{N}$; thus, [28, Lemma 2] only proved for probability distributions with finite support.

Fortunately, considering infinite-dimensional probability vector $\boldsymbol{p}=(p_{1},p_{2},\dots)$ and extendind the hypothesis of (201) for every $m\in\{l+1,l+2,\dots\}$, we can remove the hypothesis of the finite support. That is, the analyses of the proof of [28, Lemma 2] can naturally generalized to probability distributions with possibly countably infinite support.

Moreover, in the proof of [28, Lemma 2], we examined $\ell_{\infty}$-norm by majorization theory [24]. This analysis can also be extended from finite- to infinite-dimensional probability vectors, as with the proof of [20, Theorem 10] studied by Ho and Verdú. ∎

We now consider the function

[TABLE]

for each $\alpha\in(0,1)\cup(1,\infty]$ and $t>0$, where $\theta(\cdot)$ is defined in (42). Since

• •

it follows from (1) that $H_{\alpha}(P)=f_{\alpha}(\|P\|_{\alpha})$ for every $\alpha\in(0,1)\cup(1,\infty]$,

• •

if $\alpha\in(0,1)$, then $t\mapsto f_{\alpha}(t)$ is strictly increasing for $t>0$,

• •

if $\alpha\in(1,\infty]$, then $t\mapsto f_{\alpha}(t)$ is strictly decreasing for $t>0$,

Theorems 1 and 2 can be rewritten from sharp bounds on the $\ell_{s}$-norm $\|P\|_{s}$ to sharp bounds on the Rényi entropy $H_{\beta}(P)$ as shown in the following two theorems:

Theorem 3** ([28, Theorem 2]).**

Let $P$ be a discrete probability distribution with finite support, i.e., $|\!\operatorname{supp}(P)|=n$ for some $n\in\mathbb{N}$. Then, it holds that

[TABLE]

with $p=H_{\alpha}^{-1}(\boldsymbol{v}_{n}:H_{\alpha}(P))$, where $H_{\alpha}^{-1}(\boldsymbol{v}_{n}:\cdot)$ is defined in (47).

Theorem 4** ([28, Theorem 2]).**

Let $P$ be a discrete probability distribution with possibly infinite support. For any $\alpha\in(0,\infty]$, it holds that

[TABLE]

with $p=H_{\alpha}^{-1}(\boldsymbol{w}:H_{\alpha}(P))$, where $H_{\alpha}^{-1}(\boldsymbol{w}:\cdot)$ is defined in (48).

Note that Theorem 3 has a constraint of finite supports, but Theorem 4 enables us to consider countably infinite supports. If $\alpha=\infty$, then Theorem 3 is equivalent to the result by Ben-Bassat and Raviv [4, Theorem 6]; and if $\alpha=\infty$, then Theorem 4 is a stronger result than [4, Theorems 4 and 5]. In addition, Theorems 3 and 4 yield same joint ranges of pairs $(H_{\alpha}(P),H_{\beta}(P))$ considered in [16]. In [28, Theorem 2], Theorems 3 and 4 are organized in one theorem. However, in this study, Theorems 2 and 4 are extended from probability distributions with finite support to countably infinite support. Due to such extension, Theorems 3 and 4 are divided, and Theorem 4 is generalized to possibly countably infinite support. Since Theorems 3 and 4 are due to Theorems 1 and 2, and the strict monotonicity of the logarithm functions, as with Theorems 3 and 4, we can establish sharp bounds on other definitions of entropy [3, 5, 10, 18, 39], which are strictly monotonic for the $\ell_{r}$-norm of a probability distribution (cf. [30, Table I]).

In the next section, using the sharp bounds introduced in this section, we further consider to extend them to sharp bounds on the conditional Rényi entropy $H_{\alpha}(X\mid Y)$.

IV Sharp Bounds on Arimoto’s Conditional Rényi Entropy

IV-A Bounds Established from Distribution $\boldsymbol{v}_{n}(\cdot)$

In this subsection, by using the extremality of the distribution $\boldsymbol{v}_{n}(\cdot)$ discussed in Section III, we derive sharp bounds on $H_{\beta}(X\mid Y)$ with two fixed $H_{\alpha}(X\mid Y)$ and $|\!\operatorname{supp}(P_{X})|$ in some situations. We first give the sharp bounds, whose mean interplay between $H_{\alpha}(X\mid Y)$ and $H_{\infty}(X\mid Y)$ in the following theorem.

Theorem 5**.**

Let $X$ be an RV in which $|\!\operatorname{supp}(P_{X})|=n\in\mathbb{N}$, and let $Y$ be an arbitrary RV. For any $\alpha\in(0,\infty)$, it holds that

[TABLE]

with $p_{1}=H_{\infty}^{-1}(\boldsymbol{v}_{n}:H_{\infty}(X\mid Y))$ and $p_{2}=H_{\alpha}^{-1}(\boldsymbol{v}_{n}:H_{\alpha}(X\mid Y))$, respectively, where $H_{\alpha}^{-1}(\boldsymbol{v}_{n}:\cdot)$ is defined in (47).

Proof:

Let $X$ be an RV in which $|\!\operatorname{supp}(P_{X})|=n$ for some151515If $|\!\operatorname{supp}(P_{X})|=1$, it is clear that $H_{\alpha}(X\mid Y)=0$ for every $\alpha\in[0,\infty]$. That is, we omit such trivial cases in our analyses. $n\in\mathbb{N}_{\geq 2}$, and let $Y$ be an arbitrary RV. If $\alpha=1$, then (207) is equivalent to Fano’s inequality. In fact, Inequality (207) is equivalent to the right-hand inequalities of [23, Eq. (15)] and [36, Eq. (5)]. On the other hand, Inequality (208) with $\alpha=1$ can be verified as follows:

[TABLE]

with $p=H^{-1}(\boldsymbol{v}_{n}:H(X\mid Y))$, where (a) follows from (203) of Theorem 3 with $\alpha=1$ and $\beta=\infty$, Equalities (b) follow from the fact that $\|\boldsymbol{v}_{n}(p)\|_{\infty}=p$ for $p\in[1/n,1]$, and (c) follows from Jensen’s inequality and the fact that $\mu\mapsto H^{-1}(\boldsymbol{v}_{n}:\mu)$ is strictly concave in $\mu\in[0,\ln n]$. Note that the concavity of $\mu\mapsto H^{-1}(\boldsymbol{v}_{n}:\mu)$ can be verified by the following two facts:

• •

the function161616The function $h_{2}:t\mapsto-t\ln t-(1-t)\ln(1-t)$ denotes the binary entropy function. $p\mapsto H(\boldsymbol{v}_{n}(p))=h_{2}(p)+(1-p)\ln(n-1)$ is strictly decreasing for $p\in[1/n,1]$,

• •

the function $p\mapsto H(\boldsymbol{v}_{n}(p))=h_{2}(p)+(1-p)\ln(n-1)$ is strictly concave in $p\in[1/n,1]$.

Therefore, both bounds of Theorem 5 hold for $\alpha=1$.

We next consider to prove (207) of Theorem 5 for $\alpha\in(0,1)\cup(1,\infty)$. Let $s\in(0,1)\cup(1,\infty)$ be a fixed number. Note that

[TABLE]

where $\gamma(\cdot,\cdot)$ is defined in (60). If $\gamma(\infty,s)<1$, we have

[TABLE]

with $p=N_{\infty}(X\mid Y)$, where Inequality (a) follows from (186) of Theorem 1, Inequality (b) follows from Jensen’s inequality and fact that $t\mapsto\|\boldsymbol{v}_{n}(N_{\infty}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ is strictly concave in $t\in[1/n,1]$ (cf. Lemma 5), and Equality (c) follows from the fact that $N_{\infty}^{-1}(\boldsymbol{v}_{n}:t)=t$ for $t\in[1/n,1]$. Analogously, if $\gamma(\infty,s)>1$, then we also get

[TABLE]

with $p=N_{\infty}(X\mid Y)$. We now define

[TABLE]

for $\alpha\in(0,1)\cup(1,\infty)$ and $t>0$. Since

• •

it holds that $H_{\alpha}(X\mid Y)=f_{\alpha}(N_{\alpha}(X\mid Y))$ for every $\alpha\in(0,1)$,

• •

if $\alpha\in(0,1)$, then $t\mapsto f_{\alpha}(t)$ is a strictly increasing function of $t>0$,

• •

if $\alpha\in(1,\infty)$, then $t\mapsto f_{\alpha}(t)$ is a strictly decreasing function of $t>0$,

it follows from (227) and (228) that

[TABLE]

with $p=N_{\infty}(X\mid Y)$ for every $\alpha\in(0,1)\cup(1,\infty)$. In addition, since

[TABLE]

we get from (230) that

[TABLE]

with $p=H_{\infty}^{-1}(\boldsymbol{v}_{n}:H_{\infty}(X\mid Y))$ for every $\alpha\in(0,1)\cup(1,\infty)$ rather than $p=N_{\infty}(X\mid Y)$, which is (207) of Theorem 5.

We further consider to prove (208) of Theorem 5 for $\alpha\in(0,1)\cup(1,\infty)$. Note that

[TABLE]

Thus, we have

[TABLE]

with $p=N_{r}^{-1}(\boldsymbol{v}_{n}:N_{r}(X\mid Y))$, where (a) follows from (186) of Theorem 1, and (b) follows from Jensen’s inequality and fact that $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{\infty}$ is strictly concave in $t\in\mathcal{I}_{n}(r)$ (cf. Lemma 5). Thus, it holds that

[TABLE]

with $p=N_{r}^{-1}(\boldsymbol{v}_{n}:N_{r}(X\mid Y))$. Now, it follows from (45) and (47) that

[TABLE]

where (a) follows from the fact that

[TABLE]

Thus, Inequality (243) can be restated as

[TABLE]

with $p=H_{\alpha}^{-1}(\boldsymbol{v}_{n}:H_{\alpha}(X\mid Y))$ rather than $p=N_{r}^{-1}(\boldsymbol{v}_{n}:N_{r}(X\mid Y))$. This completes the proof of Theorem 5. ∎

Since the minimum average probability of error $P_{\mathrm{e}}(X\mid Y)$ satisfies

[TABLE]

Ineq. (207) of Theorem 5 can be seen as a generalization of Fano’s inequality from $H(X\mid Y)$ to $H_{\alpha}(X\mid Y)$ (see also [33]). Moreover, Theorem 5 is tighter than a generalized Fano’s inequality [22, Theorem 7], whose bounds another definition of conditional Rényi entropy proposed by Hayashi [19]. We defer to discuss this comparison until Section V-A.

On the other hand, the following theorem shows sharp bounds on $H_{\beta}(X\mid Y)$ with a fixed $H_{\alpha}(X\mid Y)$ when $X$ is a Bernoulli RV.

Theorem 6**.**

Let $X$ be a Bernoulli RV, i.e., $|\!\operatorname{supp}(P_{X})|\leq 2$, and let $Y$ be an arbitrary RV. Then, it holds that

[TABLE]

with $n=2$ and $p=H_{\alpha}^{-1}(\boldsymbol{v}_{n}:H_{\alpha}(X\mid Y))$, where $H_{\alpha}^{-1}(\boldsymbol{v}_{n}:\cdot)$ is defined in (47). In particular, if $\alpha=1$, then (255) also holds for every $0<\beta<1/2$.

Proof:

We prove this theorem in a similar manner to the proof of Theorem 5. If either $\alpha=\infty$ or $\beta=\infty$, then Theorem 6 comes from Theorem 5. If $\alpha=\beta$, then Theorem 6 is trivial171717If $\alpha=\beta$, then both inequalities of Theorem 6 hold with equality, because $H_{\alpha}(\boldsymbol{v}_{n}(H_{\alpha}^{-1}(\boldsymbol{v}_{n}:H_{\alpha}(X\mid Y))))=H_{\alpha}(X\mid Y)$.. Hence, we consider otherwise. Let $X$ be a Bernoulli RV in which $|\!\operatorname{supp}(P_{X})|=n=2$, let $Y$ be an arbitrary RV, and let $r,s\in[1/2,1)\cup(1,\infty)$ be distinct numbers. If $\gamma(r,s)<1$, then we have

[TABLE]

with $p=N_{r}^{-1}(\boldsymbol{v}_{2}:N_{r}(X\mid Y))$, where $\gamma(r,s)$ is defined in (60), Inequality (a) follows from (186) of Theorem 1, and Inequality (b) follows from Jensen’s inequality and fact that $t\mapsto\|\boldsymbol{v}_{2}(N_{r}^{-1}(\boldsymbol{v}_{2}:t))\|_{s}$ is strictly concave in $t\in\mathcal{I}_{2}(r)$ (cf. Lemma 5). Analogously, if $\gamma(r,s)>1$, then we also get

[TABLE]

with $p=N_{r}^{-1}(\boldsymbol{v}_{2}:N_{r}(X\mid Y))$. We now define

[TABLE]

for $\alpha\in(0,1)\cup(1,\infty)$ and $t>0$. Since

• •

it holds that $H_{\beta}(X\mid Y)=f_{\beta}(N_{\beta}(X\mid Y))$ for every $\beta\in(0,1)\cup(1,\infty)$,

• •

if $\beta\in(0,1)$, then $t\mapsto f_{\beta}(t)$ is a strictly increasing function of $t>0$,

• •

if $\beta\in(1,\infty)$, then $t\mapsto f_{\beta}(t)$ is a strictly decreasing function of $t>0$,

it follows from (260) and (261) that

[TABLE]

with $p=N_{\alpha}^{-1}(\boldsymbol{v}_{2}:N_{\alpha}(X\mid Y))$ for every distinct $\alpha,\beta\in[1/2,1)\cup(1,\infty)$, where (a) follows from (260) and (261). Combining (246), (263), and (264), we have Theorem 6 for distinct $\alpha,\beta\in[1/2,1)\cup(1,\infty)$.

Finally, if $\alpha=1$, then Theorem 6 can be proved by employing the concavity of Lemma 2 and the extremality of Theorem 3. In fact, it holds that for any $\beta\in(0,1)$,

[TABLE]

with $p=H^{-1}(\boldsymbol{v}_{2}:H(X\mid Y))$, where (a) follows from (204) of Theorem 3 with $\alpha=1$, and (b) follows from Jensen’s inequality and the fact that $\mu\mapsto\|\boldsymbol{v}_{2}(H^{-1}(\boldsymbol{v}_{2}:\mu))\|_{\beta}$ is strictly concave in $\mu\in[0,\ln 2]$ (cf. Lemma 2). Analogously, we also obtain

[TABLE]

with $p=H^{-1}(\boldsymbol{v}_{2}:H(X\mid Y))$ for every $\beta\in(1,\infty)$. This completes the proof of Theorem 6. ∎

In Theorems 5 and 6, we establish bounds on the conditional Rényi entropy $H_{\beta}(X\mid Y)$ by another Rényi entropy $H_{\beta}(\boldsymbol{v}_{n}(p))$ of an explicit distribution $\boldsymbol{v}_{n}(\cdot)$. Namely, these bounds are sharp, i.e., there is no tighter bound than them in these situations. Theorems 5 and 6 are proved by using the convexity/concavity of Lemmas 2 and 5. However, if $n\in\mathbb{N}_{\geq 3}$ and $r,s\in[1/2,1)\cup(1,\infty)$, then the convexity/concavity of Lemma 5 is not unique on $\mathcal{I}_{n}(r)$. Due to this reason, we cannot use same techniques as the proofs of Theorems 5 and 6 in the cases of $n\in\mathbb{N}_{\geq 3}$ and $r,s\in[1/2,1)\cup(1,\infty)$. In fact, we later show in Theorem 7 that $H_{\beta}(X\mid Y)$ cannot be always bounded by $H_{\beta}(\boldsymbol{v}_{n}(p))$ with a fixed $H_{\beta}(X\mid Y)$ in such situations. In [29, Theorem 4 and Corollary 1], we established sharp bounds on $H(X\mid Y)$ with a fixed $H_{\alpha}(X\mid Y)$ in which $\operatorname{supp}(P_{X})$ is finite, by defining a pair of RVs $(X^{\prime\prime},Y^{\prime\prime})$ [29, Definition 2] whose achieves their bounds. In this study, we also define a specific pair of RVs $(S,T)$ later in Definition 1, and establish sharp bounds on $H_{\beta}(X\mid Y)$ by using $(S,T)$ in Theorem 7. To accomplish this goal, we now give the following lemma.

Lemma 7**.**

For any fixed $n\in\mathbb{N}_{\geq 3}$ and distinct $r,s\in(0,1)\cup(1,\infty)$, the equation

[TABLE]

has a unique root $t=t^{\ast}(n;r,s)\in\mathcal{I}_{n}(r)\setminus\{1,n^{\theta(r)}\}$, where $\mathcal{I}_{n}(r)$ is defined in (43).

Proof:

Suppose that $r<s$ and $\gamma(r,s)<1$. Define

[TABLE]

We prove this lemma by showing the existence of the value $t^{\ast}(n;r,s)\in\mathcal{I}_{n}^{(1)}(r,s)$ satisfying $\chi(n,t,t^{\ast}(n;r,s);r,s)=0$, and proving its uniqueness. Letting $p=N_{r}^{-1}(\boldsymbol{v}_{n}:t)$, the chain rule of derivatives shows

[TABLE]

It follows from (45) and (95) that

• •

if $r\in(0,1)$, then $p\to 1^{-}$ as $t\to 1^{+}$,

• •

if $r\in(1,\infty)$, then $p\to 1^{-}$ as $t\to 1^{-}$,

• •

it holds that $z\to\infty$ as $p\to 1^{-}$;

and thus, if $r\in(0,1)$, then we obtain

[TABLE]

for every $n\in\mathbb{N}_{\geq 2}$ and $s\in(0,\infty)$ in which $r<s$, where (a) follows from the limiting value

[TABLE]

for every $0<r<s<\infty$. Analogously, we also get that if $r\in(1,\infty)$, then

[TABLE]

for every $n\in\mathbb{N}_{\geq 2}$ and $s\in(0,\infty)$ in which $r<s$. Therefore, we have the following:

• •

if $r\in(0,1)$, then

[TABLE]

• •

if $r\in(1,\infty)$, then

[TABLE]

for every $n\in\mathbb{N}_{\geq 2}$ and $s\in(0,\infty)$ in which $r<s$, where note that $\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:1))\|_{s}=1$ because $\|\boldsymbol{v}_{n}(1)\|_{r}=1$. Since

[TABLE]

for every $n\in\mathbb{N}_{\geq 2}$ and $r\in(0,\infty]$, it follows from (290) that

[TABLE]

for every $n\in\mathbb{N}_{\geq 2}$, $r\in(0,1)$, and $s\in(r,\infty)$. Similarly, it also follows from (291) that

[TABLE]

for every $n\in\mathbb{N}_{\geq 2}$ and $1<r<s<\infty$.

We now verify the sign of the derivative (284) as follows:

[TABLE]

for every $n\in\mathbb{N}_{\geq 2}$, $0<r<s<\infty$, and $t\in\mathcal{I}_{n}(r)$, which implies that for each $n\in\mathbb{N}_{\geq 2}$, the following holds:

• •

if $r<1<s$, then $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ is strictly decreasing for $t\in\mathcal{I}_{n}(r)$,

• •

if either $r<s<1$ or $1<r<s$, then $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ is strictly increasing for $t\in\mathcal{I}_{n}(r)$.

Recall from Lemma 5 that if $\gamma(r,s)<1$, then $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ is strictly convex in $t\in\mathcal{I}_{n}^{(2)}(r,s)$ for every fixed $1/2\leq r<s<\infty$, where note that $\mathcal{I}_{n}^{(2)}(r,s)\subset\mathcal{I}_{n}(r)$ (cf. (43) and (62)). Since

• •

the first term of the right-hand side of (275) is the slope of the secant line from the point $(n^{\theta(r)},n^{\theta(s)})$ to the point $(u,\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:u))\|_{s})$,

• •

the second term of the right-hand side of (275) is the slope of the tangent line of the curve $t\mapsto(t,\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s})$,

it follows from the monotonicity and convexity of $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ with respect to $t\in\mathcal{I}_{n}^{(2)}(r,s)$ that

[TABLE]

for every $n\in\mathbb{N}_{\geq 2}$, $1/2\leq r<s<\infty$ in which $\gamma(r,s)<1$, and $t\in\mathcal{I}_{n}^{(2)}(r,s)$, where $\tau(n;r,s)$ is the inflection point of $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ derived in Lemma 5. Combining (293), (294), and (298), and applying the intermediate value theorem for the function $u\mapsto\chi(n,t,u;r,s)$, it holds that for any $n\in\mathbb{N}_{\geq 2}$ and any $1/2\leq r<s<\infty$ in which $\gamma(r,s)<1$, there exists $t^{\ast}(n;r,s)\in\mathcal{I}_{n}^{(1)}(r,s)$ such that

[TABLE]

where $\mathcal{I}_{n}^{(1)}(r,s)$ is defined in (61). Finally, the concavity of $t\mapsto\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ with respect to $t\in\mathcal{I}_{n}^{(1)}(r,s)$ implies the uniqueness of the value $t^{\ast}(n;r,s)\in\mathcal{I}_{n}^{(1)}(r,s)$. Therefore, the assertion of Lemma 7 holds for $1/2\leq r<s<\infty$ in which $\gamma(r,s)<1$. Furthermore, the assertion of Lemma 7 for other situations can also be proved in a similar way to the above discussion. This completes the proof of Lemma 7. ∎

From the definition (45) of $N_{r}^{-1}(\boldsymbol{v}_{n}:\cdot)$, we see that

[TABLE]

thus, it follows from (80) and the chain of derivatives that (274) of Lemma 7 can be rewritten by

[TABLE]

with the change of variables (300), where note that the first-order derivatives appeared in (301) is already derived in (25). Lemma 7 also ensures that for every $n\in\mathbb{N}_{\geq 3}$ and distinct $r,s\in[1/2,1)\cup(1,\infty)$, Eq. (301) has a unique root with respect to $p=p^{\ast}(n;r,s)\in(1/n,1)$ under the relation (300). Therefore, solving the root $p^{\ast}(n;r,s)$ of (301), we can obtain the root of (274) as $t^{\ast}(n;r,s)=\|\boldsymbol{v}_{n}(p^{\ast}(n;r,s))\|_{r}$. In fact, the root $p^{\ast}(n;r,s)$ of (301) can be solved via numerical calculations. However, in general, the root $p^{\ast}(n;r,s)$ is also hard-to-express in closed-forms, as with (47) and (48). Fortunately, if either $r=1/2$ or $s=1/2$, then the root $p^{\ast}(n;r,s)$ of (301) can be written in a simple closed-form, as shown in the following.

Fact 2**.**

For any $n\in\mathbb{N}_{\geq 3}$ and $t\in(1/2,1)\cup(1,\infty)$,

[TABLE]

Fact 2 can be verified by directly substituting (302) into (301), as with the proof of [29, Fact 2]. In fact, Fact 2 yields the same value $p^{\ast}$ to181818Note that the definition of $\boldsymbol{v}_{n}(\cdot)$ used in [29] is slightly different to (14); however, these are essentially same. [29, Fact 2] as $t\to 1$. Note that $p^{\ast}(n;r,s)=p^{\ast}(n;s,r)$ holds; but $t^{\ast}(n;r,s)=t^{\ast}(n;s,r)$ does not hold in general. Employing the roots $t^{\ast}(n;r,s)$ and $p^{\ast}(n;r,s)$ of (274) and (301), respectively, we now define the pair of RVs $(S,T)$ as follows: For $n\in\mathbb{N}_{\geq 3}$ and distinct $r,s\in[1/2,1)\cup(1,\infty)$, let the real intervals $\mathcal{I}_{n}^{(\mathrm{a})}(r,s)$ and $\mathcal{I}_{n}^{(\mathrm{b})}(r,s)$ be defined by

[TABLE]

respectively, where $\theta(r)$ is defined in (42). Note that $\{\mathcal{I}_{n}^{(\mathrm{a})}(r,s),\mathcal{I}_{n}^{(\mathrm{b})}(r,s)\}$ forms a partition of the interval $\mathcal{I}_{n}(r)$ defined in (43). If $r$ and $s$ are clear from the context, for simplicity, we write (303) and (304) by $\mathcal{I}_{n}^{(\mathrm{a})}$ and $\mathcal{I}_{n}^{(\mathrm{b})}$, respectively. Using them, we give the definition of the pair of RVs $(S,T)$ as follows.

Definition 1**.**

For given distinct $r,s\in[1/2,1)\cup(1,\infty)$ and pair of RVs $(X,Y)\sim P_{X|Y}P_{Y}$ in which $|\!\operatorname{supp}(P_{X})|=n\in\mathbb{N}_{\geq 3}$, the pair of RVs $(S,T)\sim P_{S|T}P_{T}$ is defined as follows: The RV $S$ takes values from $\operatorname{supp}(P_{X})$; and the RV $T$ takes values from $\{0,1\}$, i.e., the latter is a Bernoulli RV. Let $\delta$ be chosen so that

[TABLE]

where $\theta(r)$ is defined in (42). Then, the marginal distribution $P_{T}$ is given by

[TABLE]

and the conditional distribution $P_{S|T}$ is given by

[TABLE]

with $p_{(\mathrm{a})}=p^{\ast}(n;r,s)$ and $p_{(\mathrm{b})}=N_{r}^{-1}(\boldsymbol{v}_{n}:N_{r}(X\mid Y))$, where $\mathcal{I}_{n}^{(\mathrm{a})}(r,s)$ and $\mathcal{I}_{n}^{(\mathrm{b})}(r,s)$ are defined in (303) and (304), respectively. If we want to specify the parameters $(r,s)$ for $(S,T)$, we write $(S_{(r,s)},T_{(r,s)})$.

After some algebra, for given pair of RVs $(X,Y)$ in which $|\!\operatorname{supp}(P_{X})|=n\in\mathbb{N}_{\geq 3}$ and distinct $r,s\in[1/2,1)\cup(1,\infty)$, the expectation of $\ell_{s}$-norm of $S$ given $T$ can be calculated by

[TABLE]

with

[TABLE]

where $\delta$ is given by (305), and $p^{\ast}(n;r,s)$ is the root of (301). Letting $(\alpha,\beta)=(r,s)$, for any distinct $\alpha,\beta\in[1/2,1)\cup(1,\infty)$, the conditional Rényi entropy of $S$ given $T$ can be calculated by

[TABLE]

with

[TABLE]

where $\mathcal{H}_{n}^{(\mathrm{a})}(\alpha,\beta)$ and $\mathcal{H}_{n}^{(\mathrm{b})}(\alpha,\beta)$ are two real intervals defined by

[TABLE]

respectively, and $\delta$ is given by (305) with $(r,s)=(\alpha,\beta)$. Note that $\{\mathcal{H}_{n}^{(\mathrm{a})}(\alpha,\beta),\mathcal{H}_{n}^{(\mathrm{b})}(\alpha,\beta)\}$ forms a partition of the interval $[0,\ln n]$. Namely, the quantity $H_{\beta}(S_{(\alpha,\beta)}\mid T_{(\alpha,\beta)})$ is determined by the following three arguments: (i) the number $|\!\operatorname{supp}(P_{X})|\geq 3$, (ii) the value $H_{\alpha}(X\mid Y)$, and (iii) distinct $\alpha,\beta\in[1/2,1)\cup(1,\infty)$. In fact, for any distinct $\alpha,\beta\in[1/2,1)\cup(1,\infty)$, we can verify the following:

• •

if $H_{\alpha}(X\mid Y)\in\mathcal{H}_{n}^{(\mathrm{a})}(\alpha,\beta)$, then

[TABLE]

• •

if $H_{\alpha}(X\mid Y)\in\mathcal{H}_{n}^{(\mathrm{b})}(\alpha,\beta)$, then

[TABLE]

Hence, the following theorem gives bounds on $H_{\beta}(X\mid Y)$ with a fixed $H_{\alpha}(X\mid Y)$, and the bounds are sharp because these are written by a specific pair of RVs $(S,T)$.

Theorem 7**.**

Let $X$ be an RV in which $3\leq|\!\operatorname{supp}(P_{X})|<\infty$, and let $Y$ be an arbitrary RV. For any distinct $\alpha,\beta\in[1/2,1)\cup(1,\infty)$, it holds that

[TABLE]

where the pair of RVs $(S,T)$ is defined in Definition 1.

Proof:

Let $(X,Y)$ be a pair of RVs in which $|\!\operatorname{supp}(P_{X})|=n$ for some $n\in\mathbb{N}_{\geq 3}$, and let $r,s\in[1/2,1)\cup(1,\infty)$ be distinct fixed numbers. Define

[TABLE]

where $\mathcal{I}_{n}^{(\mathrm{a})}(r,s)$ and $\mathcal{I}_{n}^{(\mathrm{b})}(r,s)$ are defined in (303) and (304), respectively, the value $\delta^{\prime}\in[0,1)$ is chosen so that

[TABLE]

and $t^{\ast}(n;r,s)$ is the root of (274) shown in Lemma 7. Note from (313) that $f_{\mathrm{ST}}(n,t;r,s)$ is defined to satisfy

[TABLE]

Then, we can verify the following statements:

• •

the function $t\mapsto f_{\mathrm{ST}}(n,t;r,s)$ is linear in $t\in\mathcal{I}_{n}^{(\mathrm{a})}(r,s)$,

• •

if $\gamma(r,s)>1$, then $t\mapsto f_{\mathrm{ST}}(n,t;r,s)$ is strictly convex in $t\in\mathcal{I}_{n}^{(\mathrm{b})}(r,s)$ (cf. Lemma 5),

• •

if $\gamma(r,s)<1$, then $t\mapsto f_{\mathrm{ST}}(n,t;r,s)$ is strictly concave in $t\in\mathcal{I}_{n}^{(\mathrm{b})}(r,s)$ (cf. Lemma 5),

where $\gamma(r,s)$ is defined in (60). Moreover, since $t^{\ast}(n;r,s)$ used in Definition 1 fulfills (274) of Lemma 7, the function $t\mapsto f_{\mathrm{ST}}(n,t;r,s)$ is differentiable $t=t^{\ast}(n;r,s)$. Therefore, the above convexity/concavity can be modified as follows:

• •

if $\gamma(r,s)>1$, then $t\mapsto f_{\mathrm{ST}}(n,t;r,s)$ is convex in $t\in\mathcal{I}_{n}(r)$,

• •

if $\gamma(r,s)<1$, then $t\mapsto f_{\mathrm{ST}}(n,t;r,s)$ is concave in $t\in\mathcal{I}_{n}(r)$,

where $\mathcal{I}_{n}(r)$ is defined in (43).

We now consider inequalities between $\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s}$ and $f_{\mathrm{ST}}(n,t;r,s)$. By definition (333), it is clear that

[TABLE]

for every $t\in\mathcal{I}_{n}^{(\mathrm{b})}(r,s)$. On the other hand, the proof of Lemma 7 shows that

• •

if $\gamma(r,s)>1$, the curve $t\mapsto(t,\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s})$ is bounded from below by the secant line from the point $(n^{\theta(r)},n^{\theta(s)})$ to the point $(t^{\ast}(n;r,s),\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t^{\ast}(n;r,s)))\|_{s})$,

• •

if $\gamma(r,s)<1$, the curve $t\mapsto(t,\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t))\|_{s})$ is bounded from above by the secant line from the point $(n^{\theta(r)},n^{\theta(s)})$ to the point $(t^{\ast}(n;r,s),\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t^{\ast}(n;r,s)))\|_{s})$.

Since the secant line from the point $(n^{\theta(r)},n^{\theta(s)})$ to the point $(t^{\ast}(n;r,s),\|\boldsymbol{v}_{n}(N_{r}^{-1}(\boldsymbol{v}_{n}:t^{\ast}(n;r,s)))\|_{s})$ can be denoted by $t\mapsto(t,f_{\mathrm{ST}}(n,t;r,s))$ for $t\in\mathcal{I}_{n}^{(\mathrm{a})}(r,s)$, it holds that

[TABLE]

for every $t\in\mathcal{I}_{n}^{(\mathrm{a})}(r,s)$. Combining (336), (337), and (338), we get

[TABLE]

for every $t\in\mathcal{I}_{n}(r)$, because $\mathcal{I}_{n}(r)=\mathcal{I}_{n}^{(\mathrm{a})}(r,s)\cup\mathcal{I}_{n}^{(\mathrm{b})}(r,s)$.

According to the above discussion, if $\gamma(r,s)>1$, we have

[TABLE]

where (a) follows from (185) of Theorem 1, and (b) follows from the convexity of $t\mapsto f_{\mathrm{ST}}(n,t;r,s)$ for $t\in\mathcal{I}_{n}(r)$. Similarly, if $\gamma(r,s)<1$, we also have

[TABLE]

Finally, we define

[TABLE]

for $\alpha\in(0,1)\cup(1,\infty)$ and $t>0$. Since

• •

it holds that $H_{\beta}(X\mid Y)=f_{\beta}(N_{\beta}(X\mid Y))$ for every $\beta\in(0,1)\cup(1,\infty)$,

• •

if $\beta\in(0,1)$, then $t\mapsto f_{\beta}(t)$ is a strictly increasing function of $t>0$,

• •

if $\beta\in(1,\infty)$, then $t\mapsto f_{\beta}(t)$ is a strictly decreasing function of $t>0$,

it follows from (346) and (347) that

[TABLE]

for every $\alpha,\beta\in[1/2,1)\cup(1,\infty)$. This completes the proof of Theorem 7. ∎

Note that if either $\alpha=1$ or $\beta=1$, then sharp bounds in a similar situation to Theorem 7 were already derived in [29, Theorem 2 and Corollary 1].

In this subsection, we derive sharp bounds on $H_{\beta}(X\mid Y)$ with a fixed $H_{\alpha}(X\mid Y)$, $\alpha\neq\beta$, by employing the extremality of the distribution $\boldsymbol{v}_{n}(\cdot)$ shown in Section III. In the next subsection, we further derive sharp bounds on $H_{\beta}(X\mid Y)$ with a fixed $H_{\alpha}(X\mid Y)$, $\alpha\neq\beta$, by employing the extremality of another distribution $\boldsymbol{w}(\cdot)$ shown in Section III.

IV-B Bounds Established from Distribution $\boldsymbol{w}(\cdot)$

In this subsection, by using extremality of the distribution $\boldsymbol{w}(\cdot)$ introduced in Section III, we derive sharp bounds on $H_{\beta}(X\mid Y)$ with a fixed $H_{\alpha}(X\mid Y)$. Unlike the bounds established in Section IV-A, as with Theorem 4, sharp bounds established in this subsection can be considered for every distributions with possibly countably infinite support. In a similar way to consider a specific pair of RVs $(S,T)$ of Definition 1, we now define another specific pair of RVs $(U,V)$ in Definition 2, whose achieves the bounds of Theorem 8 as shown later.

Definition 2**.**

For given $r\in(0,1)\cup(1,\infty]$ and pair of RVs $(X,Y)\sim P_{X|Y}P_{Y}$, the pair of RVs $(U,V)\sim P_{U|V}P_{V}$ is defined as follows: The RV $U$ takes values from $\{0,1,2,\dots\}$; and the RV $V$ takes values from $\{0,1\}$, i.e., the latter is a Bernoulli RV. Let $m\in\mathbb{N}$ and $\lambda\in[0,1]$ be chosen so that

[TABLE]

respectively, where $\theta(r)$ is defined in (42). Then, the marginal distribution $P_{V}$ is given by

[TABLE]

and the conditional distribution $P_{U|V}$ is given by

[TABLE]

If we want to specify the parameter $r$ for $(U,V)$, we write $(U_{(r)},V_{(r)})$.

After some algebra, we see that

[TABLE]

for every pair of RVs $(X,Y)$, $r\in(0,1)\cup(1,\infty]$, and $s\in(0,\infty]$, where $m\in\mathbb{N}$ and $\lambda\in[0,1]$ are given by (351) and (352), respectively. Similarly, the conditional Rényi entropy of $U$ given $V$ can be calculated as

[TABLE]

for every pair of RVs $(X,Y)$, $\alpha\in(0,1)\cup(1,\infty]$, and $\beta\in(0,1)\cup(1,\infty]$, where $m\in\mathbb{N}$ and $\lambda\in[0,1]$ are given by (351) and (352), respectively, with $r=\alpha$. Analogously, it follows that

[TABLE]

for every $\alpha\in(0,1)\cup(1,\infty]$. Thus, the quantity $H_{\beta}(U_{(\alpha)}\mid V_{(\alpha)})$ is determined by the following two arguments: (i) the value $H_{\alpha}(X\mid Y)$, and (ii) two orders $\alpha,\beta$. In fact, as with (328) and (330), it also holds that

[TABLE]

Fortunately, unlike $H_{\beta}(S_{(\alpha,\beta)}\mid T_{(\alpha,\beta)})$, the quantity $H_{\beta}(U_{(\alpha)}\mid V_{(\alpha)})$ can be expressed in closed-forms for every $\alpha\in(0,1)\cup(1,\infty]$ and $\beta\in(0,\infty]$. Employing the pair of RVs $(U,V)$, the sharp bounds on $H_{\beta}(X\mid Y)$ with a fixed $H_{\alpha}(X\mid Y)$ can be established for $\alpha\neq\beta$, as shown in the following theorem.

Theorem 8**.**

Let $X$ be a discrete RV in which $\operatorname{supp}(P_{X})$ is possibly countably infinite, and let $Y$ be an arbitrary RV. For any $\alpha\in(0,1)\cup(1,\infty]$ and any $\beta\in(0,\infty]$, it holds that

[TABLE]

where the pair of RVs $(U,V)$ is defined in Definition 2.

Proof:

Suppose that $0<r<s\leq\infty$. For given $m\in\mathbb{N}$, $r\in(0,\infty)$ and $t\in\mathcal{J}_{m}(r)$, let $\lambda\in[0,1]$ be chosen so that

[TABLE]

It follows from Lemma 6 that if $\gamma(r,s)<1$, then

[TABLE]

for every $m\in\mathbb{N}$, $t\in\mathcal{J}_{m}(r)$, and $0<r<s\leq\infty$, where (a) follows by the concavity of Lemma 6 and the definition of concave functions, and (b) follows from the fact that

[TABLE]

Similarly, if $\gamma(r,s)>1$, it also follows from Lemma 6 that

[TABLE]

for every $m\in\mathbb{N}$, $t\in\mathcal{J}_{m}(r)$, and $0<r<s\leq\infty$. Note from (359) and (374) that the function $\phi(m,t;r,s)$ fulfills

[TABLE]

for given pair of RVs $(X,Y)$ and $r,s\in(0,1)\cup(1,\infty]$. We now verify the monotonicity of the derivative

[TABLE]

with respect to $m\in\mathbb{N}$. Since $1-\theta(r)<1-\theta(s)$ whenever $r<s$ and

[TABLE]

we get that for each fixed $0<r<s\leq\infty$, the function

[TABLE]

is strictly decreasing for $m\in\mathbb{N}$. Moreover, since $\theta(s)-\theta(r)<0$ whenever $r<s$, it also follows that for each fixed $0<r<s\leq\infty$, the function $m\mapsto m^{\theta(s)-\theta(r)}$ is strictly decreasing for $m\in\mathbb{N}$. Therefore, we have

[TABLE]

for every $m\in\mathbb{N}$ and $0<r<s\leq\infty$ with $r,s\neq 1$, where (a) follows from the hypothesis: $r<s$. This implies the strict monotonicity of the derivative

[TABLE]

with respect to $m\in\mathbb{N}$. In addition, it follows from (375) that

[TABLE]

i.e, it holds that

[TABLE]

for every $m\in\mathbb{N}$ and $r,s\in(0,1)\cup(1,\infty]$. Since $t\mapsto\phi(m,t;r,s)$ is linear in $t$ (cf. (376)), combining (388) and (394), we have that for any fixed $0<r<s\leq\infty$,

• •

if $\gamma(r,s)>1$, then $t\mapsto\min\{\phi(m,t;r,s)\mid m\in\mathbb{N}\}$ is a piecewise linear function of $t\in\mathcal{J}(r)$, whose its slope never increases as $t$ increases, i.e., it is concave in $t\in\mathcal{J}(r)$,

• •

if $\gamma(r,s)<1$, then $t\mapsto\min\{\phi(m,t;r,s)\mid m\in\mathbb{N}\}$ is a piecewise linear function of $t\in\mathcal{J}(r)$, whose its slope never decreases as $t$ increases, i.e., it is convex in $t\in\mathcal{J}(r)$.

Moreover, it also follows from (388) and (394) that

[TABLE]

for every $m^{\prime}\in\mathbb{N}$ and $t\in\mathcal{J}_{m^{\prime}}(r)$; thus, we get from (370) that

[TABLE]

for every $t\in\mathcal{J}(r)$. Combining (377), (379), (397), and (397), we obtain

[TABLE]

for every $t\in\mathcal{J}(r)$ and $0<r<s\leq\infty$. Therefore, it $\gamma(r,s)<1$, we obtain

[TABLE]

for every pair of RVs $(X,Y)$ and $0<r<s\leq\infty$, where (a) follows by Theorem 2, and (b) follows by the linearity of $t\mapsto\phi(m,t;r,s)$ (cf. (376)). Analogously, it can also be verified that if $\gamma(r,s)>1$, then

[TABLE]

for every pair of RVs $(X,Y)$ and $0<r<s\leq\infty$.

Finally, we define

[TABLE]

for $\alpha\in(0,\infty]$ and $t>0$. Since

• •

it holds that $H_{\alpha}(X\mid Y)=f_{\alpha}(N_{\alpha}(X\mid Y))$ for every $0<\alpha\leq\infty$,

• •

if $\gamma(\alpha,\beta)>1$, then $t\mapsto f_{\beta}(t)$ is a strictly decreasing function of $t>0$,

• •

if $\gamma(\alpha,\beta)<1$, then $t\mapsto f_{\beta}(t)$ is a strictly increasing function of $t>0$

for every $0<\alpha<\beta\leq\infty$, it follows from (408) and (409) that

[TABLE]

for every pair of RVs $(X,Y)$ and $0<\alpha<\beta\leq\infty$. In a similar way to the above discussions, we can also prove that

[TABLE]

for every pair of RVs $(X,Y)$ and $0<\beta<\alpha\leq\infty$. This completes the proof of Theorem 8. ∎

Note that if $\alpha=1$, then sharp bounds in a similar situation to Theorem 8 were already derived in [29, Theorem 2 and Corollary 1]. We further mention that Theorem 8 has no constraint in the size of support $|\!\operatorname{supp}(P_{X})|$, i.e., the RV $X$ may take values from a countably infinite alphabet.

We now compare Theorem 8 to the inequality

[TABLE]

proved by Fehr and Berens [13, Proposition 5], which shows that $\alpha\mapsto H_{\alpha}(X\mid Y)$ is decreasing for its order $\alpha\in[0,\infty]$. It follows from (366) and (413) that

[TABLE]

which implies that (368) of Theorem 8 is tighter than (413).

V Applications

In Section IV, we established sharp bounds on $H_{\beta}(X\mid Y)$ with a fixed $H_{\alpha}(X\mid Y)$ for distinct orders $\alpha\neq\beta$. In this section, we introduce applications of these sharp bounds to other information measures. If an information measure is a strictly monotone function of $H_{\alpha}(X\mid Y)$, then our results can be applicable to it.

As an example, we can apply Theorems 5 and 8 to the minimum average probability of error $P_{\mathrm{e}}(X\mid Y)=\exp(-H_{\infty}(X\mid Y))$ defined in (13); and then, we can obtain a generalization of Fano’s inequality from $H(X\mid Y)$ to $H_{\alpha}(X\mid Y)$, as with [33]. We organize this discussion in the next subsection.

V-A Generalized Fano’s Inequality: Interplay Between Conditional Rényi Entropy and Average Probability of Error

In this subsection, we examine interplay between the conditional Rényi entropy and the probability of error, as a generalization of Fano’s inequality. We first show an unconditional version of it in the following corollary.

Corollary 1** (Unconditional version of Fano’s inequality for the Rényi entropy, see also [33, Corollary 3 and Theorem 2]).**

Let $X$ be a discrete RV taking values from a countable alphabet $\mathcal{X}$. Then, it holds that

[TABLE]

for every $\alpha\in(0,1)\cup(1,\infty)$ and $\varepsilon\in[0,P_{\mathrm{e}}(X)]$, where the minimum average probability of error $P_{\mathrm{e}}(X)$ for guessing $X$ is defined by

[TABLE]

In addition, if $\operatorname{supp}(P_{X})$ is finite, i.e., $|\!\operatorname{supp}(P_{X})|\leq n$ for some $n\in\mathbb{N}$, then

[TABLE]

for every $\alpha\in(0,1)\cup(1,\infty)$ and $\varepsilon\in[P_{\mathrm{e}}(X),1]$.

Proof:

Let $X$ be a discrete RV. It follows from Theorem 4 that

[TABLE]

for every $\alpha\in(0,1)\cup(1,\infty)$ and $\varepsilon\in[0,P_{\mathrm{e}}(X)]$, where (a) follows from the fact that $p\mapsto H_{\alpha}(\boldsymbol{w}(p))$ is strictly decreasing191919This monotonicity follows from Lemma 1 and the monotonicity of $t\mapsto(\alpha/(1-\alpha))\ln t$. for $p\in(0,1]$.

On the other hand, we suppose that $\operatorname{supp}(P_{X})$ is finite, i.e., $|\!\operatorname{supp}(P_{X})|=k\in\mathbb{N}$. It follows from Theorem 3 that

[TABLE]

for every $\alpha\in(0,1)\cup(1,\infty)$, $n\geq|\!\operatorname{supp}(P_{X})|$, and $\varepsilon\in[P_{\mathrm{e}}(X),(n-1)/n]$, where (a) follows from the fact that the right-hand side of (427) is strictly increasing for $n>1$, and (b) also follows from the fact that the right-hand side of (428) is strictly increasing for $\varepsilon\in[0,1]$. Finally, since $0\leq P_{\mathrm{e}}(X\mid Y)\leq(k-1)/k\leq(n-1)/n$, Inequality (428) can be rewritten by

[TABLE]

for every $\alpha\in(0,1)\cup(1,\infty)$, $n\geq|\!\operatorname{supp}(P_{X})|$, and $\varepsilon\in[P_{\mathrm{e}}(X),1]$. This completes the proof of Corollary 1. ∎

In the following corollary, we give sharp upper and lower bounds on $H_{\alpha}(X\mid Y)$ with a fixed probability of error, i.e., the following corollary shows generalizations of Fano’s inequality and the reverse of Fano’s inequality.

Corollary 2** (Conditional version of Fano’s inequality for the Rényi entropy, see also [33, Theorems 3 and 11]).**

Let $X$ be a discrete RV, and let $Y$ be an arbitrary RV. Then, it holds that

[TABLE]

for every $\alpha\in(0,1)\cup(1,\infty)$ and $\varepsilon\in[0,P_{\mathrm{e}}(X\mid Y)]$, where the minimum average probability of error $P_{\mathrm{e}}(X\mid Y)$ is defined in (13). In addition, if $\operatorname{supp}(P_{X})$ is finite, i.e., $|\!\operatorname{supp}(P_{X})|\leq n$ for some $n\in\mathbb{N}$, then

[TABLE]

for every $\alpha\in(0,1)\cup(1,\infty)$ and $\varepsilon\in[P_{\mathrm{e}}(X\mid Y),1]$.

Proof:

Let $X$ be a discrete RV, and let $Y$ be an arbitrary RV. It follows from Theorem 8 that

[TABLE]

for every $\alpha\in(0,1)\cup(1,\infty)$ and $\varepsilon\in[0,P_{\mathrm{e}}(X\mid Y)]$, where (a) follows from the fact that the right-hand side of (440) is strictly increasing for $\varepsilon\in[0,1)$. Note that this monotonicity can be verified as with the proof of Lemma 1.

On the other hand, we suppose that $\operatorname{supp}(P_{X})$ is finite, i.e., $|\!\operatorname{supp}(P_{X})|=k\in\mathbb{N}$. It follows from Theorem 5 that

[TABLE]

for every $\alpha\in(0,1)\cup(1,\infty)$, $n\geq|\!\operatorname{supp}(P_{X})|$, and $\varepsilon\in[P_{\mathrm{e}}(X\mid Y),(n-1)/n]$, where (a) follows from the fact that the right-hand side of (445) is strictly increasing for $n>1$, and (b) also follows from the fact that the right-hand side of (446) is strictly increasing for $\varepsilon\in[0,1]$. Finally, since $0\leq P_{\mathrm{e}}(X\mid Y)\leq(k-1)/k\leq(n-1)/n$, Inequality (446) can be rewritten by

[TABLE]

for every $\alpha\in(0,1)\cup(1,\infty)$, $n\geq|\!\operatorname{supp}(P_{X})|$, and $\varepsilon\in[P_{\mathrm{e}}(X\mid Y),1]$. This completes the proof of Corollary 2. ∎

In Fig. 3, we illustrate feasible regions of pairs $(P_{\mathrm{e}}(X\mid Y),H_{\alpha}(X\mid Y))$ established by the upper and lower bounds of Corollary 2. The well-known bounds $0\leq H_{\alpha}(X\mid Y)\leq\ln|\!\operatorname{supp}(P_{X})|$, e.g., [13, Proposition 3], immediately follow by Corollary 2. In addition, Corollary 2 also implies that

[TABLE]

for every $\alpha\in(0,1)\cup(1,\infty)$.

We now consider RVs $X$ and $Y$ taking values from same finite alphabet $\mathcal{X}$. Since $P_{\mathrm{e}}(X\mid Y)\leq\Pr(X\neq Y)$, note that (431) also holds with $\varepsilon=\Pr(X\neq Y)$. If $\varepsilon=\Pr(X\neq Y)\leq 1-1/|\mathcal{X}|$, then (431) approaches to

[TABLE]

as $\alpha\to 1$, where $h_{2}:t\mapsto-t\ln t-(1-t)\ln(1-t)$ denotes the binary entropy function. Thus, Ineq. (431) is a part of generalized Fano’s inequality in terms of Arimoto’s conditional Rényi entropy. Unlike (431), since $P_{\mathrm{e}}(X\mid Y)\leq\Pr(X\neq Y)$, note that (430) does not hold with $\varepsilon=\Pr(X\neq Y)$ in general. In fact, the reverse of Fano’s inequality (cf. [12, Theorem 1], [23, Eq. (15)], [36, Eq. (6)]) is a sharp lower bound on the conditional Shannon entropy $H(X\mid Y)$ with not fixed $\Pr(X\neq Y)$ but fixed $\varepsilon=P_{\mathrm{e}}(X\mid Y)$ as

[TABLE]

Since (430) approaches to (451) with $\varepsilon=P_{\mathrm{e}}(X\mid Y)$ as $\alpha\to 1$, Inequality (430) can be seen as a generalized reverse of Fano’s inequality in terms of Arimoto’s conditional Rényi entropy. Indeed, it can be verified that the right-hand side of (451) is same as the right-hand side of (362).

We now compare Corollary 2 with another generalized Fano’s inequality, which is an upper bound on another definition of conditional Rényi entropy

[TABLE]

proposed by Hayashi [19]. Iwamoto and Shikata [22] investigated many information theoretic properties of $H_{\alpha}^{\mathrm{H}}(X\mid Y)$. Then, they derived a different type of Fano’s inequality, as shown in the following theorem.

Theorem 9** ([22, Theorem 7]).**

Let $X$ and $Y$ be RVs taking values from same finite alphabet $\mathcal{X}$. Define

[TABLE]

Then, it holds that

[TABLE]

for every $\alpha\in(0,1)\cup(1,\infty)$ whenever202020Note that $g_{2}(\alpha,0,n)$ is undefined if $\alpha\in(0,1)$ and $g_{2}(\alpha,1,n)$ is also undefined if $\alpha\geq 2$. In [22, Theorem 7], the limiting value was considered as $\Pr(X\neq Y)\to 0$. $0<\Pr(X\neq Y)<1$.

Since

[TABLE]

(cf. [22, Theorem 1]), Inequality (431) of Corollary 2 can be relaxed by replacing $H_{\alpha}(X\mid Y)$ by $H_{\alpha}^{\mathrm{H}}(X\mid Y)$. Moreover, since the right-hand side of (431) is equal to $g_{1}(\alpha,\varepsilon,n)$ for $0\leq\varepsilon\leq(n-1)/n$, Inequality (431) of Corollary 2 can also be relaxed by replacing the right-hand side of (431) by the right-hand side of (455) for $0\leq\varepsilon\leq(n-1)/n$. Thus, Inequality (431) of Corollary 2 is tighter than (455) of Theorem 9 when $0\leq\Pr(X\neq Y)\leq(|\mathcal{X}|-1)/|\mathcal{X}|$.

Finally, we give sharp bounds on $P_{\mathrm{e}}(X\mid Y)$ with a fixed $H_{\alpha}(X\mid Y)$ by using the results of Section IV, as shown in the following corollary.

Corollary 3** (see also [33, Theorems 5 and 12]).**

Let $X$ be a discrete RV, and let $Y$ be an arbitrary RV. Then, it holds that

[TABLE]

for every $\alpha\in(0,1)\cup(1,\infty)$. In addition, if $\operatorname{supp}(P_{X})$ is finite, i.e., $|\!\operatorname{supp}(P_{X})|=n$ for some $n\in\mathbb{N}$, then

[TABLE]

for every $\alpha\in(0,\infty)$, where $H_{\alpha}^{-1}(\boldsymbol{v}_{n}:\cdot)$ is defined in (47). In particular, if either $\alpha=1/2$ or $\alpha=2$, then the following closed-form bounds hold:

[TABLE]

with $n=|\!\operatorname{supp}(P_{X})|$.

Proof:

Let $X$ be a discrete RV, and let $Y$ be an arbitrary RV. It follows from Theorem 8 that

[TABLE]

for every $\alpha\in(0,1)\cup(1,\infty)$. Along with (467), the equation

[TABLE]

yields (457).

On the other hand, we suppose that $|\!\operatorname{supp}(P_{X})|=n$ for some $n\in\mathbb{N}$. It follows from Theorem 5 that

[TABLE]

for every $\alpha\in(0,\infty)$. Combining (468) and (470), we have (458). Finally, Inequalities (459) and (460) can be obtained by substituting (458) into the closed-forms of Fact 1. This completes the proof of Corollary 3. ∎

In Fig. 4, we illustrate feasible regions of pairs $(H_{\alpha}(X\mid Y),P_{\mathrm{e}}(X\mid Y))$ established by the upper and lower bounds on Corollary 3. In this subsection, we examined interplay between $H_{\alpha}(X\mid Y)$ and $P_{\mathrm{e}}(X\mid Y)$ as a generalization of Fano’s inequality. In the next subsection, we further consider applications of the results of this study to other information measures.

V-B Other Related Information Measures

We now consider the Bhattacharrya parameter [25, Definition 17] of $X$ given $Y$, defined by

[TABLE]

where $X$ is an RV taking values from a finite alphabet $\mathcal{X}$, and $Y$ is an arbitrary RV. This quantity $Z(X\mid Y)$ is useful to analyze rate of polarization for $|\mathcal{X}|$-ary polar codes [25, Section VII-B], [31, Section 4.1.2]. After some algebra, we have

[TABLE]

therefore, our results can be applicable to $Z(X\mid Y)$. Fortunately, if $\alpha=1/2$, i.e., in the case of (478), our results can be expressed in closed-forms by Facts 1 and 2. In the following corollary, we show sharp upper and lower bounds on $Z(X\mid Y)$ with fixed $P_{\mathrm{e}}(X\mid Y)$ and $|\mathcal{X}|$.

Corollary 4** (Sharp bounds on Bhattacharrya parameter with a fixed average probability of error).**

Let $X$ be an RV taking values from a finite alphabet $\mathcal{X}$, and let $Y$ be an arbitrary RV. Then, it holds that

[TABLE]

for every $0\leq\varepsilon_{1}\leq P_{\mathrm{e}}(X\mid Y)\leq\varepsilon_{2}\leq(|\mathcal{X}|-1)/|\mathcal{X}|$.

Proof:

It follows from (430) of Corollary 2 and (478) that

[TABLE]

for every $\varepsilon\in[0,P_{\mathrm{e}}(X\mid Y)]$. In addition, it also follows from (431) of Corollary 2 and (478) that

[TABLE]

for every $\varepsilon\in[P_{\mathrm{e}}(X\mid Y),(|\mathcal{X}|-1)/|\mathcal{X}|]$. This completes the proof of Corollary 4. ∎

In Fig. 5, we illustrate a feasible region of pairs $(P_{\mathrm{e}}(X\mid Y),Z(X\mid Y))$ established by the upper and lower bounds of Corollary 4. In a similar way to the proof of Corollary 4, we can also derive sharp upper and lower bounds on $P_{\mathrm{e}}(X\mid Y)$ with fixed $Z(X\mid Y)$ and $|\mathcal{X}|$, as shown in the following corollary.

Corollary 5** (Sharp bounds on minimum average probability of error with a fixed Bhattacharrya parameter, see also [25, Lemma 22]).**

Let $X$ be an RV taking values from a finite alphabet $\mathcal{X}$, and let $Y$ be an arbitrary RV. Then, it holds that

[TABLE]

Proof:

Let $X$ be an RV taking values from a finite alphabet $\mathcal{X}$, and let $Y$ be an arbitrary RV. For simplicity, let $Z=Z(X\mid Y)$, let $\varepsilon=P_{\mathrm{e}}(X\mid Y)$, and let $n=|\mathcal{X}|$. Substituting $\alpha=1/2$ and $\exp(H_{1/2}(X\mid Y))=1+(|\mathcal{X}|-1)Z(X\mid Y)$ (see (478)) into (457), we have

[TABLE]

which is the upper bound of (487).

On the other hand, consider the right-hand inequality of (479). We readily see that

[TABLE]

By the quadratic formula, we have

[TABLE]

and the left-hand inequality is indeed the lower bound of (487). This completes the proof of Corollary 5. ∎

Corollary 5 is equivalent to [25, Lemma 22]; and thus, this study gives an alternative proof of it. Note that Corollary 5 also shows same feasible regions as Fig. 5.

So far, in this section, we presented applications of the sharp bounds on $H_{\beta}(X\mid Y)$ with a fixed $H_{\alpha}(X\mid Y)$ in the case of either $\alpha=\infty$ or $\beta=\infty$. However, the results of this study enable us to consider the sharp bounds on $H_{\beta}(X\mid Y)$ with a fixed $H_{\alpha}(X\mid Y)$ in the case of that both $\alpha$ and $\beta$ are finite orders. As an example, the following corollary shows sharp bounds on $H_{2}(X\mid Y)$ with a fixed $H_{1/2}(X\mid Y)$.

Corollary 6**.**

Let $X$ be a discrete RV, and let $Y$ be an arbitrary RV. Then, it holds that

[TABLE]

In addition, if $|\!\operatorname{supp}(P_{X})|=n$ for some $n\in\mathbb{N}$, then the following lower bounds hold:

• •

if $0\leq H_{1/2}(X\mid Y)\leq 2\ln(1+\sqrt{n-1})-\ln 2$, then

[TABLE]

where $H_{1/2}^{-1}(\boldsymbol{v}_{n}:\cdot)$ is already shown in Fact 1, and

• •

if $2\ln(1+\sqrt{n-1})-\ln 2<H_{1/2}(X\mid Y)\leq\ln n$, then

[TABLE]

Proof:

Let $X$ be a discrete RV, and let $Y$ be an arbitrary RV. It follows from Theorem 8 that

[TABLE]

Substituting (351) into (504), we obtain (495).

On the other hand, we suppose that $\operatorname{supp}(P_{X})$ is finite. Let $n=|\!\operatorname{supp}(P_{X})|$. By Fact 2, the following identities hold:

[TABLE]

Hence, it follows from (322) and (323) that

[TABLE]

respectively, where note that if $n=2$, then

[TABLE]

If $n\geq 3$ and $H_{1/2}(X\mid Y)\in\mathcal{H}_{n}^{(\mathrm{a})}(1/2,2)$, then it follows from Theorem 7 that

[TABLE]

thus, we have (497). Moreover, if $n\geq 3$ and $H_{1/2}(X\mid Y)\in\mathcal{H}_{n}^{(\mathrm{b})}(1/2,2)$, then it also follows from Theorem 7 that

[TABLE]

Finally, if $n=2$, then Theorem 6 also yields (531). Hence, we have (496). This completes the proof of Corollary 6. ∎

Analogously, we can also establish closed-form sharp bounds on $H_{1/2}(X\mid Y)$ with a fixed $H_{2}(X\mid Y)$ in a similar way to the proof of Corollary 6.

Furthermore, since $H_{\alpha}(X\mid Y)$ is closely related to Gallager’s reliability function $E_{0}$ [14] and $\alpha$-mutual information [35, 21] (cf. [2, 41]), we can also establish sharp bounds on them in some situation, as with [29, Theorem 5].

Appendix A Proof of Lemma 4

Proof:

The identity $g(n,z;r,s)=-g(n,z;s,r)$ is trivial from the definition (55). Hence, suppose throughout this proof that $0<r<s<\infty$, and we only consider the function $g(n,z;r,s)$. We first prove the first assertion of Lemma 4 for $z\in(0,1)$. It is clear that if $z\in(0,1)$, then $r\mapsto z^{r}$ is strictly decreasing for $r\in\mathbb{R}$. In addition, for each fixed $z\in(0,1)\cup(1,\infty)$, the function $r\mapsto\ln_{r}z$ is strictly decreasing for $r\in\mathbb{R}$ (cf. [29, Lemma 1]). Therefore, we obtain

[TABLE]

for every $n\in\mathbb{N}$, $z\in(0,1)$, and $0<r<s<\infty$, which is the first assertion of Lemma 4 for $z\in(0,1)$.

We next consider the second and third assertions of Lemma 4 for $z\in(1,\infty)$. Consider two functions

[TABLE]

satisfying

[TABLE]

Direct calculations show the following derivatives:

[TABLE]

We readily see that

[TABLE]

for every $n\in\mathbb{N}_{\geq 2}$ and $0<r<s<\infty$, where (a) follows from the limiting value

[TABLE]

(b) follows from the fact that

[TABLE]

and (c) follows from the limiting value

[TABLE]

for every $0<r<s<\infty$.

In particular, if $n=2$, then we get

[TABLE]

for every $z\in(1,\infty)$ and $1/2\leq r<s<\infty$, where (a) and (b) follow from the facts that

• •

for each fixed $z\in(1,\infty)$, the function $t\mapsto z^{t}-z^{1-t}$ is strictly increasing for $t\in\mathbb{R}$,

• •

$r-s<0$ whenever $r<s$,

• •

$(z^{t}-z^{1-t})|_{t=1/2}=\sqrt{z}-\sqrt{z}=0$ for every $z\in[0,\infty)$.

It follows from (567), (573), and (582) that for each fixed $1/2\leq r<s<\infty$, the function $z\mapsto g(2,z;r,s)$ is strictly decreasing for $z\in[1,\infty)$; and therefore, we observe from (556) that

[TABLE]

for every $z\in(1,\infty)$ and $1/2\leq r<s<\infty$, which is the second assertion of Lemma 4.

We further consider the third assertion of Lemma 4, i.e., the case: $n\in\mathbb{N}_{\geq 3}$. It follows from (567) and (573) that for any $n\in\mathbb{N}_{\geq 3}$ and $0<r<s<\infty$, there exists $\eta(n;r,s)\in(1,\infty)$ such that

[TABLE]

which implies that $z\mapsto g(n,z;r,s)$ is strictly increasing for $z\in[1,\eta(z;r,s)]$. By this strict monotonicity, it follows from (556) that

[TABLE]

for every $n\in\mathbb{N}_{\geq 3}$ and $0<r<s<\infty$. From (562) and (585), the intermediate value theorem shows that for any $n\in\mathbb{N}_{\geq 3}$ and $0<r<s<\infty$, there exists $\zeta(n;r,s)\in(\eta_{1}(n;r,s),\infty)$ such that

[TABLE]

It is clear from (586) that

[TABLE]

for every $n\in\mathbb{N}_{\geq 3}$ and $0<r<s<\infty$. Since

[TABLE]

we get from (586) and (587) that

[TABLE]

for every $n\in\mathbb{N}_{\geq 3}$ and $0<r<s<\infty$. Since (582) shows

[TABLE]

for every $z\in(1,\infty)$ and $1/2\leq r<s<\infty$, it follows from (592) that

[TABLE]

for every $n\in\mathbb{N}_{\geq 3}$, $z>\zeta_{1}(n;r,s)$, and $1/2\leq r<s<\infty$; thus, we have from (591) that

[TABLE]

for every $n\in\mathbb{N}_{\geq 3}$, $z>\zeta(n;r,s)$, and $1/2\leq r<s<\infty$. Therefore, combining (586), (590), and (595), we have

[TABLE]

for every $n\in\mathbb{N}_{\geq 3}$, $z\in[1,\infty)$, and $1/2\leq r<s<\infty$, which is the third assertion of Lemma 4. This completes the proof of Lemma 4. ∎