Arimoto-R\'enyi Conditional Entropy and Bayesian $M$-ary Hypothesis Testing

TL;DR

This paper explores bounds on the minimum error probability in Bayesian M-ary hypothesis testing using Arimoto-Rényi conditional entropy, extending classical inequalities and analyzing their implications for channel coding and error decay.

Contribution

It introduces new bounds on error probability based on Arimoto-Rényi entropy, generalizes Fano's inequality, and analyzes error decay in discrete memoryless channels.

Findings

Tighter bounds on error probability using Arimoto-Rényi entropy.

Generalization of Fano's inequality for finite and infinite hypotheses.

Analysis of error decay in channel coding scenarios.

Abstract

This paper gives upper and lower bounds on the minimum error probability of Bayesian $M$-ary hypothesis testing in terms of the Arimoto-R\'enyi conditional entropy of an arbitrary order $\alpha$. The improved tightness of these bounds over their specialized versions with the Shannon conditional entropy ($\alpha=1$) is demonstrated. In particular, in the case where $M$ is finite, we show how to generalize Fano's inequality under both the conventional and list-decision settings. As a counterpart to the generalized Fano's inequality, allowing $M$ to be infinite, a lower bound on the Arimoto-R\'enyi conditional entropy is derived as a function of the minimum error probability. Explicit upper and lower bounds on the minimum error probability are obtained as a function of the Arimoto-R\'enyi conditional entropy for both positive and negative $\alpha$. Furthermore, we give upper bounds on the minimum error probability as functions of the R\'enyi divergence. In the setup of discrete memoryless channels, we analyze the exponentially vanishing decay of the Arimoto-R\'enyi conditional entropy of the transmitted codeword given the channel output when averaged over a random coding ensemble.

Tables

Table 1. TABLE I : Lower bounds on ε X | Y subscript 𝜀 conditional 𝑋 𝑌 \varepsilon_{X|Y} in ( 82 ) and ( 135 ) for Example  1 .

$\alpha$	(82)	(135)
2	0.4247	0.3508
4	0.4480	0.4406
6	0.4573	0.4562
8	0.4620	0.4613
10	0.4640	0.4635
50	0.4667	0.4667

Table 2. TABLE II: Upper and Lower bounds on ε X | Y subscript 𝜀 conditional 𝑋 𝑌 \varepsilon_{X|Y} in ( 82 ) and ( 187 ) for Example  1 .

$\alpha$	lower bound	${\epsilon }_{X|Y}$	upper bound
1	0.4013	0.4667	0.6061
10	0.4640	0.4667	0.4994
100	0.4667	0.4667	0.4699

Equations

$${\epsilon }_{X|Y}\le \frac{1}{2}H(X|Y)\text{mbox}bits$$

$$\displaystyle\varepsilon_{X|Y}\leq 1-\exp\bigl{(}-H(X|Y)\bigr{)}.$$

$${\epsilon }_{X|Y}\le H_{\mathrm{o}}(X|Y)\le {\epsilon }_{X|Y}\left(2-\frac{M}{M-1}{\epsilon }_{X|Y}\right)$$

$$H_{\mathrm{o}}(X|Y)=\mathbb{E}\left[\sum _{x\in \mathcal{X}}{P}_{X|Y}(x|Y)\left(1-P_{X|Y}(x|Y)\right)\right]$$

$$H_{\alpha }(X)$$

$$=\frac{\alpha }{1-\alpha }\log \Vert P_X{\Vert }_{\alpha }$$

$$\displaystyle H_{0}(X)=\log\,\bigl{|}\mathop{\mathrm{supp}}P_{X}\bigr{|},$$

$$H_1(X)=H(X),$$

$$H_{\infty }(X)=\log \frac{1}{p_{\max }}$$

$${\epsilon }_X=1-p_{\max }$$

$$\displaystyle\varepsilon_{X}=1-\exp\bigl{(}-H_{\infty}(X)\bigr{)}.$$

$$h_{\alpha}(p)=H_{\alpha}(X_{p})=\frac{1}{1-\alpha}\,\log\bigl{(}p^{\alpha}+(1-p)^{\alpha}\bigr{)},$$

$$h(p)$$

$$H_{\alpha }(X|Y)$$

$$=\frac{\alpha }{1-\alpha }\log \mathbb{E}\left[\Vert P_{X|Y}(\cdot |Y){\Vert }_{\alpha }\right]$$

$$=\frac{\alpha }{1-\alpha }\log \sum _{y\in \mathcal{Y}}{P}_Y(y)\exp \left(\frac{1-\alpha }{\alpha }{H}_{\alpha }(X|Y=y)\right),$$

$$H_0(X|Y)$$

$$\displaystyle=\log\,\max_{y\in\mathcal{Y}}\,\bigl{|}\mathop{\mathrm{supp}}P_{X|Y}(\cdot|y)\bigr{|}$$

$$=\underset{y\in \mathcal{Y}}{\max }{H}_0(X|Y=y),$$

$$H_1(X|Y)$$

$$H_{\infty }(X|Y)$$

$$H_{-\infty }(X)=\log \frac{1}{p_{\min }},$$

$$H_{-\infty }(X|Y)=\log {\mathbb{E}}^{-1}\left[\underset{x\in \mathcal{X}:P_{X|Y}(x|Y)>0}{\min }{P}_{X|Y}(x|Y)\right]$$

$$H_{-\infty }(X|Y)=\log \frac{1}{1-\exp \left(-H_{\infty }(X|Y)\right)}.$$

$$\underset{\alpha \uparrow 0}{\lim }\frac{\alpha -1}{\alpha }{H}_{\alpha }(X|Y)$$

$$\underset{\alpha \downarrow 0}{\lim }\frac{\alpha -1}{\alpha }{H}_{\alpha }(X|Y)$$

$$H_{\beta }(X|Y)\le H_0(X|Y)\le H_{-\beta }(X|Y),$$

$$\frac{\alpha -1}{\alpha }{H}_{\alpha }(X|Y)\le \frac{\beta -1}{\beta }{H}_{\beta }(X|Y)$$

$$D_{\alpha }(P\Vert Q)$$

$$=\frac{1}{\alpha -1}\log \sum _{x\in \mathcal{X}}{P}^{\alpha }(x)Q^{1-\alpha }(x)$$

$$D_0(P\Vert Q)$$

$$D_1(P\Vert Q)$$

$$D_{\infty }(P\Vert Q)$$

$$d_{\alpha}(p\|q)=\frac{1}{\alpha-1}\;\log\Bigl{(}p^{\alpha}q^{1-\alpha}+(1-p)^{\alpha}(1-q)^{1-\alpha}\Bigr{)}.$$

$$\displaystyle\log\theta-d_{\alpha}\bigl{(}t\|\tfrac{s}{\theta}\bigr{)}=\frac{1}{1-\alpha}\;\log\left(t^{\alpha}\,s^{1-\alpha}+(1-t)^{\alpha}\,(\theta-s)^{1-\alpha}\right).$$

$$d_0(p\Vert q)$$

$$d_1(p\Vert q)$$

$$d_{\infty }(p\Vert q)$$

Full text

I Introduction

In Bayesian $M$-ary hypothesis testing, we have:

• •

$M$ possible explanations, hypotheses or models for the $\mathcal{Y}$-valued data $\{P_{Y|X=m},m\in\mathcal{X}\}$ where the set of model indices satisfies $|\mathcal{X}|=M$; and

• •

a prior distribution $P_{X}$ on $\mathcal{X}$.

The minimum probability of error of $X$ given $Y$, denoted by $\varepsilon_{X|Y}$, is achieved by the maximum-a-posteriori (MAP) decision rule. Summarized in Sections I-A and I-C, a number of bounds on $\varepsilon_{X|Y}$ involving Shannon information measures or their generalized Rényi measures have been obtained in the literature. As those works attest, there is considerable motivation for the study of the relationships between error probability and information measures. The minimum error probability of Bayesian $M$-ary hypothesis testing is rarely directly computable, and the best lower and upper bounds are information theoretic. Furthermore, their interplay is crucial in the proof of coding theorems.

I-A Existing bounds involving Shannon information measures

Fano’s inequality [24] gives an upper bound on the conditional entropy $H(X|Y)$ as a function of $\varepsilon_{X|Y}$ when $M$ is finite. 2. 2.

Shannon’s inequality [71] (see also [83]) gives an explicit lower bound on $\varepsilon_{X|Y}$ as a function of $H(X|Y)$, also when $M$ is finite. 3. 3.

Tightening another bound by Shannon [70], Poor and Verdú [57] gave a lower bound on $\varepsilon_{X|Y}$ as a function of the distribution of the conditional information (whose expected value is $H(X|Y)$). This bound was generalized by Chen and Alajaji [14]. 4. 4.

Baladová [6], Chu and Chueh [15, (12)], and Hellman and Raviv [38, (41)] showed that

[TABLE]

when $M$ is finite. It is also easy to show that (see, e.g., [25, (21)])

[TABLE]

Tighter and generalized upper bounds on $\varepsilon_{X|Y}$ were obtained by Kovalevsky [48], Tebbe and Dwyer [76], and Ho and Verdú [39, (109)]. 5. 5.

Based on the fundamental tradeoff of a certain auxiliary binary hypothesis test, Polyanskiy et al. [55] gave the meta-converse implicit lower bound on $\varepsilon_{X|Y}$, which for some choice of auxiliary quantities is shown to be tight in [82]. 6. 6.

Building up on [38], Kanaya and Han [45] showed that in the case of independent identically distributed (i.i.d.) observations, $\varepsilon_{X|Y^{n}}$ and $H(X|Y^{n})$ vanish exponentially at the same speed, which is governed by the Chernoff information between the closest hypothesis pair. In turn, Leang and Johnson [51] showed that the same exponential decay holds for those cost functions that have zero cost for correct decisions. 7. 7.

Birgé [9] gave an implicit lower bound on the minimax error probability ($\varepsilon_{X|Y}$ maximized over all possible priors) as a function of the pairwise relative entropies among the various models. 8. 8.

Generalizing Fano’s inequality, Han and Verdú [34] gave lower bounds on the mutual information $I(X;Y)$ as a function of $\varepsilon_{X|Y}$. 9. 9.

Grigoryan et al. [33] and Sason [67] obtained error bounds, in terms of relative entropies, for hypothesis testing with a rejection option. Such bounds recently proved useful in the context of an almost-fixed size hypothesis decision algorithm, which bridges the gap in performance between fixed-sample and sequential hypothesis testing [49].

I-B Rényi’s information measures

In this paper, we give upper and lower bounds on $\varepsilon_{X|Y}$ not in terms of $H(X|Y)$ but in terms of the Arimoto-Rényi conditional entropy $H_{\alpha}(X|Y)$ of an arbitrary order $\alpha$. Loosening the axioms given by Shannon [69] as a further buttress for $H(X)$, led Rényi [62] to the introduction of the Rényi entropy of order $\alpha\in[0,\infty]$, $H_{\alpha}(X)$, as well as the Rényi divergence $D_{\alpha}(P\|Q)$. Rényi’s entropy and divergence coincide with Shannon’s and Kullback-Leibler’s measures, respectively, for $\alpha=1$. Among other applications, $H_{\alpha}(X)$ serves to analyze the fundamental limits of lossless data compression [13, 16, 19]. Unlike Shannon’s entropy, $H_{\alpha}(X)$ with $\alpha\neq 1$ suffers from the disadvantage that the inequality $H_{\alpha}(X_{1},X_{2})\leq H_{\alpha}(X_{1})+H_{\alpha}(X_{2})$ does not hold in general. Moreover, if we mimic the definition of $H(X|Y)$ and define conditional Rényi entropy as $\sum_{y\in\mathcal{Y}}P_{Y}(y)\,H_{\alpha}(X|Y=y)$, we find the unpleasant property that the conditional version may be larger than $H_{\alpha}(X)$. To remedy this situation, Arimoto [5] introduced a notion of conditional Rényi entropy, which in this paper we denote as $H_{\alpha}(X|Y)$, and which is indeed upper bounded by $H_{\alpha}(X)$. The corresponding $\alpha$-capacity $\underset{X}{\max}\{H_{\alpha}(X)-H_{\alpha}(X|Y)\}$ yields a particularly convenient expression for the converse bound found by Arimoto [4] (see also [56]). The Arimoto-Rényi conditional entropy has also found applications in guessing and secrecy problems with side information ([2], [10], [35], [36] and [75]), sequential decoding [2], task encoding with side information available to both the task-describer (encoder) and the task-performer [11], and the list-size capacity of discrete memoryless channels in the presence of noiseless feedback [12]. In [65], the joint range of $H(X|Y)$ and $H_{\alpha}(X|Y)$ is obtained when the random variable $X$ is restricted to take its values on a finite set with a given cardinality. Although outside the scope of this paper, Csiszár [19] and Sibson [74] proposed implicitly other definitions of conditional Rényi entropy, which lead to the same value of the $\alpha$-capacity and have found various applications ([19] and [85]). The quantum generalizations of Rényi entropies and their conditional versions ([35, 36, 37, 43, 47, 54, 61, 77]) have recently served to prove strong converse theorems in quantum information theory ([36, 52] and [86]).

I-C Existing bounds involving Rényi’s information measures and related measures

In continuation to the list of existing bounds involving the conditional entropy and relative entropy in Section I-A, bounds involving Rényi’s information measures and related measures include:

For Bayesian binary hypothesis testing, Hellman and Raviv [38] gave an upper bound on $\varepsilon_{X|Y}$ as a function of the prior probabilities and the Rényi divergence of order $\alpha\in[0,1]$ between the two models. The special case of $\alpha=\frac{1}{2}$ yields the Bhattacharyya bound [44]. 2. 11.

In [21] and [81], Devijver and Vajda derived upper and lower bounds on $\varepsilon_{X|Y}$ as a function of the quadratic Arimoto-Rényi conditional entropy $H_{2}(X|Y)$. 3. 12.

In [17], Cover and Hart showed that if $M$ is finite, then

[TABLE]

where

[TABLE]

is referred to as conditional quadratic entropy. The bounds in (3) coincide if $X$ is equiprobable on $\mathcal{X}$, and $X$ and $Y$ are independent. 4. 13.

One of the bounds by Han and Verdú [34] in Item 8) was generalized by Polyanskiy and Verdú [56] to give a lower bound on the $\alpha$-mutual information ([74, 85]). 5. 14.

In [73], Shayevitz gave a lower bound, in terms of the Rényi divergence, on the maximal worst-case miss-detection exponent for a binary composite hypothesis testing problem when the false-alarm probability decays to zero with the number of i.i.d. observations. 6. 15.

Tomamichel and Hayashi studied optimal exponents of binary composite hypothesis testing, expressed in terms of Rényi’s information measures ([37, 78]). A measure of dependence was studied in [78] (see also Lapidoth and Pfister [50]) along with its role in composite hypothesis testing. 7. 16.

Fano’s inequality was generalized by Toussiant in [80, Theorem 2] (see also [22] and [43, Theorem 6]) not with the Arimoto-Rényi conditional entropy but with the average of the Rényi entropies of the conditional distributions (which, as we mentioned in Section I-B, may exceed the unconditional Rényi entropy). 8. 17.

A generalization of the minimax lower bound by Birgé [9] in Item 7) to lower bounds involving $f$-divergences has been studied by Guntuboyina [32].

I-D Main results and paper organization

It is natural to consider generalizing the relationships between $\varepsilon_{X|Y}$ and $H(X|Y)$, itemized in Section I-A, to the Arimoto-Rényi conditional entropy $H_{\alpha}(X|Y)$. In this paper we find pleasing counterparts to the bounds in Items 1), 4), 6), 10), 11), and 13), resulting in generally tighter bounds. In addition, we enlarge the scope of the problem to consider not only $\varepsilon_{X|Y}$ but the probability that a list decision rule (which is allowed to output a set of $L$ hypotheses) does not include the true one. Previous work on extending Fano’s inequality to the setup of list decision rules includes [1, Section 5], [46, Lemma 1], [59, Appendix 3.E] and [84, Chapter 5].

Section II introduces the basic notation and definitions of Rényi information measures. Section III finds bounds for the guessing problem in which there are no observations; those bounds, which prove to be instrumental in the sequel, are tight in the sense that they are attained with equality for certain random variables. Section IV contains the main results in the paper on the interplay between $\varepsilon_{X|Y}$ and $H_{\alpha}(X|Y)$, giving counterparts to a number of those existing results mentioned in Sections I-A and I-C. In particular:

An upper bound on $H_{\alpha}(X|Y)$ as a function of $\varepsilon_{X|Y}$ is derived for positive $\alpha$ (Theorem 3); it provides an implicit lower bound on $\varepsilon_{X|Y}$ as a function of $H_{\alpha}(X|Y)$; 2. 19.

Explicit lower bounds on $\varepsilon_{X|Y}$ are given as a function of $H_{\alpha}(X|Y)$ for both positive and negative $\alpha$ (Theorems 5, 6, and 7); 3. 20.

The lower bounds in Items 18) and 19) are generalized to the list-decoding setting (Theorems 8, 9, and 10); 4. 21.

As a counterpart to the generalized Fano’s inequality, we derive a lower bound on $H_{\alpha}(X|Y)$ as a function of $\varepsilon_{X|Y}$ capitalizing on the Schur concavity of Rényi entropy (Theorem 11); 5. 22.

Explicit upper bounds on $\varepsilon_{X|Y}$ as a function of $H_{\alpha}(X|Y)$ are obtained (Theorem 12);

Section V gives explicit upper bounds on $\varepsilon_{X|Y}$ as a function of the error probability of associated binary tests (Theorem 14) and of the Rényi divergence (Theorems 13 and 15) and Chernoff information (Theorem 15).

Section VI analyzes the exponentially vanishing decay of the Arimoto-Rényi conditional entropy of the transmitted codeword given the channel output when averaged over a code ensemble (Theorems 17 and 18). Concluding remarks are given in Section VII.

II Rényi Information Measures: Definitions and Basic Properties

Definition 1

[62]** Let $P_{X}$ be a probability distribution on a discrete set $\mathcal{X}$. The Rényi entropy of order $\alpha\in(0,1)\cup(1,\infty)$ of $X$, denoted by $H_{\alpha}(X)$ or $H_{\alpha}(P_{X})$, is defined as

[TABLE]

where $\|P_{X}\|_{\alpha}=\left(\sum_{x\in\mathcal{X}}P_{X}^{\alpha}(x)\right)^{\frac{1}{\alpha}}$. By its continuous extension,

[TABLE]

where $\mathop{\mathrm{supp}}P_{X}=\{x\in\mathcal{X}\colon P_{X}(x)>0\}$ is the support of $P_{X}$, and $p_{\max}$ is the largest of the masses of $X$.

When guessing the value of $X$ in the absence of an observation of any related quantity, the maximum probability of success is equal to $p_{\max}$; it is achieved by guessing that $X=x_{0}$ where $x_{0}$ is a mode of $P_{X}$, i.e., $P_{X}(x_{0})=p_{\max}$. Let

[TABLE]

denote the minimum probability of error in guessing the value of $X$. From (9) and (10), the minimal error probability in guessing $X$ without side information is given by

[TABLE]

Definition 2

For $\alpha\in(0,1)\cup(1,\infty)$, the binary Rényi entropy of order $\alpha$ is the function $h_{\alpha}\colon[0,1]\to[0,\log 2]$ that is defined, for $p\in[0,1]$, as

[TABLE]

where $X_{p}$ takes two possible values with probabilities $p$ and $1-p$. The continuous extension of the binary Rényi entropy at $\alpha=1$ yields the binary entropy function:

[TABLE]

for $p\in(0,1)$, and $h(0)=h(1)=0$.

In order to put forth generalizations of Fano’s inequality and bounds on the error probability, we consider Arimoto’s proposal for the conditional Rényi entropy (named, for short, the Arimoto-Rényi conditional entropy).

Definition 3

[5*]**

Let $P_{XY}$ be defined on $\mathcal{X}\times\mathcal{Y}$, where $X$ is a discrete random variable. The Arimoto-Rényi conditional entropy of order $\alpha\in[0,\infty]$ of $X$ given $Y$ is defined as follows:*

• •

If $\alpha\in(0,1)\cup(1,\infty)$, then

[TABLE]

where (16) applies if $Y$ is a discrete random variable.

• •

By its continuous extension, the Arimoto-Rényi conditional entropy of orders [math], $1$, and $\infty$ are defined as

[TABLE]

where $\operatorname*{\mathrm{ess}\sup}$ in (17) denotes the essential supremum, and (18) and (19) apply if $Y$ is a discrete random variable.

Although not nearly as important, sometimes in the context of finitely valued random variables, it is useful to consider the unconditional and conditional Rényi entropies of negative orders $\alpha\in(-\infty,0)$ in (5) and (14) respectively. By continuous extension

[TABLE]

where $p_{\min}$ in (22) is the smallest of the nonzero masses of $X$, and $\mathbb{E}^{-1}[Z]$ in (23) denotes the reciprocal of the expected value of $Z$. In particular, note that if $|\mathcal{X}|=2$, then

[TABLE]

Basic properties of $H_{\alpha}(X|Y)$ appear in [27], and in [43, 77] in the context of quantum information theory. Next, we provide two additional properties, which are used in the sequel.

Proposition 1

$H_{\alpha}(X|Y)$* is monotonically decreasing in $\alpha$ throughout the real line.*

The proof of Proposition 1 is given in Appendix A. In the special case of $\alpha\in[1,\infty]$ and discrete $Y$, a proof can be found in [8, Proposition 4.6].

We will also have opportunity to use the following monotonicity result, which holds as a direct consequence of the monotonicity of the norm of positive orders.

Proposition 2

$\frac{\alpha-1}{\alpha}\,H_{\alpha}(X|Y)$* is monotonically increasing in $\alpha$ on $(0,\infty)$ and $(-\infty,0)$.*

Remark 1

Unless $X$ is a deterministic function of $Y$, it follows from (17) that $H_{0}(X|Y)>0$; consequently,

[TABLE]

It follows from Proposition 1 that if $\beta>0$, then

[TABLE]

and, from Proposition 2,

[TABLE]

if $0<\alpha<\beta$ or $\alpha<\beta<0$.

The third Rényi information measure used in this paper is the Rényi divergence. Properties of the Rényi divergence are studied in [23], [68, Section 8] and [73].

Definition 4

[62]** Let $P$ and $Q$ be probability measures on $\mathcal{X}$ dominated by $R$, and let their densities be respectively denoted by $p=\frac{\mathrm{d}P}{\mathrm{d}R}$ and $q=\frac{\mathrm{d}Q}{\mathrm{d}R}$. The Rényi divergence of order $\alpha\in[0,\infty]$ is defined as follows:

• •

If $\alpha\in(0,1)\cup(1,\infty)$, then

[TABLE]

where $Z\sim R$ in (29), and (30) holds if $\mathcal{X}$ is a discrete set.

• •

By the continuous extension of $D_{\alpha}(P\|Q)$,

[TABLE]

with $Z\sim R$.

Definition 5

For all $\alpha\in(0,1)\cup(1,\infty)$, the binary Rényi divergence of order $\alpha$, denoted by $d_{\alpha}(p\|q)$ for $(p,q)\in[0,1]^{2}$, is defined as $D_{\alpha}([p~{}1-p]\|[q~{}1-q])$. It is the continuous extension to $[0,1]^{2}$ of

[TABLE]

Used several times in this paper, it is easy to verify the following identity satisfied by the binary Rényi divergence. If $t\in[0,1]$ and $s\in[0,\theta]$ for $\theta>0$, then

[TABLE]

By analytic continuation in $\alpha$, for $(p,q)\in(0,1)^{2}$,

[TABLE]

where $d(\cdot\|\cdot)$ denotes the binary relative entropy. Note that if $t\in[0,1]$ and $M>1$, then

[TABLE]

A simple nexus between the Rényi entropy and the Rényi divergence is

[TABLE]

when $X$ takes values on a set of $M$ elements on which $U$ is equiprobable.

Definition 6

The Chernoff information of a pair of probability measures defined on the same measurable space is equal to

[TABLE]

Definition 7

([74], [85]) Given the probability distributions $P_{X}$ and $P_{Y|X}$, the $\alpha$-mutual information is defined for $\alpha>0$ as

[TABLE]

As for the conventional mutual information, which refers to $\alpha=1$, sometimes it is useful to employ the alternative notation

[TABLE]

In the discrete case, for $\alpha\in(0,1)\cup(1,\infty)$,

[TABLE]

where

[TABLE]

denotes Gallager’s error exponent function [29]. Note that, in general, $I_{\alpha}(X;Y)$ does not correspond to the difference $H_{\alpha}(X)-H_{\alpha}(X|Y)$, which is equal to

[TABLE]

where $P_{X_{\alpha}}$ is the scaled version of $P_{X}$, namely the normalized version of $P_{X}^{\alpha}$. Since the equiprobable distribution is equal to its scaled version for any $\alpha\geq 0$, if $X$ is equiprobable on a set of $M$ elements, then

[TABLE]

III Upper and Lower Bounds on $H_{\alpha}(X)$

In this section, we obtain upper and lower bounds on the unconditional Rényi entropy of order $\alpha$ which are tight in the sense that they are attained with equality for certain random variables.

III-A Upper bounds

In this subsection we limit ourselves to finite alphabets.

Theorem 1

Let $\mathcal{X}$ and $\mathcal{Y}$ be finite sets, and let $X$ be a random variable taking values on $\mathcal{X}$. Let $f\colon\mathcal{X}\to\mathcal{Y}$ be a deterministic function, and denote the cardinality of the inverse image by $L_{y}=|f^{-1}(y)|$ for every $y\in\mathcal{Y}$. Then, for every $\alpha\in[0,1)\cup(1,\infty)$, the Rényi entropy of $X$ satisfies

[TABLE]

which holds with equality if and only if either $\alpha=0$ or $\alpha\in(0,1)\cup(1,\infty)$ and $P_{X}$ is equiprobable on $f^{-1}(y)$ for every $y\in\mathcal{Y}$ such that $P_{X}(f^{-1}(y))>0$.

Proof:

It can be verified from (7) that (48) holds with equality for $\alpha=0$ (by assumption $\mathop{\mathrm{supp}}P_{X}=\mathcal{X}$, and $\underset{y\in\mathcal{Y}}{\sum}L_{y}=|\mathcal{X}|$).

Suppose $\alpha\in(0,1)\cup(1,\infty)$. Let $|\mathcal{X}|=M<\infty$, and let $U$ be equiprobable on $\mathcal{X}$. Let $V=f(X)$ and $W=f(U)$, so $(V,W)\in\mathcal{Y}^{2}$ and

[TABLE]

for all $y\in\mathcal{Y}$. To show (48), note that

[TABLE]

where (51) is (40); (52) holds due to the data processing inequality for the Rényi divergence (see [23, Theorem 9 and Example 2]); and (53) follows from (30), (49) and (50).

From (51)–(54), the upper bound in (48) is attained with equality if and only if the data processing inequality (52) holds with equality. For all $\alpha\in(0,1)\cup(1,\infty)$,

[TABLE]

where (55) is satisfied by the definition in (30) with $U$ being equiprobable on $\mathcal{X}$ and $|\mathcal{X}|=M$; (56) holds by expressing the sum over $\mathcal{X}$ as a double sum over $\mathcal{Y}$ and the elements of $\mathcal{X}$ that are mapped by $f$ to the same element in $\mathcal{Y}$; inequality (57) holds since $|f^{-1}(y)|=L_{y}$ for all $y\in\mathcal{Y}$, and for every random variable $Z$ we have $\mathbb{E}[Z^{\alpha}]\geq\mathbb{E}^{\alpha}[Z]$ if $\alpha\in(1,\infty)$ or the opposite inequality if $\alpha\in[0,1)$; finally, (58)–(59) are due to (30), (49) and (50). Note that (57) holds with equality if and only if $P_{X}$ is equiprobable on $f^{-1}(y)$ for every $y\in\mathcal{Y}$ such that $P_{X}(f^{-1}(y))>0$. Hence, for $\alpha\in(0,1)\cup(1,\infty)$, (52) holds with equality if and only if the condition given in the theorem statement is satisfied. ∎

Corollary 1

Let $X$ be a random variable taking values on a finite set $\mathcal{X}$ with $|\mathcal{X}|=M$, and let $\mathcal{L}\subseteq\mathcal{X}$ with $|\mathcal{L}|=L$. Then, for all $\alpha\in[0,1)\cup(1,\infty)$,

[TABLE]

with equality in (60) if and only if $X$ is equiprobable on both $\mathcal{L}$ and $\mathcal{L}^{\mathrm{c}}$.

Proof:

Inequality (60) is a specialization of Theorem 1 to a binary-valued function $f\colon\mathcal{X}\to\{0,1\}$ such that $f^{-1}(0)=\mathcal{L}$ and $f^{-1}(1)=\mathcal{L}^{\mathrm{c}}$. Equality (61) holds by setting $\theta=M$, $t=\mathbb{P}[X\in\mathcal{L}]$, and $s=L$ in (35). ∎

Moreover, specializing Corollary 1 to $L=1$, we obtain:

Corollary 2

Let $X$ be a random variable taking values on a finite set $\mathcal{X}$ with $|\mathcal{X}|=M$. Then, for all $\alpha\in[0,1)\cup(1,\infty)$,

[TABLE]

with equality in (62) if and only if $P_{X}$ is equiprobable on $\mathcal{X}\setminus\{x^{\ast}\}$ where $x^{\ast}\in\mathcal{X}$ attains the maximum in (63).

Remark 2

Taking the limit $\alpha\to 1$ in the right side of (62) yields

[TABLE]

with the same necessary and sufficient condition for equality in Corollary 2. Furthermore, choosing $x\in\mathcal{X}$ to be a mode of $P_{X}$ gives

[TABLE]

where (66) follows from (10), and the symmetry of the binary entropy function around $\tfrac{1}{2}$.

If we loosen Corollary 2 by, instead of minimizing the right side of (62), choosing $x\in\mathcal{X}$ to be a mode of $P_{X}$, we recover [7, Theorem 6]:

Corollary 3

Let $X$ be a random variable taking $M$ possible values, and assume that its largest mass is $p_{\max}$. Then, for all $\alpha\in[0,1)\cup(1,\infty)$,

[TABLE]

with equality in (67) under the condition in the statement of Corollary 2 with $x^{\ast}$ being equal to a mode of $P_{X}$.

Remark 3

In view of the necessary and sufficient condition for equality in (62), the minimization of the upper bound in the right side of (62) for a given $P_{X}$ does not necessarily imply that the best choice of $x\in\mathcal{X}$ is a mode of $P_{X}$. For example, let $\mathcal{X}=\{0,1,2\}$, $P_{X}(0)=0.2$, and $P_{X}(1)=P_{X}(2)=0.4$. Then, $H_{2}(X)=1.474$ bits, which coincides with its upper bound in (62) by choosing $x=0$; on the other hand, (67) yields $H_{2}(X)\leq 1.556$ bits.

III-B Lower bound

The following lower bound provides a counterpart to the upper bound in Corollary 3 without restricting to finitely valued random variables.

Theorem 2

Let $X$ be a discrete random variable attaining a maximal mass $p_{\max}$. Then, for $\alpha\in(0,1)\cup(1,\infty)$,

[TABLE]

where, for $x\in\mathbb{R}$, $\lfloor x\rfloor$ denotes the largest integer that is smaller than or equal to $x$. Equality in (69) holds if and only if $P_{X}$ has \Big{\lfloor}\tfrac{1}{p_{\max}}\Big{\rfloor} masses equal to $p_{\max}$, and an additional mass equal to 1-p_{\max}\,\Big{\lfloor}\frac{1}{p_{\max}}\Big{\rfloor} whenever $\frac{1}{p_{\max}}$ is not an integer.

Proof:

For $\alpha\in(0,1)\cup(1,\infty)$, the Rényi entropy of the distribution identified in the statement of Theorem 2 as attaining the condition for equality is equal to the right side of (69). Furthermore, that distribution majorizes any distribution whose maximum mass is $p_{\max}$. The result in (69) follows from the Schur-concavity of the Rényi entropy in the general case of a countable alphabet (see [40, Theorem 2]). In view of Lemma 1 and Theorem 2 of [40], the Schur concavity of the Rényi entropy is strict for any $\alpha\in(0,1)\cup(1,\infty)$ and therefore (69) holds with strict inequality for any distribution other than the one specified in the statement of this result. To get (70), let s=\Bigl{\lfloor}\frac{1}{p_{\max}}\Bigr{\rfloor}, $t=s\,p_{\max}$ and $\theta=s+1$ in (35). ∎

Remark 4

Let $X$ be a discrete random variable, and consider the case of $\alpha=0$. Unless $X$ is finitely valued, (7) yields $H_{0}(X)=\infty$. If $X$ is finitely valued, then (7) and the inequality $p_{\max}\,|\mathop{\mathrm{supp}}P_{X}|\geq 1$ yield

[TABLE]

where, for $x\in\mathbb{R}$, $\lceil x\rceil$ denotes the smallest integer that is larger than or equal to $x$. The bound in (69) therefore holds for $\alpha=0$ with the convention that $0^{0}=0$, and it then coincides with (71). Equality in (71) holds if and only if $|\mathop{\mathrm{supp}}P_{X}|=\left\lceil\tfrac{1}{p_{\max}}\right\rceil$ (e.g., if $X$ is equiprobable).

Remark 5

Taking the limit $\alpha\to 1$ in Theorem 2 yields

[TABLE]

with equality in (72) if and only if the condition for equality in Theorem 2 holds. Hence, the result in Theorem 2 generalizes the bound in (72), which is due to Kovalevsky [48] and Tebbe and Dwyer [76] in the special case of a finitely-valued $X$ (rediscovered in [25]), and to Ho and Verdú [39, Theorem 10] in the general setting of a countable alphabet.

IV Arimoto-Rényi Conditional Entropy and Error Probability

Section IV forms the main part of this paper, and its results are outlined in Section I-D (see Items 18)–22)).

IV-A Upper bound on the Arimoto-Rényi conditional entropy: Generalized Fano’s inequality

The minimum error probability $\varepsilon_{X|Y}$ can be achieved by a deterministic function (maximum-a-posteriori decision rule) $\mathcal{L}^{\ast}\colon\mathcal{Y}\to\mathcal{X}$:

[TABLE]

where (77) is the minimum error probability achievable among blind decision rules that disregard the observations (see (10)).

Fano’s inequality links the decision-theoretic uncertainty $\varepsilon_{X|Y}$ and the information-theoretic uncertainty $H(X|Y)$ through

[TABLE]

where the identity in (80) is (39) with $t=\varepsilon_{X|Y}$. Although the form of Fano’s inequality in (79) is not nearly as popular as (80), it turns out to be the version that admits an elegant (although not immediate) generalization to the Arimoto-Rényi conditional entropy. It is straightforward to obtain (80) by averaging a conditional version of (66) with respect to the observation. This simple route to the desired result is not viable in the case of $H_{\alpha}(X|Y)$ since it is not an average of Rényi entropies of conditional distributions. The conventional proof of Fano’s inequality (e.g., [18, pp. 38–39]), based on the use of the chain rule for entropy, is also doomed to failure for the Arimoto-Rényi conditional entropy of order $\alpha\neq 1$ since it does not satisfy the chain rule.

Before we generalize Fano’s inequality by linking $\varepsilon_{X|Y}$ with $H_{\alpha}(X|Y)$ for $\alpha\in(0,\infty)$, note that for $\alpha=\infty$, the following generalization of (11) holds in view of (21) and (76):

[TABLE]

Theorem 3

Let $P_{XY}$ be a probability measure defined on $\mathcal{X}\times\mathcal{Y}$ with $|\mathcal{X}|=M<\infty$. For all $\alpha\in(0,\infty)$,

[TABLE]

Let $\mathcal{L}^{\ast}\colon\mathcal{Y}\to\mathcal{X}$ be a deterministic MAP decision rule. Equality holds in (82) if and only if, for all $y\in\mathcal{Y}$,

[TABLE]

Proof:

If in Corollary 3 we replace $P_{X}$ by $P_{X|Y=y}$, then $p_{\max}$ is replaced by $\underset{x\in\mathcal{X}}{\max}\,P_{X|Y}(x|y)$ and we obtain

[TABLE]

where we have defined the conditional error probability given the observation:

[TABLE]

which satisfies with $Y\sim P_{Y}$,

[TABLE]

For $\alpha\in(0,1)\cup(1,\infty)$, $(\beta,\gamma)\in(0,\infty)^{2}$, define the function $f_{\alpha,\beta,\gamma}\colon[0,1]\to(0,\infty)$:

[TABLE]

If $\alpha>1$, then (16) allows us to bound a monotonically decreasing function of $H_{\alpha}(X|Y)$:

[TABLE]

where $\beta=(M-1)^{1-\alpha}$ in (88)–(90); (88) follows from (16) and (84); (90) follows from (86) and Jensen’s inequality, due to the convexity of $f_{\alpha,\beta,\gamma}\colon[0,1]\to(0,\infty)$ for $\alpha\in(1,\infty)$ and $(\beta,\gamma)\in(0,\infty)^{2}$ (see Lemma 1 following this proof); finally, (91) follows from (87). For $\alpha\in(1,\infty)$, (82) follows from (88)–(91) and identity (35) with $(\theta,s,t)=(M,\,M-1,\,\varepsilon_{X|Y})$.

For $\alpha\in(0,1)$, inequality (88) is reversed; and due to the concavity of $f_{\alpha,\beta,\gamma}(\cdot)$ on $[0,1]$ (Lemma 1), inequality (90) is also reversed. The rest of the proof proceeds as in the case where $\alpha>1$.

The necessary and sufficient condition for equality in (82) follows from the condition for equality in the statement of Corollary 3 when $P_{X}$ is replaced by $P_{X|Y=y}$ for $y\in\mathcal{Y}$. Under (83), it follows from (74)–(78) and (85) that $\varepsilon_{X|Y}(y)=\varepsilon_{X|Y}$ for all $y\in\mathcal{Y}$; this implies that inequalities (88) and (90) for $\alpha\in(1,\infty)$, and the opposite inequalities for $\alpha\in(0,1)$, hold with equalities. ∎

Lemma 1

Let $\alpha\in(0,1)\cup(1,\infty)$ and $(\beta,\gamma)\in(0,\infty)^{2}$. The function $f_{\alpha,\beta,\gamma}\colon[0,1]\to(0,\infty)$ defined in (87) is strictly convex for $\alpha\in(1,\infty)$, and strictly concave for $\alpha\in(0,1)$.

Proof:

The second derivative of $f_{\alpha,\beta,\gamma}(\cdot)$ in (87) is given by

[TABLE]

which is strictly negative if $\alpha\in(0,1)$, and strictly positive if $\alpha\in(1,\infty)$ for any $u\in[0,1]$. ∎

Remark 6

From (37), Fano’s inequality (see (79)–(80)) is recovered by taking the limit $\alpha\to 1$ in (82).

Remark 7

A pleasing feature of Theorem 3 is that as $\alpha\to\infty$, the bound becomes tight. To see this, we rewrite the identity in (81) as

[TABLE]

where (94) follows from (38) since $\varepsilon_{X|Y}\leq 1-\tfrac{1}{M}$ (see (74)–(78)).

Remark 8

For $\alpha=0$, (82) also holds. To see this, it is useful to distinguish two cases:

• •

$\varepsilon_{X|Y}=0$. Then, $H_{0}(X|Y)=0$, and the right side of (82) is also equal to zero since $d_{0}(0\|q)=-\log(1-q)$ for all $q\in[0,1)$.

• •

$\varepsilon_{X|Y}>0$. Then, the right side of (82) is equal to $\log M$ (see (36)) which is indeed an upper bound to $H_{0}(X|Y)$. The condition for equality in this case is that there exists $y\in\mathcal{Y}$ such that $P_{X|Y}(x|y)>0$ for all $x\in\mathcal{X}$.

Remark 9

Since d_{\alpha}\bigl{(}\cdot\|1-\tfrac{1}{M}\bigr{)} is monotonically decreasing in $[0,1-\tfrac{1}{M}]$, Theorem 3 gives an implicit lower bound on $\varepsilon_{X|Y}$. Although, currently, there is no counterpart to Shannon’s explicit lower bound as a function of $H(X|Y)$ [71], we do have explicit lower bounds as a function of $H_{\alpha}(X|Y)$ in Section IV-B.

Remark 10

If $X$ and $Y$ are random variables taking values on a set of $M$ elements and $X$ is equiprobable, then [56, Theorem 5.3] shows that

[TABLE]

which, together with (47), yields (82). However, note that in Theorem 3 we do not restrict $X$ to be equiprobable.

In information-theoretic problems, it is common to encounter the case in which $X$ and $Y$ are actually vectors of dimension $n$. Fano’s inequality ensures that vanishing error probability implies vanishing normalized conditional entropy as $n\to\infty$. As we see next, the picture with the Arimoto-Rényi conditional entropy is more nuanced.

Theorem 4

Let $\{X_{n}\}$ be a sequence of random variables, with $X_{n}$ taking values on $\mathcal{X}_{n}$ for $n\in\{1,2,3,\ldots\}$, and assume that there exists an integer $M\geq 2$ such that $|\mathcal{X}_{n}|\leq M^{n}$ for all $n$.111Note that this encompasses the conventional setting in which $\mathcal{X}_{n}=\mathcal{A}^{n}$. Let $\{Y_{n}\}$ be an arbitrary sequence of random variables, for which $\varepsilon_{X_{n}|Y_{n}}\to 0$ as $n\to\infty$.

1. a)

If $\alpha\in(1,\infty]$, then $H_{\alpha}(X_{n}|Y_{n})\to 0$; 2. b)

If $\alpha=1$, then $\frac{1}{n}\,H(X_{n}|Y_{n})\to 0$; 3. c)

If $\alpha\in[0,1)$, then $\frac{1}{n}\,H_{\alpha}(X_{n}|Y_{n})$ is upper bounded by $\log M$; nevertheless, it does not necessarily tend to 0.

Proof:

1. a)

For $\alpha\in(1,\infty)$,

[TABLE]

where (96) follows from (82) and $|\mathcal{X}_{n}|\leq M^{n}$; (97) holds due to (35) by setting the parameters $\theta=M^{n}$, $s=M^{n}-1$ and $t=\varepsilon_{X^{n}|Y^{n}}$; hence, $H_{\alpha}(X_{n}|Y_{n})\to 0$ since $\alpha>1$ and $\varepsilon_{X_{n}|Y_{n}}\to 0$. Item a) also holds for $\alpha=\infty$ since $H_{\alpha}(X_{n}|Y_{n})$ is monotonically decreasing in $\alpha$ throughout the real line (Proposition 1). 2. b)

For $\alpha=1$, from Fano’s inequality,

[TABLE]

Hence, not only does $\tfrac{1}{n}\,H(X_{n}|Y_{n})\to 0$ if $\varepsilon_{X_{n}|Y_{n}}=o(1)$ but also $H(X_{n}|Y_{n})\to 0$ if $\varepsilon_{X_{n}|Y_{n}}=o\left(\frac{1}{n}\right)$. 3. c)

Proposition 1 implies that if $\alpha>0$, then

[TABLE]

∎

A counterexample where $\varepsilon_{X_{n}|Y_{n}}\to 0$ exponentially fast, and yet $\frac{1}{n}H_{\alpha}(X_{n}|Y_{n})\not\to 0$ for all $\alpha\in[0,1)$ is given as follows.

Remark 11

In contrast to the conventional case of $\alpha=1$, it is not possible to strengthen Theorem 4c) to claim that $\varepsilon_{X_{n}|Y_{n}}\to 0$ implies that $\frac{1}{n}H_{\alpha}(X_{n}|Y_{n})\to 0$ for $\alpha\in[0,1).$ By Proposition 1, it is sufficient to consider the following counterexample: fix any $\alpha\in(0,1)$, and let $Y_{n}$ be deterministic, $\mathcal{X}_{n}=\{1,\ldots,M^{n}\}$, and $X_{n}\sim P_{X_{n}}$ where

[TABLE]

with

[TABLE]

Then,

[TABLE]

and

[TABLE]

where (105) holds since $Y_{n}$ is deterministic; (106) follows from (5) and (102); (107) holds due to (103). Consequently, since $\alpha\in(0,1)$, $M\geq 2$, and $\beta>1$, the second term in the right side of (108) tends to 0. In conclusion, normalizing (105)–(108) by $n$, and letting $n\to\infty$ yields

[TABLE]

Remark 12

Theorem 4b) is due to [39, Theorem 15]. Furthermore, [39, Example 2] shows that Theorem 4b) cannot be strengthened to $H(X_{n}|Y_{n})\to 0$, in contrast to the case where $\alpha\in(1,\infty]$ in Theorem 4a).

IV-B Explicit lower bounds on $\varepsilon_{X|Y}$

The results in Section IV-A yield implicit lower bounds on $\varepsilon_{X|Y}$ as a function of the Arimoto-Rényi conditional entropy. In this section, we obtain several explicit bounds. As the following result shows, Theorem 3 readily results in explicit lower bounds on $\varepsilon_{X|Y}$ as a function of $H_{\frac{1}{2}}(X|Y)$ and of $H_{2}(X|Y)$.

Theorem 5

Let $X$ be a discrete random variable taking $M\geq 2$ possible values. Then,

[TABLE]

where

[TABLE]

Proof:

Since $0\leq H_{2}(X|Y)\leq H_{\frac{1}{2}}(X|Y)\leq\log M$, (112)–(113) imply that $1\leq\xi_{1}\leq\xi_{2}\leq M$. To prove (110), setting $\alpha=\tfrac{1}{2}$ in (82) and using (35) with $(\theta,s,t)=(M,M-1,\varepsilon_{X|Y})$ we obtain

[TABLE]

Substituting

[TABLE]

yields the inequality

[TABLE]

If the right side of (117) is non-negative, then (117) is transformed to the following quadratic inequality in $v$:

[TABLE]

which yields

[TABLE]

If, however, the right side of (117) is negative then

[TABLE]

which implies the satisfiability of (119) also in the latter case. Hence, (119) always holds. In view of (112), (115), (116), and since $\xi_{1}\in[1,M]$, it can be verified that the right side of (119) is non-negative. Squaring both sides of (119), and using (112), (115) and (116) give (110).

Similarly, setting $\alpha=2$ in (82) and using (35) with $(\theta,s,t)=(M,M-1,\varepsilon_{X|Y})$ yield a quadratic inequality in $\varepsilon_{X|Y}$, from which (111) follows. ∎

Remark 13

Following up on this work, Renes [60] has generalized (110) to the quantum setting.

Remark 14

The corollary to Theorem 3 in (111) is equivalent to [21, Theorem 3].

Remark 15

Consider the special case where $X$ is an equiprobable binary random variable, and $Y$ is a discrete random variable which takes values on a set $\mathcal{Y}$. Following the notation in [44, (7)], let $\rho\in[0,1]$ denote the Bhattacharyya coefficient

[TABLE]

where (122) holds due to Bayes’ rule which implies that $P_{Y|X}(y|x)=2P_{X|Y}(x|y)P_{Y}(y)$ for all $x\in\{0,1\}$ and $y\in\mathcal{Y}$. From (14) and (122), we obtain

[TABLE]

Since $X$ is a binary random variable, it follows from (112) and (123) that $\xi_{1}=\frac{2}{1+\rho}$; hence, the lower bound on the minimal error probability in (110) is given by

[TABLE]

recovering the bound in [44, (49)] (see also [79]).

Remark 16

The lower bounds on $\varepsilon_{X|Y}$ in (110) and (111) depend on $H_{\alpha}(X|Y)$ with $\alpha=\tfrac{1}{2}$ and $\alpha=2$, respectively; due to their dependence on different orders $\alpha$, none of these bounds is superseded by the other, and they both prove to be useful in view of their relation to the conditional Bayesian distance in [21, Definition 2] and the Bhattacharyya coefficient (see Remarks 14 and 15).

Remark 17

Taking the limit $M\to\infty$ in the right side of (111) yields

[TABLE]

which is equivalent to [81, (8)] (see also [21, Theorem 2]), and its loosening yields [21, Corollary 1]. Theorem 7 (see also Theorem 10) tightens (125).

Remark 18

Arimoto [3] introduced a different generalization of entropy and conditional entropy parameterized by a continuously differentiable function $\mathtt{f}\colon(0,1]\to[0,\infty)$ satisfying $\mathtt{f}(1)=0$:

[TABLE]

where the infimum is over all the distributions defined on the same set as $X$. Arimoto [3, Theorem 3] went on to show the following generalization of Fano’s inequality:

[TABLE]

A functional dependence can be established between the Rényi entropy and $H_{\mathtt{f}}(X)$ for a certain choice of $\mathtt{f}$ (see [3, Example 2]); in view of (16), the Arimoto-Rényi conditional entropy can be expressed in terms of $H_{\mathtt{f}}(X|Y)$, although the analysis of generalizing Fano’s inequality with the Arimoto-Rényi conditional entropy becomes rather convoluted following this approach.

For convenience, we assume throughout the rest of this subsection that

[TABLE]

The following bound, which is a special case of Theorem 9, involves the Arimoto-Rényi conditional entropy of negative orders, and bears some similarity to Arimoto’s converse for channel coding [4].

Theorem 6

Let $P_{XY}$ be a probability measure defined on $\mathcal{X}\times\mathcal{Y}$ with $|\mathcal{X}|=M<\infty$, which satisfies (129). For all $\alpha\in(-\infty,0)$,

[TABLE]

It can be verified that the bound in [64, (23)] is equivalent to (130). A different approach can be found in the proof of Theorem 9.

Remark 19

By the assumption in (129), it follows from (27) that, for $\alpha\in(-\infty,0)$, the quantity in the exponent in the right side of (130) satisfies

[TABLE]

Hence, by letting $\alpha\to 0$, the bound in (130) is trivial; while by letting $\alpha\to-\infty$, it follows from (23), (129) and (130) that

[TABLE]

The $\alpha\in(-\infty,0)$ that results in the tightest bound in (130) is examined numerically in Example 2, which illustrates the utility of Arimoto-Rényi conditional entropies of negative orders.

Remark 20

For binary hypothesis testing, the lower bound on $\varepsilon_{X|Y}$ in (130) is asymptotically tight by letting $\alpha\to-\infty$ since, in view of (24) and (81),

[TABLE]

Next, we provide a lower bound depending on the Arimoto-Rényi conditional entropy of orders greater than 1. A more general version of this result is given in Theorem 10 by relying on our generalization of Fano’s inequality for list decoding.

Theorem 7

Let $P_{XY}$ be a probability measure defined on $\mathcal{X}\times\mathcal{Y}$ which satisfies (129), with $\mathcal{X}$ being finite or countably infinite. For all $\alpha\in(1,\infty)$

[TABLE]

Proof:

In view of the monotonicity property for positive orders in (28), we obtain that

[TABLE]

and the desired result follows in view of (81). (Note that for $\alpha\in(0,1]$, the right side of (135) is nonpositive.) ∎

Remark 21

The implicit lower bound on $\varepsilon_{X|Y}$ given in (91) (same as (82)) is tighter than the explicit lower bound in (135).

Remark 22

The lower bounds on $\varepsilon_{X|Y}$ in Theorems 3 and 7, which hold for positive orders of $H_{\alpha}(X|Y)$, are both asymptotically tight by letting $\alpha\to\infty$. In contrast, the lower bound on $\varepsilon_{X|Y}$ of Theorem 6, which holds for negative orders of $H_{\alpha}(X|Y)$, is not asymptotically tight by letting $\alpha\to-\infty$ (unless $X$ is a binary random variable).

In the following example, the lower bounds on $\varepsilon_{X|Y}$ in Theorems 3 and 7 are examined numerically in their common range of $\alpha\in(1,\infty)$.

Example 1

Let $X$ and $Y$ be random variables defined on the set $\mathcal{X}=\{1,2,3\}$, and let

[TABLE]

It can be verified that $\varepsilon_{X|Y}=\tfrac{21}{45}\approx 0.4667$. Note that although in this example the bound in (135) is only slightly looser than the bound in (82) for moderate values of $\alpha>1$ (see Table I)222Recall that for $\alpha=2$, (82) admits the explicit expression in (111)., both are indeed asymptotically tight as $\alpha\to\infty$; furthermore, (135) has the advantage of providing a closed-form lower bound on $\varepsilon_{X|Y}$ as a function of $H_{\alpha}(X|Y)$ for $\alpha\in(1,\infty)$.

Example 2

Let $X$ and $Y$ be random variables defined on the set $\mathcal{X}=\{1,2,3,4\}$, and let

[TABLE]

In this case $\varepsilon_{X|Y}=\tfrac{121}{200}=0.6050$, and the tightest lower bound in (130) for $\alpha\in(-\infty,0)$ is equal to 0.4877 (obtained at $\alpha=-2.531$). Although $\varepsilon_{X|Y}$ can be calculated exactly when $H_{\infty}(X|Y)$ is known (see (81)), this example illustrates that conditional Arimoto-Rényi entropies of negative orders are useful in the sense that (130) gives an informative lower bound on $\varepsilon_{X|Y}$ by knowing $H_{\alpha}(X|Y)$ for a negative $\alpha$.

IV-C List decoding

In this section we consider the case where the decision rule outputs a list of choices. The extension of Fano’s inequality to list decoding was initiated in [1, Section 5] (see also [59, Appendix 3.E]). It is useful for proving converse results in conjunction with the blowing-up lemma [20, Lemma 1.5.4]. The main idea of the successful combination of these two tools is that, given an arbitrary code, one can blow-up the decoding sets in such a way that the probability of decoding error can be as small as desired for sufficiently large blocklengths; since the blown-up decoding sets are no longer disjoint, the resulting setup is a list decoder with subexponential list size.

A generalization of Fano’s inequality for list decoding of size $L$ is [84]333See [46, Lemma 1] for a weaker version of (146).

[TABLE]

where $P_{\mathcal{L}}$ denotes the probability of $X$ not being in the list. As we noted before, averaging a conditional version of (48) with respect to the observation is not viable in the case of $H_{\alpha}(X|Y)$ with $\alpha\neq 1$ (see (14)). A pleasing generalization of (146) to the Arimoto-Rényi conditional entropy does indeed hold as the following result shows.

Theorem 8

Let $P_{XY}$ be a probability measure defined on $\mathcal{X}\times\mathcal{Y}$ with $|\mathcal{X}|=M$. Consider a decision rule444$\binom{\mathcal{X}}{L}$ stands for the set of all the subsets of $\mathcal{X}$ with cardinality $L$, with $L\leq|\mathcal{X}|$. $\mathcal{L}\colon\mathcal{Y}\to\binom{\mathcal{X}}{L}$, and denote the decoding error probability by

[TABLE]

Then, for all $\alpha\in(0,1)\cup(1,\infty)$,

[TABLE]

with equality in (148) if and only if

[TABLE]

Proof:

Instead of giving a standalone proof, for brevity, we explain the differences between this and the proof of Theorem 3:

• •

Instead of the conditional version of (67), we use a conditional version of (60) given the observation $Y=y$;

• •

The choices of the arguments of the function $f_{\alpha,\beta,\gamma}$ in (88)–(90) namely, $\beta=(M-1)^{1-\alpha}$ and $\gamma=1$, are replaced by $\beta=(M-L)^{1-\alpha}$ and $\gamma=L^{1-\alpha}$;

• •

(149) follows from (35) with $(\theta,s,t)=(M,M-L,P_{\mathcal{L}})$;

• •

Equality in (148) holds if and only if the condition for equality in the statement of Corollary 1, conditioned on the observation, is satisfied; the latter implies that, given $Y=y$, $X$ is equiprobable on both $\mathcal{L}(y)$ and $\mathcal{L}^{\mathrm{c}}(y)$ for all $y\in\mathcal{Y}$.

∎

Next, we give the fixed list-size generalization of Theorem 6.

Theorem 9

Let $P_{XY}$ be a probability measure defined on $\mathcal{X}\times\mathcal{Y}$ with $|\mathcal{X}|=M<\infty$, which satisfies (129), and let $\mathcal{L}\colon\mathcal{Y}\to\binom{\mathcal{X}}{L}$. Then, for all $\alpha\in(-\infty,0)$, the probability that the decoding list does not include the correct decision satisfies

[TABLE]

Proof:

Let $\xi\colon\mathcal{X}\times\mathcal{Y}\to\{0,1\}$ be given by the indicator function

[TABLE]

Let $u\colon\mathcal{X}\times\mathcal{Y}\to[0,\infty)$, and $\beta>1$. By Hölder’s inequality,

[TABLE]

Therefore,

[TABLE]

Under the assumption in (129), we specialize (155) to

[TABLE]

for all $(x,y)\in\mathcal{X}\times\mathcal{Y}$. From (14), (152) and (156), we have

[TABLE]

and

[TABLE]

Assembling (155), (161) and (165) and substituting $\alpha=\frac{1}{1-\beta}$ results in (151). ∎

The fixed list-size generalization of Theorem 7 is the following.

Theorem 10

Let $P_{XY}$ be a probability measure defined on $\mathcal{X}\times\mathcal{Y}$ which satisfies (129), with $\mathcal{X}$ being finite or countably infinite, and let $\mathcal{L}\colon\mathcal{Y}\to\binom{\mathcal{X}}{L}$. Then, for all $\alpha\in(1,\infty)$,

[TABLE]

Proof:

From (149), for all $\alpha\in(1,\infty)$,

[TABLE]

which gives the bound in (166). ∎

Remark 23

The implicit lower bound on $\varepsilon_{X|Y}$ given by the generalized Fano’s inequality in (148) is tighter than the explicit lower bound in (166) (this can be verified from the proof of Theorem 10).

IV-D Lower bounds on the Arimoto-Rényi conditional entropy

The major existing lower bounds on the Shannon conditional entropy $H(X|Y)$ as a function of the minimum error probability $\varepsilon_{X|Y}$ are:

In view of [39, Theorem 11], (1) (shown in [6, Theorem 1], [15, (12)] and [38, (41)] for finite alphabets) holds for a general discrete random variable $X$. As an example where (1) holds with equality, consider a Z-channel with input and output alphabets $\mathcal{X}=\mathcal{Y}=\{0,1\}$, and assume that P_{X}(0)=1-P_{X}(1)=p\in\bigl{(}0,\tfrac{1}{2}\bigr{]}, and

[TABLE]

It can be verified that $\varepsilon_{X|Y}=p$, and $H(X|Y)=2p$ bits. 2. 2.

Due to Kovalevsky [48], Tebbe and Dwyer [76] (see also [25]) in the finite alphabet case, and to Ho and Verdú [39, (109)] in the general case,

[TABLE]

where $\phi\colon[0,1)\to[0,\infty)$ is the piecewise linear function that is defined on the interval t\in\bigl{[}1-\frac{1}{k},1-\frac{1}{k+1}\bigr{)} as

[TABLE]

where $k$ is an arbitrary positive integer. Note that (170) is tighter than (1) since $\phi(t)\geq 2t\log 2$.

In view of (81), since $H_{\alpha}(X|Y)$ is monotonically decreasing in $\alpha$, one can readily obtain the following counterpart to Theorem 7:

[TABLE]

for $\alpha\in[0,\infty]$ with equality if $\alpha=\infty$.

Remark 24

As a consequence of (172), it follows that if $\alpha\in[0,\infty)$ and $\{X_{n}\}$ is a sequence of discrete random variables, then $H_{\alpha}(X_{n}|Y_{n})\to 0$ implies that $\varepsilon_{X_{n}|Y_{n}}\to 0$, thereby generalizing the case $\alpha=1$ shown in [39, Theorem 14].

Remark 25

Setting $\alpha=2$ in (172) yields [21, Theorem 1], and setting $\alpha=1$ in (172) yields (2). Recently, Prasad [58, Section 2.C] improved the latter bound in terms of the entropy of tilted probability measures, instead of $H_{\alpha}(X|Y)$.

The next result gives a counterpart to Theorem 3, and a generalization of (170).

Theorem 11

If $\alpha\in(0,1)\cup(1,\infty)$, then

[TABLE]

where the piecewise linear function $g_{\alpha}\colon[0,1)\to\mathcal{D}_{\alpha}$, with $\mathcal{D}_{\alpha}=[1,\infty)$ for $\alpha\in(0,1)$ and $\mathcal{D}_{\alpha}=(0,1]$ for $\alpha\in(1,\infty)$, is defined by

[TABLE]

on the interval t\in\bigl{[}1-\frac{1}{k},1-\frac{1}{k+1}\bigr{)} for an arbitrary positive integer $k$.

Proof:

For every $y\in\mathcal{Y}$ such that $P_{Y}(y)>0$, Theorem 2 and (85) yield

[TABLE]

where $s_{\alpha}\colon[0,1)\to[0,\infty)$ is given by

[TABLE]

In the remainder of the proof, we consider the cases $\alpha\in(0,1)$ and $\alpha\in(1,\infty)$ separately.

• •

$\alpha\in(0,1)$. Define the function $f_{\alpha}\colon[0,1)\to[1,\infty)$ as

[TABLE]

The piecewise linear function $g_{\alpha}(\cdot)$ in (174) coincides with the monotonically increasing function $f_{\alpha}(\cdot)$ in (177) at $t_{k}=1-\frac{1}{k}$ for every positive integer $k$ (note that $f_{\alpha}(t_{k})=g_{\alpha}(t_{k})=k^{\frac{1}{\alpha}-1}$). It can be also verified that $g_{\alpha}(\cdot)$ is the lower convex envelope of $f_{\alpha}(\cdot)$ (i.e., $g_{\alpha}(\cdot)$ is the largest convex function for which $g_{\alpha}(t)\leq f_{\alpha}(t)$ for all $t\in[0,1)$). Hence,

[TABLE]

where (178) follows from (16), (175), (177), and since $f_{\alpha}\colon[0,1)\to[1,\infty)$ is a monotonically increasing function for $\alpha\in(0,1)$; (179) follows from $f_{\alpha}\geq g_{\alpha}$; (86) and Jensen’s inequality for the convex function $g_{\alpha}(\cdot)$ result in (180).

• •

$\alpha\in(1,\infty)$. The function $f_{\alpha}(\cdot)$ in (177) is monotonically decreasing in $[0,1)$, and the piecewise linear function $g_{\alpha}(\cdot)$ in (174) coincides with $f_{\alpha}(\cdot)$ at $t_{k}=1-\frac{1}{k}$ for every positive integer $k$. Furthermore, $g_{\alpha}(\cdot)$ is the smallest concave function that is not below $f_{\alpha}(\cdot)$ on the interval $[0,1)$. The proof now proceeds as in the previous case, and (178)–(180) continue to hold for $\alpha>1$.

∎

Remark 26

The most useful domain of applicability of Theorem 11 is $\varepsilon_{X|Y}\in[0,\tfrac{1}{2}]$, in which case the lower bound specializes to ($k=1$)

[TABLE]

which yields (1) as $\alpha\to 1$.

Remark 27

Theorem 11 is indeed a generalization of (170) since for all $\tau\in[0,1]$,

[TABLE]

with $\phi$ defined in (171).

Remark 28

As $\alpha\to\infty$, (173) is asymptotically tight: from (174),

[TABLE]

so the right side of (173) converges to $\log\frac{1}{1-\varepsilon_{X|Y}}$ which, recalling (81), proves the claim.

Remark 29

It can be shown that the function in (174) satisfies

[TABLE]

Moreover, if $t=1-\frac{1}{k}$, for $k\in\{1,2,3,\ldots\}$, then

[TABLE]

Therefore, Theorem 11 gives a tighter bound than (172), unless \varepsilon_{X|Y}\in\bigl{\{}\frac{1}{2},\frac{2}{3},\frac{3}{4},\ldots\frac{M-1}{M}\bigr{\}} ($M$ is allowed to be $\infty$ here) in which case they are identical, and independent of $\alpha$ as it is illustrated in Figure 1.

The following proposition applies when $M$ is finite, but does not depend on $M$. It is supported by Figure 1.

Proposition 3

Let $M\in\{2,3,\ldots\}$ be finite, and let the upper and lower bounds on $H_{\alpha}(X|Y)$ as a function of $\varepsilon_{X|Y}$, as given in Theorems 3 and 11, be denoted by $u_{\alpha,M}(\cdot)$ and $l_{\alpha}(\cdot)$, respectively. Then,

1. a)

these bounds coincide if and only if $X$ is a deterministic function of the observation $Y$ or $X$ is equiprobable on the set $\mathcal{X}$ and independent of $Y$; 2. b)

the limit of the ratio of the upper-to-lower bounds when $\varepsilon_{X|Y}\to 0$ is given by

[TABLE]

Proof:

See Appendix B. ∎

Remark 30

The low error probability limit of the ratio of upper-to-lower bounds in (186) decreases monotonically with $\alpha\in(1,\infty)$ from $\infty$ to $1$.

IV-E Explicit upper bounds on $\varepsilon_{X|Y}$

In this section, we give counterparts to the explicit lower bounds on $\varepsilon_{X|Y}$ given in Section IV-B by capitalizing on the bounds in Section IV-D.

The following result is a consequence of Theorem 11:

Theorem 12

Let $k\in\{1,2,3,\ldots\}$, and $\alpha\in(0,1)\cup(1,\infty)$. If $\log k\leq H_{\alpha}(X|Y)<\log(k+1)$, then

[TABLE]

Furthermore, the upper bound on $\varepsilon_{X|Y}$ as a function of $H_{\alpha}(X|Y)$ is asymptotically tight in the limit where $\alpha\to\infty$.

Proof:

The left side of (173) is equal to $\log k$ when $\varepsilon_{X|Y}=1-\frac{1}{k}$ for $k\in\{1,2,3,\ldots\}$, and it is also monotonically increasing in $\varepsilon_{X|Y}$. Hence, if \varepsilon_{X|Y}\in\bigl{[}1-\frac{1}{k},1-\frac{1}{k+1}\bigr{)}, then the lower bound on $H_{\alpha}(X|Y)$ in the left side of (173) lies in the interval $[\log k,\log(k+1))$. The bound in (187) can be therefore verified to be equivalent to Theorem 11. Finally, the asymptotic tightness of (187) in the limit where $\alpha\to\infty$ is inherited by the same asymptotic tightness of the equivalent bound in Theorem 11. ∎

Remark 31

By letting $\alpha\to 1$ in the right side of (187), we recover the bound by Ho and Verdú [39, (109)]:

[TABLE]

if $\log k\leq H(X|Y)<\log(k+1)$ for an arbitrary $k\in\{1,2,3,\ldots\}$. In the special case of finite alphabets, this result was obtained in [25], [48] and [76]. In the finite alphabet setting, Prasad [58, Section 5] recently refined the bound in (188) by lower bounding $H(X|Y)$ subject to the knowledge of the first two largest posterior probabilities rather than only the largest one; following the same approach, [58, Section 6] gives a refinement of Fano’s inequality.

Example 1 (cont.) Table II compares the asymptotically tight bounds in Theorems 3 and 12 for three values of $\alpha$.

As the following example illustrates, the bounds in (82) and (187) are in general not monotonic in $\alpha$.

Example 3

Suppose $X$ and $Y$ are binary random variables with joint distribution

[TABLE]

In this case $\varepsilon_{X|Y}=0.1944$, and its upper and lower bounds are depicted in Figure 2 as a function of $\alpha$.

V Upper Bounds on $\varepsilon_{X|Y}$ Based on Binary Hypothesis Testing

In this section, we give explicit upper bounds on $\varepsilon_{X|Y}$ by connecting the $M$-ary hypothesis testing problem for finite $M$ with the associated $\binom{M}{2}$ binary hypothesis testing problems. The latter bounds generalize the binary hypothesis testing upper bound on $\varepsilon_{X|Y}$ by Hellman and Raviv [38, Theorem 1] to any number of hypotheses, including the tightening of the upper bound by Kanaya and Han [45], and Leang and Johnson [51].

Hellman and Raviv [38, Theorem 1] generalized the Bhattacharyya bound to give an upper bound on the error probability for binary hypothesis testing in terms of the Rényi divergence between both models $P_{{\mathtt{0}}}$ and $P_{{\mathtt{1}}}$ (with prior probabilities $\mathbb{P}[\mathsf{H}_{{\mathtt{0}}}]$, and $\mathbb{P}[\mathsf{H}_{{\mathtt{1}}}]$ respectively):

[TABLE]

To generalize (192) to any number of hypotheses (including infinity), it is convenient to keep the same notation denoting the models by

[TABLE]

with prior probabilities $\mathbb{P}[\mathsf{H}_{i}]$ for $i\in\{1,\ldots,M\}$, and let

[TABLE]

Theorem 13

[TABLE]

Proof:

Let

[TABLE]

and, for $m\in\{1,\ldots,M\}$, denote the densities

[TABLE]

For $\alpha\in(0,1)$ and for all $i,k\in\{1,\ldots,M\}$ with $i\neq k$,

[TABLE]

where

• •

(200) follows from Bayes’ rule:

[TABLE]

• •

(201) follows from the definition in (195),

• •

(203) follows from $\min\{t,s\}\leq t^{\alpha}s^{1-\alpha}$ if $\alpha\in[0,1]$ and $t,s\geq 0$,

• •

(204) follows from (29).

Minimizing the right side of (204) over all $(i,k)$ with $i\neq k$ gives (196). ∎

We proceed to generalize Theorem 13 along a different direction. We can obtain upper and lower bounds on the Bayesian $M$-ary minimal error probability in terms of the minimal error probabilities of the $\binom{M}{2}$ associated binary hypothesis testing problems. For that end, denote the random variable $X$ restricted to $\{i,j\}\subseteq\mathcal{X}$, $i\neq j$, by $X_{ij}$, namely,

[TABLE]

Guessing the hypothesis without the benefit of observations yields the error probability

[TABLE]

The minimal error probability of the binary hypothesis test

[TABLE]

with the prior probabilities in (206) and (207) is denoted by $\varepsilon_{X_{ij}|Y_{ij}}$, and it is given by

[TABLE]

with

[TABLE]

and

[TABLE]

The error probability achieved by the Bayesian $M$-ary MAP decision rule can be upper bounded in terms of the error probabilities in (212) of the $\tfrac{1}{2}\,M(M-1)$ binary hypothesis tests in (210)–(211), with $1\leq i<j\leq M$, by means of the following result.

Theorem 14

[TABLE]

where $Y\sim P_{Y}$ with $P_{Y}$ in (197), and $Y_{ij}\sim P_{Y_{ij}}$ in (213).

Proof:

Every probability mass function $(q_{1},\ldots,q_{M})$ satisfies

[TABLE]

because, due to symmetry, if without any loss of generality $q_{1}$ attains the maximum in the left side of (217), then the partial sum in the right side of (217) over the $M-1$ terms that involve $q_{1}$ is equal $q_{2}+\ldots+q_{M}=1-q_{1}$ which is equal to the left side of (217). Hence,

[TABLE]

where (220) follows from (217) with $q_{i}\leftarrow\mathbb{P}[\mathsf{H}_{i}|Y=y]$ for $i\in\{1,\ldots,M\}$ and $y\in\mathcal{Y}$. This proves (215).

To show (216), note that for all $i,j\in\{1,\ldots,M\}$ with $i\neq j$,

[TABLE]

where (222) and (223) hold due to (212) and (214), respectively; (224) follows from (206) and (207); (225) follows from the equality $\mathbb{P}[\mathsf{H}_{i}|Y=y]=\mathbb{P}[\mathsf{H}_{i}]\,\frac{\mathrm{d}P_{i}}{\mathrm{d}P_{Y}}(Y=y)$. Finally, (216) follows from (215) and (222)–(225). ∎

The following immediate consequence of Theorem 14 is the bound obtained in [51, (3)].

Corollary 4

[TABLE]

We proceed to give upper bounds on $\varepsilon_{X|Y}$ based on Rényi divergence and Chernoff information.

Theorem 15

[TABLE]

Proof:

Consider the binary hypothesis test in (210)–(211) with its minimal error probability $\varepsilon_{X_{ij}|Y_{ij}}$. Substituting the prior probabilities in (206)–(207) into the Hellman-Raviv bound (192), we obtain

[TABLE]

which, upon substitution in (216), yields (227). To show (228), note that for all $t,s\geq 0$ and $\alpha\in(0,1)$

[TABLE]

Hence, we get

[TABLE]

where (232) holds due to (227) and (231) with $t\leftarrow\mathbb{P}[\mathsf{H}_{i}]$ and $s\leftarrow\mathbb{P}[\mathsf{H}_{j}]$; (233) holds by the definition in (41) with $P\leftarrow P_{i}$ and $Q\leftarrow P_{j}$. The sum in the right side of (234) satisfies

[TABLE]

The result in (228) follows from (232)–(237). ∎

Remark 32

The bound on $\varepsilon_{X|Y}$ in (228) can be improved by relying on the middle term in (231), which then gives that for $t,s\geq 0$ and $\alpha\in(0,1)$

[TABLE]

Following the analysis in (232)–(237), we get

[TABLE]

which yields an improvement over (228) by a factor of at most $\tfrac{1}{2}$, attained when $X$ is equiprobable.

Remark 33

Using the identity

[TABLE]

for all $\alpha\in(0,1)$ and probability measures $P$ and $Q$, it is easy to check that Theorems 13 and 15 result in the same bound for binary hypothesis testing.

Remark 34

Kanaya and Han [45] and Leang and Johnson [51] give a weaker bound with $\frac{1}{2}M(M-1)$ in lieu of $M-1$ in (228).

Remark 35

In the absence of observations, the bound in (196) is tight since $\varepsilon_{X|Y}=1-\max_{i}\mathbb{P}[\mathsf{H}_{i}].$ On the other hand, (226) and (227) need not be tight in that case, and in fact may be strictly larger than 1.

Numerical experimentation with Examples 1–3 and others shows that, in general, the Arimoto-Rényi conditional entropy bounds are tighter than (196) and (227).

VI Arimoto-Rényi Conditional Entropy Averaged over Codebook Ensembles

In this section we consider the channel coding setup with a code ensemble $\mathcal{C}$, over which we are interested in averaging the Arimoto-Rényi conditional entropy of the channel input given the channel output. We denote such averaged quantity by \mathbb{E}_{\mathcal{C}}\bigl{[}H_{\alpha}(X^{n}|Y^{n})\bigr{]} where $X^{n}=(X_{1},\ldots,X_{n})$ and $Y^{n}=(Y_{1},\ldots,Y_{n})$. Some motivation for this study arises from the fact that the normalized equivocation $\frac{1}{n}H(X^{n}|Y^{n})$ as a reliability measure was used by Shannon [69] in his proof that reliable communication is impossible at rates above channel capacity; furthermore, the asymptotic convergence to zero of the equivocation $H(X^{n}|Y^{n})$ at rates below capacity was studied by Feinstein [26, Section 3].

We can capitalize on the convexity of $d_{\alpha}(\cdot\|q)$ for $\alpha\in[0,1]$ (see [23, Theorem 11] and [23, Section 6.A]) to claim that in view of Theorem 3, for any ensemble $\mathcal{C}$ of size-$M$ codes and blocklength $n$,

[TABLE]

where $\alpha\in[0,1]$ and $\mkern 1.5mu\overline{\mkern-1.5mu\varepsilon\mkern-1.5mu}\mkern 1.5mu$ is the error probability of the maximum-likelihood decoder averaged over the code ensemble, or an upper bound thereof which does not exceed $1-\frac{1}{M}$ (for such upper bounds see, e.g., [66, Chapters 1–4]). Analogously, in view of Theorem 8 for list decoding with a fixed list size $L$, the same convexity argument implies that

[TABLE]

where $\mkern 1.5mu\overline{\mkern-1.5muP_{\mathcal{L}}\mkern-1.5mu}\mkern 1.5mu$ denotes the minimum list decoding error probability averaged over the ensemble, or an upper bound thereof which does not exceed $1-\frac{L}{M}$ (for such bounds see, e.g., [28], [30, Problem 5.20], [42, Section 5] and [53]).

In the remainder of this section, we consider the discrete memoryless channel (DMC) model

[TABLE]

an ensemble $\mathcal{C}$ of size-$M$ codes and blocklength $n$ such that the $M$ messages are assigned independent codewords, drawn i.i.d. with per-letter distribution $P_{X}$ on the input alphabet

[TABLE]

In this setting, Feder and Merhav show in [25] the following result for the Shannon conditional entropy.

Theorem 16

[25*, Theorem 3]**

For a DMC with transition probability matrix $P_{Y|X}$, the conditional entropy of the transmitted codeword given the channel output, averaged over a random coding selection with per-letter distribution $P_{X}$ such that $I(P_{X},P_{Y|X})>0$, is bounded by*

[TABLE]

with

[TABLE]

where $E_{\mathrm{r}}$ in the right side of (246) denotes the random-coding error exponent, given by (recall (44))

[TABLE]

and the argument that maximizes (248) is denoted in (246) by $\rho^{\ast}(R,P_{X})$.

Remark 36

Since $H_{\alpha}(X^{n}|Y^{n})$ is monotonically decreasing in $\alpha$ (see Proposition 1), the upper bounds in (245)–(246) also apply to \mathbb{E}_{\mathcal{C}}\bigl{[}H_{\alpha}(X^{n}|Y^{n})\bigr{]} for $\alpha\in[1,\infty]$.

We are now ready to state and prove the main result in this section.

Theorem 17

The following results hold under the setting in Theorem 16:

For all $\alpha>0$, and rates $R$ below the channel capacity $C$,

[TABLE]

where $E_{\text{sp}}(\cdot)$ denotes the sphere-packing error exponent

[TABLE]

with the maximization in the right side of (250) over all single-letter distributions $Q_{X}$ defined on the input alphabet. 2. 2.

For all $\alpha\in(0,1)$,

[TABLE]

provided that

[TABLE]

where $R_{\alpha}(P_{X},P_{Y|X})$ is the unique solution $r\in(0,I(P_{X},P_{Y|X}))$ to

[TABLE] 3. 3.

The rate $R_{\alpha}(P_{X},P_{Y|X})$ is monotonically increasing and continuous in $\alpha\in(0,1)$, and

[TABLE]

Proof:

In proving the first two items we actually give an asymptotic lower bound and a non-asymptotic upper bound, which yield the respective results upon taking $\underset{n\to\infty}{\lim\sup}-\frac{1}{n}\log(\cdot)$ and $\underset{n\to\infty}{\lim\inf}-\frac{1}{n}\log(\cdot)$, respectively.

From (181) and the sphere-packing lower bound [72, Theorem 2] on the decoding error probability, the following inequality holds for all rates $R\in[0,C)$ and $\alpha\in(0,1)\cup(1,\infty)$,

[TABLE]

where $o_{1}(n)$ and $o_{2}(n)$ tend to zero as $n\to\infty$. Note that $\eqref{eq: Barcelona}$ holds when the error probability of the optimal code lies in $[0,\tfrac{1}{2}]$, but this certainly holds for all $R<C$ and sufficiently large $n$ (since this error probability decays exponentially to zero). 2. 2.

Note that $E_{\mathrm{r}}(R,P_{X})>0$ since, for $\alpha\in(0,1)$,

[TABLE]

where the left inequality in (257) is the condition in (252), and the right inequality in (257) holds by the definition of $R_{\alpha}(P_{X},P_{Y|X})$ in (253) (see its equivalent form in (264)). In view of the random coding bound [29], let

[TABLE]

be an upper bound on the decoding error probability of the code ensemble $\mathcal{C}$, which does not exceed $1-\frac{1}{M}$ with $M=\exp(nR)$. From (241), for all $\alpha\in(0,1)$,

[TABLE]

where equality (260) follows from (35) by setting $(\theta,s,t)=(\exp(nR),\,\exp(nR)-1,\,\mkern 1.5mu\overline{\mkern-1.5mu\varepsilon\mkern-1.5mu}\mkern 1.5mu_{n})$; the inequality in (262) holds with equality for all sufficiently large $n$ such that \mkern 1.5mu\overline{\mkern-1.5mu\varepsilon\mkern-1.5mu}\mkern 1.5mu_{n}=\exp\bigl{(}-nE_{\mathrm{r}}(R,P_{X})\bigr{)} (see (258)). The result in (251) follows consequently by taking $\underset{n\to\infty}{\lim\inf}-\frac{1}{n}\log(\cdot)$ of (259)–(262). To that end, note that the second exponent in the right side of (262) decays to zero whenever

[TABLE]

It can be verified that the representation of $R_{\alpha}(P_{X},P_{Y|X})$ in (264) is indeed equivalent to the way it is defined in (253) because $E_{\mathrm{r}}(r,P_{X})$ is monotonically decreasing and continuous in $r$ (see [30, (5.6.32)]), the right side of (253) is strictly monotonically increasing and continuous in $r$ for $\alpha\in(0,1)$, and the left and right sides of (253) vanish, respectively, at $r=I(P_{X},P_{Y|X})$ and $r=0$. The equivalent representation of $R_{\alpha}(P_{X},P_{Y|X})$ in (264) yields the uniqueness of the solution $r=R_{\alpha}(P_{X},P_{Y|X})$ of (253) in the interval $(0,I(P_{X},P_{Y|X}))$ for all $\alpha\in(0,1)$, and it justifies in particular the right inequality in (257). 3. 3.

From (253), for $\alpha\in(0,1)$, $r=R_{\alpha}(P_{X},P_{Y|X})$ is the unique solution $r\in(0,I(P_{X},P_{Y|X}))$ to

[TABLE]

Since $E_{\mathrm{r}}(r,P_{X})$ is positive and monotonically decreasing in $r$ on the interval $(0,I(P_{X},P_{Y|X}))$, so is the left side of (265) as a function of $r$. Since also the right side of (265) is positive and monotonically decreasing in $\alpha\in(0,1)$, it follows from (265) that $r=R_{\alpha}(P_{X},P_{Y|X})$ is monotonically increasing in $\alpha$. The continuity of $R_{\alpha}(P_{X},P_{Y|X})$ in $\alpha$ on the interval $(0,1)$ holds due to (265) and the continuity of $E_{\mathrm{r}}(\cdot,P_{X})$ (see [30, p. 143]). The limits in (254) and (255) follow from (253): (254) holds since we have $\underset{r\downarrow 0}{\lim}\,E_{\mathrm{r}}(r,P_{X})=E_{\mathrm{r}}(0,P_{X})\in(0,\infty)$, whereas $\underset{\alpha\downarrow 0}{\lim}\left(\frac{1}{\alpha}-1\right)=+\infty$; (255) holds since $E_{\mathrm{r}}(R,P_{X})$ is equal to zero at $R=I(P_{X},P_{Y|X})$, and it is positive if $R<I(P_{X},P_{Y|X})$ (see [30, p. 142]).

∎

Remark 37

For $\alpha=1$, Theorem 17 strengthens the result in Theorem 16 by also giving the converse in Item 1). This result enables to conclude that if $P_{X}$ is the input distribution which maximizes the random-coding error exponent

[TABLE]

then

[TABLE]

at all rates between the critical rate $R_{\mathrm{c}}$ and channel capacity $C$ of the DMC, since for all $R\in[R_{\mathrm{c}},C]$ [30, Section 5.8]

[TABLE]

where $E_{\text{sp}}(\cdot)$ is given in (250).

Remark 38

With respect to (268), recall that although the random-coding error exponent is tight for the average code [31], it coincides with the sphere-packing error exponent (for the optimal code) at the high-rate region between the critical rate and channel capacity. As shown by Gallager in [31], the fact that the random coding error exponent is not tight for optimal codes at rates below the critical rate of the DMC stems from the poor performance of the bad codes in this ensemble rather than a weakness of the bounding technique in [29].

Remark 39

The result in Theorem 17 can be extended to list decoding with a fixed size $L$ by relying on (242), and the upper bound on the list decoding error probability in [30, Problem 5.20] where

[TABLE]

with $M=\exp(nR)$, and

[TABLE]

This result can be further generalized to structured code ensembles by relying on upper bounds on the list decoding error probability in [42, Section 5].

The following result is obtained by applying Theorem 17 in the setup of communication over a memoryless binary-input output-symmetric channel with a symmetric input distribution. The following rates, specialized to the considered setup, are required for the presentation of our next result.

The cutoff rate is given by

[TABLE]

where in the right side of (271), $E_{0}(\rho,P_{X}^{\ast})$ is given in (45) with the symmetric binary input distribution P_{X}^{\ast}=\bigl{[}\tfrac{1}{2}~{}\tfrac{1}{2}\bigr{]}, and $\rho_{Y|X}\in[0,1]$ in the right side of (272) denotes the Bhattacharyya constant given by

[TABLE]

with an integral replacing the sum in the right side of (273) if $\mathcal{Y}$ is a non-discrete set. 2. 2)

The critical rate is given by

[TABLE]

where the differentiation of $E_{0}$ in the right side of (274) is with respect to $\rho$ at $\rho=1$. 3. 3)

The channel capacity is given by

[TABLE]

In general, the random coding error exponent is given by (see [30, Section 5.8])

[TABLE]

and

[TABLE]

Consequently, $E_{\mathrm{r}}(R)$ in (276) is composed of two parts: a straight line at rates $R\in[0,R_{\mathrm{c}}]$, starting from $R_{{\mathtt{0}}}$ at zero rate with slope $-1$, and the sphere-packing error exponent at rates $R\in[R_{\mathrm{c}},C]$.

Theorem 18

Let $P_{Y|X}$ be the transition probability matrix of a memoryless binary-input output-symmetric channel, and let P_{X}^{\ast}=\bigl{[}\tfrac{1}{2}~{}\tfrac{1}{2}\bigr{]}. Let $R_{\mathrm{c}}$, $R_{{\mathtt{0}}}$, and $C$ denote the critical and cutoff rates and the channel capacity, respectively, and let555In general $\alpha_{\mathrm{c}}\in[0,1]$ (see (277)); the cases $\alpha_{\mathrm{c}}=0$ and $\alpha_{\mathrm{c}}=1$ imply that Theorem 18-a) or b) are not applicable, respectively.

[TABLE]

The rate $R_{\alpha}=R_{\alpha}(P_{X}^{\ast},P_{Y|X})$, as introduced in Theorem 17–2) with the symmetric input distribution $P_{X}^{\ast}$, can be expressed as follows:

1. a)

for $\alpha\in(0,\alpha_{\mathrm{c}}]$,

[TABLE] 2. b)

for $\alpha\in(\alpha_{\mathrm{c}},1)$, $R_{\alpha}\in(R_{\mathrm{c}},C)$ is the solution to

[TABLE]

$R_{\alpha}$* is continuous and monotonically increasing in $\alpha\in[\alpha_{\mathrm{c}},1)$ from $R_{\mathrm{c}}$ to $C$.*

Proof:

For every memoryless binary-input output-symmetric channel, the symmetric input distribution $P_{X}^{\ast}$ achieves the maximum of the error exponent $E_{\mathrm{r}}(R)$ in the right side of (266). Consequently, (279) and (280) readily follow from (253), (268), (276) and (278). Due to Theorem 17-3), $R_{\alpha}$ is monotonically increasing and continuous in $\alpha\in(0,1)$. From (278) and (279)

[TABLE]

and (255) and (275) yield

[TABLE]

Consequently, $R_{\alpha}$ is monotonically increasing in $\alpha\in[\alpha_{\mathrm{c}},1)$ from $R_{\mathrm{c}}$ to $C$. ∎

Example 4

Let $P_{Y|X}(0|0)=P_{Y|X}(1|1)=1-\delta$, $P_{Y|X}(0|1)=P_{Y|X}(1|0)=\delta$, and $P_{X}(0)=P_{X}(1)=\tfrac{1}{2}$. In this case, it is convenient to express all rates in bits; in particular, it follows from (271)–(275) that the cutoff rate, critical rate and channel capacity are given, respectively, by (see, e.g., [30, p. 146])

[TABLE]

The sphere-packing error exponent for the binary symmetric channel is given by (see, e.g., [30, (5.8.26)–(5.8.27)])

[TABLE]

where the normalized Gilbert-Varshamov distance (see, e.g., [63, Theorem 4.10]) is denoted by

[TABLE]

with h^{-1}\colon[0,\log 2]\to\bigl{[}0,\tfrac{1}{2}\bigr{]} standing for the inverse of the binary entropy function.

In view of Theorem 18, Figure 3 shows $R_{\alpha}$ with $\alpha\in(0,1)$ for the case where the binary symmetric channel has capacity $\frac{1}{2}$ bit per channel use, namely, \delta=h^{-1}\bigl{(}\tfrac{1}{2}\bigr{)}=0.110, in which case $\alpha_{\mathrm{c}}=0.5791$ and $R_{\alpha_{\mathrm{c}}}=R_{\mathrm{c}}=0.1731$ bits per channel use (see (284)).

VII Conclusions

The interplay between information-theoretic measures and the analysis of hypothesis testing has been a fruitful area of research. Along these lines, we have shown new bounds on the minimum Bayesian error probability $\varepsilon_{X|Y}$ of arbitrary $M$-ary hypothesis testing problems as a function of information measures. In particular, our major focus has been the Arimoto-Rényi conditional entropy of the hypothesis index given the observation. We have seen how changing the conventional form of Fano’s inequality from

[TABLE]

to the right side of (289), where $d(\cdot\|\cdot)$ is the binary relative entropy, allows a natural generalization where the Arimoto-Rényi conditional entropy of an arbitrary positive order $\alpha$ is upper bounded by

[TABLE]

with $d_{\alpha}(\cdot\|\cdot)$ denoting the binary Rényi divergence.

Likewise, thanks to the Schur-concavity of Rényi entropy, we obtain a lower bound on $H_{\alpha}(X|Y)$ in terms of $\varepsilon_{X|Y}$, which holds even if $M=\infty$. Again, we are able to recover existing bounds by letting $\alpha\to 1$.

In addition to the aforementioned bounds on $H_{\alpha}(X|Y)$ as a function of $\varepsilon_{X|Y}$, it is also of interest to give explicit lower and upper bounds on $\varepsilon_{X|Y}$ as a function of $H_{\alpha}(X|Y)$. As $\alpha\to\infty$, these bounds converge to $\varepsilon_{X|Y}$.

The list decoding setting, in which the hypothesis tester is allowed to output a subset of given cardinality and an error occurs if the true hypothesis is not in the list, has considerable interest in information theory. We have shown that our techniques readily generalize to that setting and we have found generalizations of all the $H_{\alpha}(X|Y)$–$\varepsilon_{X|Y}$ bounds to the list decoding setting.

We have also explored some facets of the role of binary hypothesis testing in analyzing $M$-ary hypothesis testing problems, and have shown new bounds in terms of Rényi divergence.

As an illustration of the application of the $H_{\alpha}(X|Y)$–$\varepsilon_{X|Y}$ bounds, we have analyzed the exponentially vanishing decay of the Arimoto-Rényi conditional entropy of the transmitted codeword given the channel output for discrete memoryless channels and random coding ensembles.

Appendix A Proof of Proposition 1

We prove that $H_{\alpha}(X|Y)\leq H_{\beta}(X|Y)$ in three different cases:

• •

$1<\beta<\alpha$

• •

$0<\beta<\alpha<1$

• •

$\beta<\alpha<0$.

Proposition 1 then follows by transitivity, and the continuous extension of $H_{\alpha}(X|Y)$ at $\alpha=0$ and $\alpha=1$. The following notation is handy.

[TABLE]

Case 1: If $1<\beta<\alpha$, then $(\rho,\theta)\in(0,1)^{2}$, and

[TABLE]

where the outer expectations are with respect to $Y\sim P_{Y}$; (293) follows from (14) and the equality $\frac{\beta\rho}{\beta-1}=\frac{\alpha}{\alpha-1}$ (see (291) and (292)); (294) follows from $\tfrac{\beta}{\beta-1}>0$, $\tfrac{1}{\beta}>0$, Jensen’s inequality and the concavity of $t^{\rho}$ on $[0,\infty)$; (295) follows from (292) and (291); (296) holds due to the concavity of $t^{\theta}$ on $[0,\infty)$ and Jensen’s inequality; (297) follows from (291); (298) follows from (14).

Case 2: If $0<\beta<\alpha<1$, then $(\rho,\theta)\in(1,\infty)^{2}$, and both $t^{\rho}$ and $t^{\theta}$ are convex. However, (294) and (296) continue to hold since now $\tfrac{\beta}{\beta-1}<0$.

Case 3: If $\beta<\alpha<0$, then $\rho\in(0,1)$ and $\theta\in(1,\infty)$. Inequality (294) holds since $t^{\rho}$ is concave on $[0,\infty)$ and $\tfrac{\beta}{\beta-1}>0$; furthermore, although $t^{\theta}$ is now convex on $[0,\infty)$, (296) also holds since $\tfrac{1}{\beta}<0$.

Appendix B Proof of Proposition 3

Proof of Proposition 3a): We need to show that the lower bound on $H_{\alpha}(X|Y)$ in (173) coincides with its upper bound in (82) if and only if $\varepsilon_{X|Y}=0$ or $\varepsilon_{X|Y}=1-\frac{1}{M}$. This corresponds, respectively, to the cases where $X$ is a deterministic function of the observation $Y$ or $X$ is equiprobable on the set $\mathcal{X}$ and independent of $Y$. In view of (35), (82) and (173), and by the use of the parameter t=\varepsilon_{X|Y}\in\bigl{[}0,1-\frac{1}{M}\bigr{]}, this claim can be verified by proving that

• •

if $\alpha\in(0,1)$ and $k\in\{1,\ldots,M-1\}$, then

[TABLE]

for all $t\in[1-\frac{1}{k},1-\frac{1}{k+1})$ with the point $t=0$ excluded if $k=1$;

• •

the opposite inequality in (299) holds if $\alpha\in(1,\infty)$.

Let the function v_{\alpha}\colon\bigl{[}0,1-\frac{1}{M}\bigr{]}\to\mathbb{R} be defined such that for all $k\in\{1,\ldots,M-1\}$

[TABLE]

where v_{\alpha,k}\colon\bigl{[}1-\tfrac{1}{k},1-\tfrac{1}{k+1}\bigr{)}\to\mathbb{R} is given by

[TABLE]

Note that $v_{\alpha,k}(t)$ is the difference between the left and right sides of (299). Moreover, let $v_{\alpha}(\cdot)$ be continuously extended at $t=1-\frac{1}{M}$; it can be verified that $v_{\alpha}(0)=v_{\alpha}\left(1-\frac{1}{M}\right)=0$. We need to prove that for all $\alpha\in(0,1)$

[TABLE]

together with the opposite inequality in (302) for $\alpha\in(1,\infty)$. To that end, we prove that

1. a)

for $\alpha\in(0,1)\cup(1,\infty)$, $v_{\alpha}(\cdot)$ is continuous at all points

[TABLE]

To show continuity, note that the right continuity at $t_{0}=0$ and the left continuity at $t_{M-1}=1-\tfrac{1}{M}$ follow from (300); for $k\in\{1,\ldots,M-2\}$, the left continuity of $v_{\alpha}(\cdot)$ at $t_{k}$ is demonstrated by showing that

[TABLE]

and its right continuity at $t_{k}$ is trivial from (300), (301) and (303); 2. b)

for all $k\in\{0,1,\ldots,M-1\}$

[TABLE]

which implies that $v_{\alpha}(\cdot)$ is zero at the endpoints of the interval \bigl{[}0,\frac{M-1}{M}\bigr{]}, and if $M\geq 3$

[TABLE]

for all $k\in\{1,\ldots,M-2\}$; 3. c)

the following convexity results hold:

• •

for $\alpha\in(0,1)$, the function $v_{\alpha}(\cdot)$ is strictly concave on $[t_{k},t_{k+1}]$ for all $k\in\{0,\ldots,M-1\}$;

• •

for $\alpha\in(1,\infty)$, the function $v_{\alpha}(\cdot)$ is strictly convex on $[t_{k},t_{k+1}]$ for all $k\in\{0,\ldots,M-1\}$.

These properties hold since, due to the linearity in $t$ of the right side of (299), $v_{\alpha}(\cdot)$ is convex or concave on \bigl{[}t_{k},t_{k+1}] for all $k\in\{0,1,\ldots,M-1\}$ if and only if the left side of (299) is, respectively, a convex or concave function in $t$; due to Lemma 1, the left side of (299) is strictly concave as a function of $t$ if $\alpha\in(0,1)$ and it is strictly convex if $\alpha\in(1,\infty)$.

Due to Items a)–c), (302) holds for $\alpha\in(0,1)$ and the opposite inequality holds for $\alpha\in(1,\infty)$. In order to prove (302) for $\alpha\in(0,1)$, note that $\left[0,1-\tfrac{1}{M}\right]=\bigcup_{i=1}^{M-1}\left[t_{i-1},t_{i}\right]$ where $v_{\alpha}(t_{0})=v_{\alpha}(t_{M-1})=0$ and $v_{\alpha}(t_{k})>0$ for every $k\in\{1,\ldots,M-2\}$, $v_{\alpha}(\cdot)$ is continuous at all points $t_{k}$ for $k\in\{0,\ldots,M-1\}$, and $v_{\alpha}$ is strictly concave on $[t_{k},t_{k+1}]$ for all $k\in\{0,\ldots,M-2\}$, yielding the positivity of $v_{\alpha}(\cdot)$ on the interval $(0,1-\tfrac{1}{M})$. The justification for the opposite inequality of (302) if $\alpha\in(1,\infty)$ is similar, yielding the negativity of $v_{\alpha}(\cdot)$ on $(0,1-\tfrac{1}{M})$.

Proof of Proposition 3b): In view of (35), (82) and (181), for all $\varepsilon_{X|Y}\in[0,\tfrac{1}{2})$ and $\alpha\in(0,1)\cup(1,\infty)$:

[TABLE]

Equality (186) follows by using L’Hôpital’s rule for the calculation of the limit in the left side of (186).

Acknowledgment

The authors gratefully acknowledge the detailed reading of the manuscript by the Associate Editor and anonymous referees.