Almost exact analysis of soft covering lemma via large deviation

TL;DR

This paper precisely characterizes the expected deviation in the soft covering lemma using large deviation techniques, providing new insights and methods applicable to all memoryless channels.

Contribution

It introduces a novel proof approach for the exact order of deviation in the soft covering lemma, extending results to general memoryless channels.

Findings

Exact order of expected deviation determined

New inequalities for independent variables derived

Duality between channel coding and soft covering established

Abstract

This paper investigates the soft covering lemma under both the relative entropy and the total variation distance as the measures of deviation. The exact order of the expected deviation of the random i.i.d. code for the soft covering problem problem, is determined. The proof technique used in this paper significantly differs from the previous techniques for deriving exact exponent of the soft covering lemma. The achievability of the exact order follows from applying the change of measure trick (which has been broadly used in the large deviation) to the known one-shot bounds in the literature. For the ensemble converse, some new inequalities of independent interest derived and then the change of measure trick is applied again. The exact order of the total variation distance is similar to the exact order of the error probability, thus it adds another duality between the channel coding and soft covering. Finally, The results of this paper are valid for any memoryless channels, not only channels with finite alphabets.

Equations

$${ı}_{P\Vert Q}(x)=\log \frac{\mathrm{d}P}{\mathrm{d}Q}(x).$$

$$D(P||Q)$$

$$\Vert P-Q\Vert$$

$$I_{\alpha }(X;Y)=\frac{\alpha }{\alpha -1}\log \mathbb{E}[{\mathbb{E}}^{\frac{1}{\alpha }}[\exp (\alpha {ı}_{X;Y}(X;\tilde{Y}))|\tilde{Y}]],$$

$${\mathsf{P}}_Y(.):=\frac{1}{\mathsf{M}}\sum _{k=1}^{\mathsf{M}}{P}_{Y|X}(.|X(k)).$$

$$\mathbb{E}[\Vert {\mathsf{P}}_Y-P_Y\Vert ]$$

$$I_{\alpha }^c(X;Y)=\underset{Q_Y}{\inf }\mathbb{E}[D_{\alpha }(P_{Y|X=X^{\prime }}||Q_Y)]$$

$$\mathbb{E}[\Vert {\mathsf{P}}_Y-P_Y\Vert ]\le \mathbb{P}[{\mathcal{F}}^c]$$

$$+\frac{1}{2}\mathbb{E}\left[\sqrt{{\mathsf{M}}^{-1}\mathbb{E}[\exp (ı(X;Y))\text{mathbbm}1\{(X,Y)\in \mathcal{F}\}|Y]}\right]$$

$$\displaystyle\mathcal{F}:=\Big{\{}(x,y):\exp(\imath(x;y))$$

$$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\leq\left(\mathsf{M}\mathbb{E}[\exp(\lambda\imath(X;Y))|Y=y]\right)^{\frac{1}{1+\lambda}}\triangleq\kappa_{\lambda}\Big{\}}$$

$$\mathbb{P}[{\mathcal{F}}^c|Y=y]\le {\mathsf{M}}^{-\frac{\lambda }{1+\lambda }}{\mathbb{E}}^{\frac{1}{1+\lambda }}[\exp (\lambda ı(X;Y))|Y=y]$$

$$\sqrt{{\mathsf{M}}^{-1}\mathbb{E}[\exp (ı(X;Y))|Y=y]\text{mathbbm}1\{(X,Y)\in \mathcal{F}\}}$$

$$\le \sqrt{{\mathsf{M}}^{-1}{\kappa }_{\lambda }^{1-\lambda }\mathbb{E}[\exp (\lambda ı(X;Y))|Y=y]}$$

$$={\mathsf{M}}^{-\frac{\lambda }{1+\lambda }}{\mathbb{E}}^{\frac{1}{1+\lambda }}[\exp (\lambda ı(X;Y))|Y=y]$$

$$\mathbb{E}[\exp (\lambda ı(X;Y))|Y=y]=\mathbb{E}[\exp ((1+\lambda )ı(X;y))]$$

$${\tau }^{\ast }=\arg \underset{0\le \tau \le 1}{\max }\tau R-\log \mathbb{E}[\exp (\tau {ı}_{X;Y}(X;Y))].$$

$$\mathbb{E}\left[D({\mathsf{P}}_{Y^n}||P_Y^{\otimes n})\right]$$

$$\displaystyle=\left\{\begin{array}[]{ll}\Theta\left(\frac{\exp(-n\tau^{*}R)}{\sqrt{n}}\mathbb{E}^{n}[\exp(\tau^{*}\imath(X;Y))]\right)&\tau^{*}<1\\ \Theta\left({\exp(-nR)}\mathbb{E}^{n}[\exp(\imath(X;Y))]\right)&\tau^{*}=1\end{array}\right.$$

$${\rho }^{\ast }=\arg$$

$$\mathbb{E}\left[\Vert {\mathsf{P}}_{Y^n}-P_Y^{\otimes n}\Vert \right]$$

$$\displaystyle=\left\{\begin{array}[]{ll}\Theta\Big{(}{{n^{-\frac{\beta^{*}}{2}}}}\exp(-{n\rho^{*}}(R-I_{\frac{1}{1-\rho^{*}}}(X;Y))\Big{)}&\rho^{*}<\frac{1}{2}\\ \Theta\left(\exp(-\frac{n}{2}(R-I_{2}(X;Y))\right)&\rho^{*}=\frac{1}{2}\end{array}\right.$$

$${\mathsf{L}}_m=\mathbb{E}\left[D\left({\mathsf{P}}_{\mathbf{Y}}^{(m)}||P_Y^{\otimes n}\right)\right]$$

$${\mathsf{V}}_m=\mathbb{E}\left[\Vert {\mathsf{P}}_{\mathbf{Y}}^{(m)}-P_Y^{\otimes n}\Vert \right]$$

$${\mathsf{L}}_{\exp (nR)}$$

$${\mathsf{V}}_{\exp (nR)}$$

$$\mathbb{E}\left[{\mathsf{L}}_M\right]$$

$$\le nI\left(X;Y\right){\epsilon }_{\frac{1}{2}}^{{\mu }_n}+{\mathsf{L}}_{\exp \left(nR\right)}.$$

$$\mathbb{E}\left[{\mathsf{V}}_M\right]$$

$$\le {\epsilon }_{\frac{1}{2}}^{{\mu }_n}+{\mathsf{V}}_{\exp \left(nR\right)}.$$

$$T=\frac{1}{{\mu }_n}\sum _{k=1}^M\exp \left(ı\left(\mathbf{X}(k);\mathbf{Y}\right)\right).$$

$$\mathbb{E}\left[{\mathsf{L}}_M\right]$$

$$=\frac{1}{2}\mathbb{E}\left[T\log T\right]-O\left(\exp (-nR)\right)$$

$$\mathbb{E}\left[{\mathsf{V}}_M\right]$$

$$\displaystyle=\frac{1}{4}\mathbb{E}\left[\Big{|}T-1\Big{|}\right]-O\left(\exp(-n\frac{R}{2})\right).$$

$${\mathsf{L}}_{\exp (nR)}$$

$${\mathsf{V}}_{\exp (nR)}$$

Full text

I Introduction

There exist two commonly used different (but dual) approaches for the investigating of the error exponent in the literature, which are closely related to the approaches used in the theory of large deviation [1, 2];

The first one is based on the method of types. This approach is closely related to Sanov’s approach111The general Sanov’s approach is applicable to any random variable. Here by Sanov, we mean the approach used for discrete random variables which is also based on the method of types. in the large deviation (LD) theory. A comprehensive exposition of this approach for the basic problems can be found in [3]. 2. 2.

Gallager’s approach [4], which is similar in its spirit to the Cràmer approach in the LD.

While Sanov’s method is more general than the Cramer’s one, the latter has the advantage of being strong enough to find the exact order of the desired probability (such as the probability of deviating from the zero of the sum of independent r.v.’s). This observation was made by Bahadur and Rao, see [1]. In the same way, the exact order of random coding bound has been recently derived in [5, 6], using an approach related to Bahadur-Rao.

Soft covering lemma (also known as channel resolvability) [7, 8, 9] is another basic problem which has many applications such as secrecy problems, simulation of channels, etc. Further, it is somehow the dual to the channel coding problem. Recently, the exact exponent of the soft covering lemma under different measures of closeness has been derived in [10, 11, 12]. Although the techniques used in these papers are different, all are based on the method of types and thus the results are limited to channels with finite alphabets. This motivates us to investigate other techniques similar to those used by Gallager and Cramer. Fortunately, such approach gives us a new proof which is not only a different proof of the exact exponent that is valid for any channel with some regularity condition, but leads to the exact order of the soft covering lemma.

The outline of the paper is as follows: after stating the problem in section II, we first present a one-shot upper bound for soft covering problem using the total variation distance in Subsection III-A, then we present our main results on the exact order of soft covering lemma using both the total variation distance and relative entropy in the rest of Section III. Section IV is devoted to the ensemble converse proof of the results, while Section V is devoted to the achievability proof.

II Notations and Definitions

We follow closely the notation of Verdú’s book, [13], with the exception of using Boldface letters to denote vectors (e.g. $\mathbf{x}=(x_{1},\cdots,x_{n})$). Throughout the the paper, the base of the $\log$ and $\exp$ is $\mathrm{e}$. Also, we use the asymptotic notations such as $O(.),\Theta(.),\Omega(.)$ in the paper.

Definition 1

Relative information*. Given two measures $P$ and $Q$ on the same probability space such that $P\ll Q$, the relative information $\imath_{P||Q}$ is defined as*

[TABLE]

Definition 2

Information density.* Given a joint distribution $P_{XY}$, the information density is defined as $\imath_{X;Y}(x,y)\triangleq\imath_{P_{XY}||P_{X}\times P_{Y}}(x,y).$ Throughout of the paper, we usually omit the subscript, whenever it is clear from the context.*

Definition 3

For two distributions $P$ and $Q$ such that $P\ll Q$, the relative entropy and the total variation (TV) distance are defined as follows,

[TABLE]

where $X\sim P$ , $\overline{X}\sim Q$ and $[x]_{+}\triangleq\max\{0,x\}$.

Definition 4

Given $P_{XY}$, the $\alpha$-mutual information $I_{\alpha}(X;Y)$ [13] is defined by

[TABLE]

where $(X,\widetilde{Y})\sim P_{X}P_{Y}$.

II-A Problem Statement

Let $\mathcal{C}=\{X(k)\}_{k=1}^{\mathsf{M}}$ be a random codebook, in which its codewords are generated according to $P_{X}$. Given a channel $P_{Y|X}$, the output distribution $\mathsf{P}_{Y}$ induced by selecting uniformly an index $k$ from $[1:\mathsf{M}]$ and then transmitting $X(k)$ through the channel, is

[TABLE]

We are interested to evaluate the closeness of the induced distribution $\mathsf{P}_{Y}$ to $P_{Y}$, where $P_{X}\rightarrow P_{Y|X}\rightarrow P_{Y}$. We use the relative entropy and total variation distance to measure the closeness.

III Exact Soft Covering Order

III-A Gallager Type one-shot upper bound on TV-distance

We begin the investigation of soft covering problem by stating a Gallager type upper bound on the TV-distance in the one-shot regime.

Theorem 1

The TV-distance between the induced distribution $\mathsf{P}_{Y}$ and the desired distribution $P_{Y}$, is upper bounded by

[TABLE]

Remark 1

The one-shot bound (3) readily implies that the exact exponent of the soft covering lemma for the i.i.d. codebook [12, Theorem 1] and for the constant-composition codebook [12, Theorem 2] is achievable, respectively. This follows by,

• •

i.i.d. codebook. In this case, the codewords are drawn from $P_{X}^{\otimes n}$ and the channel $P_{Y^{n}|X^{n}}=\prod P_{Y|X}$ is memoryless. In this setting the assertion of **[12, Theorem 1]** follows from the identity $I_{\frac{1}{1-\rho}}(\mathbf{X};\mathbf{Y})=nI_{\frac{1}{1-\rho}}(X;Y)$.

• •

Constant composition codebook. In this case, the codewords are drawn uniformly from the set of all $\mathbf{x}$ with the same type $P_{X}$, where $P_{X}$ is an $n$-type. In this setting, the assertion **[12, Theorem 2]** follows from the inequality $I_{\frac{1}{1-\rho}}(\mathbf{X};\mathbf{Y})\leq nI^{c}_{\frac{1}{1-\rho}}(X;Y)$, where the Csiszar’s $\alpha$-mutual information $I^{c}_{\alpha}(X;Y)$ is defined as

[TABLE]

where $X^{\prime}\sim P_{X}$.

Moreover, this exponent is achievable for any memoryless channel (not only the finite discrete memoryless one) with the assumption that the r.h.s. of (3) is finite.

Remark 2

In a recent work [14] , Mojahedian, et. al. consider wiretap channel $P_{YZ|X}$ and derive a lower bound bound on the exponent of the TV-distance between the joint distribution $P_{M\mathbf{Z}}$ of message $M$ and eavesdropper’s observation $\mathbf{Z}$ and the product distribution $P_{M}P_{\mathbf{Z}}$ in term of Csiszar-$\alpha$-mutual information $I_{\alpha}(X;Z)$ (with the same exponent as (3) for the channel $P_{Z|X}$), using a completely different proof.

Remark 3

Duality between Gallager’s bound for channel coding and the exponent of the soft covering. The expression of (3) is the same as the Gallager’s one [15, Eq. 78] for the channel coding with the exception that $\rho$ is replaced by $-\rho$.

Proof:

The proof follows from the one-shot bound in [8, Corollary VII.2] with a simple modification. The [8, inequality (106)] asserts that

[TABLE]

where $(X,Y)\sim P_{XY}$ and $\mathcal{F}$ is an arbitrary event222Cuff [8] only considered specific event $\mathcal{F}$ that gives a simple upper bound on the second term in (4). However the analysis is valid for any event $\mathcal{F}$. . For any $0\leq\lambda\leq 1$, define

[TABLE]

For a given $Y=y$, we have

[TABLE]

where the inequality follows from the Markov inequality. Next, consider

[TABLE]

where we used the definition of $\mathcal{F}$ and $\kappa_{\lambda}$. Observe that

[TABLE]

by change of measure argument. Using this fact, substituting (6) and (8) in (4) and setting $\rho=\frac{\lambda}{1+\lambda}$ imply (3). ∎

III-B Exact order of Soft covering under relative entropy

In the rest of the paper, we consider the soft covering problem for the memoryless channel $P_{Y|X}$ in the $n$-shot regime, with the codebook $\mathcal{C}$ consisting of $\mathsf{M}_{n}=\exp(nR)$ codewords such that the codewords are generated according to the i.i.d. distribution $P^{\otimes n}_{X}:=\prod_{k=1}^{n}P_{X}$. Here we denote the induced distribution with $\mathsf{P}_{Y^{n}}$.

Theorem 2

Suppose that $R>I(X;Y)>0$. Further, assume that the moment generating function of $\imath(X;Y)$ is finite in the neighborhood of origin, that is $\mathbb{E}[\exp(\tau\imath(X;Y)]<\infty$ in the neighborhood of origin. Let

[TABLE]

Then

[TABLE]

III-C Exact order of Soft covering under TV distance

Definition 5

A channel $P_{Y|X}$ is said singular, if $\mathrm{Var}[\imath(X;Y)|Y]=0$, almost surly w.r.t. $P_{Y}$.333For the discrete channel, this definition is equivalent to the definition of singular channel in [5, Definition 1]. Otherwise, the channel is non-singular.

Theorem 3

Suppose that $R>I(X;Y)>0$ and $\mathbb{E}[\exp(\tau\imath(X;Y)]<\infty$ in the neighborhood of origin. Let

[TABLE]

*To state the exact order, we should distinguish between singular and non-singular channels. We have *

[TABLE]

where $\beta^{*}=1-\rho^{*}$, for the non-singular channels and $\beta^{*}=1$ for the singular channels.

Remark 4

Duality. Again, the expression of the (16) is similar to the expression of the exact order of random coding bound for the channel coding [5], except that $\rho$ is replaced by $-\rho$. While the expressions are similar, the proofs are quite different.

IV Exact analysis for Ensemble Converse

In this section, we present the ensemble converse proof of the Theorem 2 and Theorem 3. The proof is divided to four main steps. To make the analysis concise, we utilize the idea of Poissonizating the problem, which have been used in [12] to eliminate the correlation between weakly dependent r.v’s. Using the concentration of the Poisson r.v. around its mean, we show that the exact order of the relative entropy and TV-distance after Poissonization is the same the main problems for the fixed rate. So it suffices to find the exact oreder of the Poissonizated problem. To evaluate the exact order of Poissonizated problem, we use the thinning property of certain Poisson random sum to find lower bounds on the desired parameters. Next, we further lower bounded bounds in terms of moments of certain r.v. We present these steps in parallel for both the relative entropy and TV-distance. Then, we continue the analysis separately for these two cases, although the main trick is the same. We use the change of measure trick in the same way as the one used in the converse proof of the Bahadur-Rao [1] theorem (i.e. the exact order of the probability of the deviating of a sum from the mean) to find the exact order.

IV-A Poissonization

By Poissonization, we assume that the number of codewords is not fixed, but is a Poisson random variable, with the mean close to the size of the codebook. More precisely, we assume that the Poisson-codebook is $\{\mathbf{X}(k)\}_{k\in\mathbb{N}}$, where the codewords are generated according to $P^{\otimes n}_{X}$. Further, we assume that $M$ is a Poisson r.v. with mean $\mu_{n}=2\exp(nR)$.

Let $\mathsf{L}_{m}$ and $\mathsf{V}_{m}$ be the average of the relative entropy and the TV distance, respectively, when the number of codewords is $m$, that is

[TABLE]

where $\mathsf{P}_{\mathbf{Y}}^{(m)}(.):=\frac{1}{m}\sum^{m}_{k=1}P_{\mathbf{Y}|\mathbf{X}}\left(.|\mathbf{X}\left(k\right)\right)$. We will show that $\mathbb{E}[\mathsf{L}_{M}]$ is a good approximation for $\mathsf{L}_{\exp(nR)}$. Also $\mathbb{E}[\mathsf{V}_{M}]$ is a good approximation for $\mathsf{V}_{\exp(nR)}$. More precisely, we have,

Lemma 1

[TABLE]

where $\varepsilon_{\frac{1}{2}}=\sqrt{2}{\mathrm{e}^{-\frac{1}{2}}}<1$.

Proof:

By Lemma 10 in the Appendix B , the sequence $\{\mathsf{L}_{m}\}$ is a decreasing sequence of $m$. Further $\mathsf{L}_{1}=\mathbb{E}[D(P_{\mathbf{Y}|\mathbf{X}=\mathbf{X}_{1}}||P_{\mathbf{Y}})]=nI(X;Y)$, because $\mathbf{X}_{1}\sim P_{X}^{\otimes n}$. Thus,

[TABLE]

where the last inequality follows from [16, Theorem 5.4].

Similarly, the sequence $\{\mathsf{V}_{m}\}$ is decreasing by Lemma 10, thus

[TABLE]

where we have used $\mathsf{V}_{1}\leq 1$. ∎

Let $T$ be a random variable defined by444It is worthy to note that the randomness in $T$ comes from the randomness of the codebook, poisson r.v. $M$ and the r.v. $\mathbf{Y}\sim P_{Y}^{\otimes n}$.

[TABLE]

Lemma 2

[TABLE]

where $\varepsilon_{\frac{3}{2}}=\frac{\mathrm{e}^{.5}}{1.5^{1.5}}<1$.

Lemma 3

[TABLE]

The proofs are Lemma 2 and Lemma 3 are provided in the Appendix C and Appendix D, respectively. Comparing Lemma 1 with Lemmas 2 and 3, we get

Corollary 1

[TABLE]

IV-B Negligibility of the O-terms in the Corollary 1

We show that the O-terms in (30) and (31) are negligible with respect to the exact expressions (12) and (16), respectively. Thus, it is only required to prove the exact expressions are lower bounds for the expectation terms in the Corollary 1.

Observe that the exact exponent in the exact order (12) is

[TABLE]

where we used the Jensen inequality for the concave function $\log x$ and the assumption $I(X;Y)>0$. Thus the O-term $O\left(\exp(-nR)\right)$ is negligible w.r.t. the exact order (12).

Similarly, the exact exponent in the exact order (16) is

[TABLE]

where we used the fact that $I_{s}(X;Y)$ is an increasing function of $s$.

IV-C Lower bounding using Tinning property of Poisson random sum

Let $\mathcal{F}$ be an arbitrary event. To obtain a lower bound on $\mathbb{E}[T\log T]$ (for the TV- case, $\mathbb{E}[|T-1|]$), we split $T$ to two parts $T_{1}$ and $T_{2}$ defined below,

[TABLE]

It is clear that $T=T_{1}+T_{2}$. Further conditioned on any instance $\mathbf{Y}=\mathbf{y}$, the random variables $U_{k}:=\exp\left(\imath\left(\mathbf{X}(k);\mathbf{Y}\right)\right)$ are i.i.d. So the tinning property of the Poisson random sum $\sum_{k=1}^{M}U_{k}$, (which is proved in the Appendix G) shows that $T_{1}$ and $T_{2}$ are independent given $\mathbf{Y}=\mathbf{y}$.

Moreover,

[TABLE]

Thus using the Jensen inequality for the convex function $f(x)=x\log x$, we have

[TABLE]

Similarly the Jensen inequality for the convex function $f(x)=|x-1|$ implies

[TABLE]

IV-D Useful bounds on $\mathbb{E}[U\log U]$ and $\mathbb{E}[|U-\mathbb{E}[U]|]$ in terms of the moments and their consequences

The following lemma, which is of independent interest, plays the key role in proving the converse for the relative entropy. Its proof is given in the Appendix A.

Lemma 4

For a positive random variable $U$ with $\mathbb{E}[U]=1$, we have

[TABLE]

Also, the following lemma is the TV-counterpart of the previous lemma.

Lemma 5

For any positive random variable $U$, we have

[TABLE]

Proof:

For any r.v. $V$, we have

[TABLE]

where we have used the Holder inequality. Setting $V\leftarrow U-\mathbb{E}[U]$ and rearranging (43) yield (42). ∎

Using Lemma 4, we prove the following lemma in the Appendix E-A.

Lemma 6

For any event $\mathcal{F}$,

[TABLE]

where $(\mathbf{X},\mathbf{Y})\sim P_{XY}^{\otimes n}$.

Also, using Lemma 5, we prove the following lemma in the Appendix E-B.

Lemma 7

For any event $\mathcal{F}$,

[TABLE]

where $(\mathbf{X},\mathbf{Y})\sim P_{XY}^{\otimes n}$.

IV-E Large deviation type analysis for Relative entropy

To evaluate the bound in Lemma 6, we use the change of measure trick in the same spirit as the one used in the large deviation for proving Cramer theorem and its extension by Bahadur-Rao, see [1].

Define the tilted distribution $P_{X_{*}Y_{*}}$ via the following Radon-Nikodym derivative,

[TABLE]

where $\tau^{*}$ was defined in (9). We consider the cases $\tau^{*}<1$ and $\tau^{*}=1$, separately.

Case I: $\tau^{*}<1$. Differentiating the function inside (9) and equalling it to zero gives,

[TABLE]

Now set,

[TABLE]

Here we choose the positive constant $A$ large enough such that $\mathbb{P}[(\mathbf{X}_{*},\mathbf{Y}_{*})\in\mathcal{F}]\geq\frac{C}{\sqrt{n}}$ for some positive constant $C$, where $(\mathbf{X}_{*},\mathbf{Y}_{*})\sim P_{X_{*}Y_{*}}^{\otimes n}$. The existence of such $A$ is guaranteed by the application of Berry-Essen CLT to the r.v. $\imath_{\mathbf{X};\mathbf{Y}}(\mathbf{X}_{*};\mathbf{Y}_{*})=\sum_{i=1}^{n}\imath_{X;Y}(X_{*,i};Y_{*,i})$. Then for $\tau^{*}<1$ , we have,

[TABLE]

where (49) follows by change of measure using the definition of $P_{X_{*}Y_{*}}$ and (50) follows from the definition of the event $\mathcal{F}$.

Similarly we have,

[TABLE]

Substituting (51) and (53) Lemma (6), implies that for some $C_{1}>0$,

[TABLE]

Putting this in (30) concludes the converse proof of Theorem 2.

Case II: $\tau^{*}=1$. Lemma 13 in the Appendix H implies

[TABLE]

where $P_{X_{*}Y_{*}}$ is defined by (46) with $\tau^{*}=1$. Now set,

[TABLE]

We have,

[TABLE]

where (58) follows by change of measure using the definition of $P_{X_{*}Y_{*}}$.

Similarly we have,

[TABLE]

Substituting (59) and (61) in Lemma (6), yields

[TABLE]

where the last inequality follows from (56) and the Berry-Esseen approximation $\mathbb{P}[(\mathbf{X}_{*},\mathbf{Y}_{*})\in\mathcal{F}]=\frac{1}{2}+O(n^{-\frac{1}{2}})$. Putting this in (30) concludes the converse proof of Theorem 2.

IV-F Large deviation type analysis for TV-distance

To evaluate the bound in Lemma 7, we use again the change of measure trick, although it is more involved w.r.t. the one used for the relative entropty.

Define the tilted conditional distribution $P_{\overline{X}|\overline{Y}}$ and distribution $P_{\overline{Y}}$ via the following Radon-Nikodym derivatives,

[TABLE]

where $\rho^{*}$ was defined in (13). Also, for brevity let

[TABLE]

where $(X,Y)\sim P_{XY}$.

We consider the cases $\rho^{*}<\frac{1}{2}$ and $\rho^{*}=1$, separately.

Case I: $\rho^{*}<\frac{1}{2}$. By Corollary 3 in Appendix H , $R$ and $\rho^{*}$ satisfy the following identity,

[TABLE]

where the r.v. $Z$ is defined through,

[TABLE]

Also for $k=1,\cdots,n$, let

[TABLE]

where $(\overline{X}_{1},\overline{Y}_{1}),\cdots,(\overline{X}_{n},\overline{Y}_{n})$ are i.i.d and drawn from $P_{\overline{X},\overline{Y}}$. Now, we compute the expressions appeared in Lemma 7 in terms of $Z_{1},\cdots,Z_{n}$. First, consider

[TABLE]

where (208) follows by change of measure and (209) follows from the definition of $Z_{k}$. Similarly, we have

[TABLE]

Using (73) and (75), the lower bound in Lemma 7 simplifies as follows,

[TABLE]

where

• •

(73) and (75) yields (76)

• •

change of measure $P_{Y}\rightarrow P_{\overline{Y}}$ implies (77). Also, here $\mathsf{M}_{n}:=\exp(nR)$.

• •

The identity $R=\mathbb{E}[Z]$ gives (78).

Now ${\sf S}^{n}{\sf M}_{n}^{-\rho^{*}}=\exp(-n\rho^{*}(R-I_{\frac{1}{1-\rho^{*}}}(X;Y)))$, which is the exponent appeared in Theorem 3. So it is remained to bound the expectation inside (78) to get the desired pre-factor. Let

[TABLE]

and set

[TABLE]

where $a$ is a large enough fixed value, such that $\mathbb{P}[(\overline{X},\overline{Y})\in\mathcal{F}]\geq\frac{c}{\sqrt{n}}$ for some $c>0$. Then the pre-factor ${\sf P}$ can be lower-bounded as follows,

[TABLE]

where $K_{1}=\exp(3(1-2\rho^{*})a)$, $K_{2}=6\exp((1-2\rho^{*})a)$ and $C$ is a positive constant. Finally, (84) completes the converse proof of Theorem 3 for the singular channels with $\tau^{*}<1$.

Non-Singular channels:

The choice of $\mathcal{F}$ in (80) led to pre-factor scale $\frac{1}{\sqrt{n}}$ which is optimal for the singular channels. However there is a gap between it and the optimal pre-factor scale $n^{-\frac{1-\rho^{*}}{2}}$ for the non-singular channels. Here, we perturb the definition of $\mathcal{F}$ to get the optimal pre-factor scale.

Let

[TABLE]

and

[TABLE]

where for a r.v. $X$, $\mathrm{M}_{3}[X]\triangleq\mathbb{E}[|X-\mathbb{E}[X]|^{3}]$. The key property of non-singular channel is that $Z$ is not a function of $Y$, a.s. $P_{Y}$. Since $P_{Y}\ll\gg P_{\overline{Y}}$, it is not also a function of $Y$, a.s. $P_{\overline{Y}}$. As a result, $\mathbb{E}[\mathrm{Var}[Z|\overline{Y}]]$ and $\mathbb{E}[\mathrm{M}_{3}[Z|\overline{Y}]]$ are strictly positive.

Lemma 8

For large enough $n$,

There exists a constant $C_{0}$ such that for any $\mathbf{y}\in\mathcal{G}$,

[TABLE] 2. 2.

There exists a constant $C_{1}$ such that

[TABLE]

The proof of this lemma is relegated to the Appendix F.

Using Lemma 8, the pre-factor $\mathsf{P}$ in (79) is lower-bounded as follows,

[TABLE]

where $K_{1}$ and $K_{2}$ were defined before, (90) follows from the definition of $\mathcal{F}$, (91) follows from the first item of Lemma 8 and (92) follows from the second item of Lemma 8.

Putting (92) in (78), concludes the converse proof of Theorem 3 for the non-singular channels.

Case II, $\rho^{*}=\frac{1}{2}$.

It is shown in Appendix H that

[TABLE]

where $Z$ is defined by (70) with $\rho^{*}=\frac{1}{2}$. Now set,

[TABLE]

Then the lower-bound in Lemma 7 is lower-bounded as follows,

[TABLE]

where

• •

(95) follows from (77) (which is valid for any $\rho^{*}$) with $\rho^{*}=\frac{1}{2}$,

• •

The definition of $\mathcal{F}$ gives (96),

• •

$R\geq\mathbb{E}[Z]$ yields (97),

• •

Berry-Essen approximation $\mathbb{P}[(\overline{\mathbf{X}},\overline{\mathbf{Y}})\in\mathcal{F}]=\frac{1}{2}+O(\frac{1}{\sqrt{n}})$ results in (98).

Finally, this concludes the converse proof for $\rho^{*}=\frac{1}{2}$.

V Exact Analysis for the Achievability

V-A Relative Entropy

The starting point in the achievability proof is the following well-known upper bound on the relative entropy, ( which is an one-shot bound, see [17, Appendix IV] among many others),

[TABLE]

where $\mathsf{M}_{n}:=\exp(nR)$. We know proceed to get an almost exact computable expression for the r.h.s. of (99). To do this, we prove the following general result,

Theorem 4

Let $(V_{1},\cdots,V_{n})$ be i.i.d. r.v.’s with $V_{i}\sim P_{V}$ and $\mathbb{E}[V]<0$. Further, assume that $V$ has finite moment generating function in the neighborhood of the origin. Let

[TABLE]

Then, if $\tau^{*}<1$, we have for some $C>0$ which does not depend on $n$ and depends only on $\tau^{*}$,

[TABLE]

Remark 5

The previous technique [17] for bounding the l.h.s. of (99) gives an upper bound with the same exponent but without the pre-factor $\frac{1}{\sqrt{n}}$.

Setting $V_{i}\leftarrow(\imath_{X;Y}(X_{i};Y_{i})-R)$, in Theorem 4 implies (notice that $\mathbb{E}[V]=I(X;Y)-R<0$),

[TABLE]

where $\tau^{*}$ was defined in (9). This completes the proof of the achievability for the case $\tau^{*}<1$. The case $\tau^{*}=1$ is follows from the known result in [17].

Proof:

Define the tilted distribution $P_{\overline{V}}$ via the following Radon-Nikodym derivative,

[TABLE]

if $\tau^{*}<1$, then the following equation holds,

[TABLE]

Now we can write,

[TABLE]

where $(\overline{V}_{1},\cdots,\overline{V}_{n})$ are i.i.d. and distributed according to $P_{\overline{V}}$. The equality follows by change of measure.

Let $S_{n}=\frac{1}{\sqrt{n}}\sum_{k=1}^{n}\overline{V_{k}}$ and $g(x):=\exp\left({-\tau^{*}}x\right)\log\left(1+\exp\left(x\right)\right)$. Then we have,

[TABLE]

where (106) follows by integration by part and (107) is due to the fact that $g$ vanishes at $\pm\infty$ (this is true, since $0<\tau^{*}<1$).

Let $\sigma^{2}:=\mathbb{E}\left[\overline{V}^{2}\right]$ and $\rho=\mathbb{E}\left[|\overline{V}|^{3}\right]$. By the Berry-Esseen theorem [16, Theorem 9.8],

[TABLE]

where $F_{Y_{\sigma}}(x)$ is the c.d.f. of a mean zero Gaussian random variable $Y_{\sigma}$ with variance $\sigma^{2}.$ Hence

[TABLE]

where (112) holds, since $\int_{-\infty}^{\infty}g^{\prime}(x)dx=0$ (because $|g^{\prime}|$ is integrable and again $g$ vanishes at infinities). Here, $K=\frac{1}{\sqrt{2\pi\sigma^{2}}}$ is the upper bound on $F^{\prime}_{Y_{\sigma}}=f_{Y_{\sigma}}$. It is easy to verify that $g^{\prime}(x)$ decays exponentially fast at $\pm\infty$. Thus both the integrals inside (113) are convergent. In summary, we conclude that there exists a constant $C$ depending only on the distribution $P_{V}$, such that

[TABLE]

∎

V-B TV-distance

Theorem 1 implies achievability of the bound (16) without the pre-factor for any $\rho^{*}$. As a result, it yields the achievability of the bound (16) for the case $\rho^{*}=\frac{1}{2}$. So it is only required to investigate the case $\rho^{*}<\frac{1}{2}$.

As in the proof of Theorem 1, we start with the following $n$-shot version of the lower bound (4),

[TABLE]

where ${\sf M}_{n}=\exp(nR)$.

To compute this bound for the appropriate choice of $\mathcal{F}$ (will be determined later), we need the following lemma,

Lemma 9** **(

[18, Lemma 47] )

Let $(V_{1},\cdots,V_{n})$ be i.i.d. zero mean r.v.’s with $V_{i}\sim P_{V}$. Further, assume that $V$ has finite third moment $T_{3}$ and non-zero variance $\sigma^{2}$. Then, for any $A$,

[TABLE]

where $C=\frac{2}{\sigma}(\frac{\log 2}{\sqrt{2\pi}}+\frac{12T_{3}}{\sigma^{2}})$.

We investigate the non-singular channels and singular channels, separately.

V-B1 Non-singular channels

Recall the definitions of $P_{\overline{X}|\overline{Y}}$, $P_{\overline{Y}}$, $\mathsf{S}$, $Z$, $Z_{k}$ and $\rho^{*}$ in the equations (65)—(71). Also, let

[TABLE]

We compute each term of (115) separately. First consider

[TABLE]

where

• •

change of measure $P_{XY}\rightarrow P_{\overline{X}\overline{Y}}$ implies (118),

• •

definitions of $Z_{k}$ and $\mathcal{F}$ gives (119),

• •

the identity $R=\mathbb{E}[Z]$ gives (120)

• •

Lemma 9 with $V_{k}\leftarrow\rho^{*}(Z_{k}-\mathbb{E}[Z])$ and $A\leftarrow-\frac{\rho^{*}}{2}\log n$ yields (121).

Next consider the second term of (115),

[TABLE]

where

• •

putting (73) in (122) yields (123),

• •

change of measure $P_{Y}\rightarrow P_{\overline{Y}}$ and the definitions of $Z_{k}$ and $\mathcal{F}$, imply (124),

• •

Jensen inequality for the concave mapping $x\mapsto\sqrt{x}$ gives (125),

• •

the identity $R=\mathbb{E}[Z]$ gives (126),

• •

Lemma 9 with $V_{k}\leftarrow-(1-2\rho^{*})(Z_{k}-\mathbb{E}[Z])$ and $A\leftarrow\frac{1-2\rho^{*}}{2}\log n$ yields (127).

Finally putting (121) and (127) together gives the desired result for the non-singular channels.

V-B2 Singular channel

The Definition of singular channel implies that $\imath_{X;Y}(X;Y)$ is a function of $Y$, almost surely $P_{Y}$. For brevity, let $\imath_{X;Y}(x;y):=g(y)$. Then, the definitions of $P_{\overline{Y}}$ and $\mathsf{S}$ are reduced to

[TABLE]

[TABLE]

Also, the identity (69) is reduced to,

[TABLE]

Set,

[TABLE]

where $(\overline{Y}_{1},\cdots,\overline{Y}_{n})\sim P^{\otimes n}_{\overline{Y}}$. Now, we compute the terms inside (115). First consider,

[TABLE]

where

• •

change of measure $P_{Y}\rightarrow P_{\overline{Y}}$ implies (132),

• •

the identity $R=\mathbb{E}[g(\overline{Y})]$ gives (133),

• •

Lemma 9 with $V_{k}\leftarrow-\rho^{*}(g(\overline{Y}_{k})-\mathbb{E}[\overline{Y}])$ and $A=0$ yields (134).

Next, consider

[TABLE]

where

• •

change of measure $P_{Y}\rightarrow P_{\overline{Y}}$ implies (135),

• •

the identity $R=\mathbb{E}[g(\overline{Y})]$ gives (136),

• •

Lemma 9 with $V_{k}\leftarrow-(\frac{1}{2}-\rho^{*})(g(\overline{Y}_{k})-\mathbb{E}[\overline{Y}])$ and $A=0$ yields (137).

Finally putting (134) and (137) together gives the desired result for the singular channels.

Appendix A Proof of Lemma 4

Using the identity $u\log u+1-u=(u-1)^{2}\int_{0}^{1}\frac{1-t}{1+t(u-1)}\mathsf{d}t$, we have

[TABLE]

where (140) follows from Cauchy-Schwarz inequality.

Appendix B Monotonicity of $\mathsf{L}_{m}$ and $\mathsf{V}_{m}$

We prove a more general result. Let $f:\mathbb{R}^{\geq 0}\mapsto\mathbb{R}$ be a convex function with $f(1)=0$ and $D_{f}(P||Q)$ (defined below) is the $f$-divergence between $P$ and $Q$,

[TABLE]

where $Z\sim Q$.

Let

[TABLE]

where $\mathsf{P}_{Y}^{(m)}(.):=\frac{1}{m}\sum_{k=1}^{m}P_{Y|X}(.|X(k))$, in which $(X_{1},\cdots,X_{m})\sim P_{X}\otimes\cdots\otimes P_{X}$.

Lemma 10

$\mathsf{L}_{m}^{(f)}$* is a decreasing sequence of $m$.*

Proof:

Observe that

[TABLE]

Let

[TABLE]

Then, we have

[TABLE]

Using this and the Jensen inequality for the convex function $f$, we get

[TABLE]

where (145) follows from Jensen inequality and (146) follows from symmetry. ∎

Appendix C Proof of Lemma 2

Consider,

[TABLE]

where

• •

Identity (149) follows from the definition of relative entropy and the definition of $\mathsf{L}_{M}$ in (17),

• •

Equality (153) follows from the following identity,

[TABLE]

, since $\mathbb{E}[\exp(\imath(\mathbf{X}_{k};\mathbf{Y}))]=1$ for any $k$.

• •

Inequality (154) follows from the following inequality for the poisson r.v. $M$,

[TABLE]

where we used the inequality $\log x\leq x-1$, $\mathbb{E}[M^{2}]=\mu_{n}^{2}+\mu_{n}$ and $\mathbb{E}[M]=\mu_{n}$.

• •

Similar to (158), equality (155) follows from the following identity

[TABLE]

• •

(156) follows, because $\mathsf{L}_{k}$ is a decreasing sequence,

• •

Simple algebraic calculation for the Poisson r.v. $M$ implies (157).

Finally, applying the following tail probability of the Poisson r.v. concludes the proof,

[TABLE]

Appendix D Proof of Lemma 3

The proof modifies the one given in [12].

Proof:

[TABLE]

where

• •

Identity (161) follows from the definition of relative entropy and the definition of $\mathsf{V}_{M}$ in (18),

• •

Equality (165) follows from the triangle inequality and the following identity,

[TABLE]

• •

Inequality (166) follows from $\mathbb{E}[|X-\mathbb{E}[X]|]\leq\mathrm{Var}[X]$ for any r.v. $X$, $\mathrm{Var}[M]=\mu_{n}$ and $\mathsf{V}_{k}\leq 1$ for any $k$.

• •

The inequality leading to (167) is currently proven in (157) and (160).

∎

Appendix E Proofs of the Lemma 4 and Lemma 5

E-A Proof of Lemma 4

Write $T_{1}=\sum_{k=1}^{M}Z_{k}$, where

[TABLE]

Utilizing Lemma 4 with $U\leftarrow T_{1}+1-\mathbb{E}[T_{1}|Y]$ in (39), yields

[TABLE]

where (171) is a result of simple algebraic calculations using the moments of the Poisson r.v. $M$ and (172) follows by applying the Jensen inequality to the jointly convex function $f(x,y):=\frac{x^{2}}{y}$.

Next consider,

[TABLE]

where the equalities (174) and (175) follow from change of measure, since $(\mathbf{X}(1),\mathbf{Y})\sim P^{\otimes n}_{X}P^{\otimes n}_{Y}$. Substituting (174) and (175) in (173) concludes the proof.

E-B Proof of Lemma 5

Utilizing Lemma 5 with $U\leftarrow T_{1}$ in (40), yields

[TABLE]

where (178) is a result of simple algebraic calculations using the moments of the Poisson r.v. $M$. Next consider,

[TABLE]

where the equalities (179) and (180) follows from change of measure, since $(\mathbf{X}(1),\mathbf{Y})\sim P^{\otimes n}_{X}P^{\otimes n}_{Y}$. Substituting (179) and (180) in (178) concludes the proof.

Appendix F Proof of Lemma 8

F-A * Proof of item 1:*

Using the Berry-Essen theorem for the sum $\sum_{k=1}^{n}Z_{k}$, we get the following approximation for any $\mathbf{y}\in\mathcal{G}$,

[TABLE]

where $N$ is the standard normal random variable,

[TABLE]

and $d=\frac{9\sqrt{8}\mathbb{E}[\mathrm{M}_{3}[Z|\overline{Y}]]}{\mathbb{E}[\mathrm{Var}[Z|\overline{Y}]]^{\frac{3}{2}}}$.

Observe that for large enough $n$

[TABLE]

Hence

[TABLE]

Putting (182) and (183) together yields that for large enough $a$,

[TABLE]

for some constant $C$.

F-B * Proof of item 2:*

Using Chebyshev inequality, it is easy to show that the probability of violating the second constraint and the third constraint of (86) is of order $O\left(n^{-.5}\right)$. Further, the Berry-Essen theorem implies that the probability of deviating of order $\sqrt{n}$ from the mean is lower-bounded by some non-zero constant $K$. Thus for large enough $n$, the probability of $\mathcal{G}$ is lower bounded by $\frac{K}{2}$.

Appendix G Thinning property of a random Poisson sum

Let $X_{1},X_{2},\cdots$ be a sequence of i.i.d. random variables with the distribution $P_{X}$ and characteristic function $\varphi_{X}(t):=\mathbb{E}[\exp(\mathrm{i}tX)]$, where $X\sim P_{X}$. Let $M$ be a poisson r.v. with mean $\mu$ and independent of the sequence $(X_{1},X_{2},\cdots)$. Further let $\mathcal{F}\subseteq\mathcal{X}$ be a measurable event w.r.t. to $P_{X}$. The following lemma is related to the thinning property of Poisson random process [19].

Lemma 11

Let $U=\sum_{k=1}^{M}X_{k}\mathbbm{1}\{X_{k}\in\mathcal{F}\}$ and $V=\sum_{k=1}^{M}X_{k}\mathbbm{1}\{X_{k}\notin\mathcal{F}\}$. Then $U$ and $V$ are independent.

Proof:

Let $\varphi_{U,V}(s,t):=\mathbb{E}[\exp(\mathrm{i}(sU+tV))]$, be the joint characteristic function of the pair $(U,V)$. It suffices to show

[TABLE]

The characteristic function of the random sum $U$ is given by (see [20, Equation 2.4, P. 504])

[TABLE]

Observe that

[TABLE]

Hence,

[TABLE]

Similarly,

[TABLE]

Thus,

[TABLE]

Next consider,

[TABLE]

where we have used again the formula [20, Equation 2.4, P. 504] for the Poisson random sum $\sum_{k=1}^{M}X_{k}(s\mathbbm{1}\{X_{k}\in\mathcal{F}\}+t\mathbbm{1}\{X_{k}\notin\mathcal{F}\})$.

Now observe,

[TABLE]

Substituting (195) in (193) and comparing the result with (190) yield (185). ∎

Appendix H On the optimum values $\tau^{*}$ and $\rho^{*}$

Lemma 12

The mapping $G:[0,1)\rightarrow\mathbb{R}$ defined by

[TABLE]

where $(X,{Y})\sim P_{XY}$, is convex.

Proof:

Let F(\rho):={\mathbb{E}\left[\mathbb{E}^{1-\rho}\left[\exp\left(\frac{\rho}{1-\rho}\imath_{X;Y}(X;{Y})\right)\Big{|}{Y}\right]\right]}. It suffices to show that for any $\theta,\alpha,\beta\in[0,1]$,

[TABLE]

where $\bar{\theta}=1-\theta$. The proof follows from repeatedly applying Jensen inequality, as follows,

[TABLE]

where (199) and (202) follow from the Holder inequality and the fact that the mapping $x\mapsto x^{\theta(1-\alpha)+\bar{\theta}(1-\beta)}$ is increasing. ∎

Corollary 2

The function $H:[0,1)\mapsto\mathbb{R}$ defined by

[TABLE]

is increasing.

Let $(X_{\rho},Y_{\rho})$ be a pair of tilted random variables defined by the following pair of Radon-Nikodym derivatives,

[TABLE]

Also let $Z_{\rho}$ be a r.v. defined by

[TABLE]

Using these definitions, the r.h.s. of the derivative (204) can be simplified as follows,

[TABLE]

where (208) and (209) follow from the change of measures $P_{X|Y}\rightarrow P_{X_{\rho}|Y_{\rho}}$ and $P_{Y}\rightarrow P_{Y_{\rho}}$, respectively.

Now let $R>I(X;Y)$ be a fixed number and define,

[TABLE]

Since $G$ is convex, the function $f(\rho):=\rho R-G(\rho)$ gets its maximum either at the end-points $0,\frac{1}{2}$ or an interior point $\rho^{*}$ of the interval $[0,\frac{1}{2}]$ such that $\rho^{*}=H(\rho^{*})=\mathbb{E}[Z_{\rho^{*}}]$. Also $\rho^{*}\neq 0$, because $f^{\prime}(0)=R-I(X;Y)>0$. Therefore, if $\rho^{*}$ is not an interior point, it should be $\rho^{*}=\frac{1}{2}$ and it happens, if $R\geq\mathbb{E}[Z_{\frac{1}{2}}]$ due to the fact that $\mathbb{E}[Z_{\rho}]=H(\rho)$ is increasing. In summary, we proved the following,

Corollary 3

We have,

[TABLE]

A similar argument shows the following counterpart for the optimization in (9).

Lemma 13

We have

[TABLE]

where $(X_{\tau},Y_{\tau})$ is defined via the following Radon-Nikodym derivative,

[TABLE]