Sharp Bounds for Mutual Covering

TL;DR

This paper introduces new sharp bounds for the mutual covering lemma using advanced concentration inequalities, with applications showing rapid convergence and exact error exponents in distribution simulation.

Contribution

It develops novel mutual covering bounds employing Talagrand's inequality and applies them to derive exponential convergence rates and precise error exponents.

Findings

Covering probability converges doubly exponentially fast when set probability is bounded away from 0 and 1.

The maximum difference in probabilities relates to the total variation distance between distributions.

Exact error exponents are derived for the joint distribution simulation problem.

Abstract

A fundamental tool in network information theory is the covering lemma, which lower bounds the probability that there exists a pair of random variables, among a give number of independently generated candidates, falling within a given set. We use a weighted sum trick and Talagrand's concentration inequality to prove new mutual covering bounds. We identify two interesting applications: 1) When the probability of the set under the given joint distribution is bounded away from 0 and 1, the covering probability converges to 1 \emph{doubly} exponentially fast in the blocklength, which implies that the covering lemma does not induce penalties on the error exponents in the applications to coding theorems. 2) Using Hall's marriage lemma, we show that the maximum difference between the probability of the set under the joint distribution and the covering probability equals half the minimum total variation distance between the joint distribution and any distribution that can be simulated by selecting a pair from the candidates. Thus we use the mutual covering bound to derive the exact error exponent in the joint distribution simulation problem. In both applications, the determination of the exact exponential (or doubly exponential) behavior relies crucially on the sharp concentration inequality used in the proof of the mutual covering lemma.

Equations

$$\mathbb{P}\left[\bigcup _{m=1}^M\{(U_m,V)\in \mathcal{F}\}\right]$$

$$\underset{\mathcal{F}:P_{UV}(\mathcal{F})\ge \frac{1}{2}}{\sup }\mathbb{P}\left[\bigcap _{m=1}^M\bigcap _{l=1}^L(U_m,V_l)\notin \mathcal{F}\right]$$

$$\approx \max \left\{{\exp }_e(-M),{\exp }_e(-L),{\exp }_e\left(-\frac{ML}{I(U;V)}\right)\right\}.$$

$$\underset{n\to \infty }{\lim }\frac{1}{n}\log M_n{L}_n>I(\mathsf{U};\mathsf{V})$$

$$\underset{\mathcal{F}}{\sup }\left\{P_{UV}(\mathcal{F})-\mathbb{P}\left[\bigcup _{m=1}^M\bigcup _{l=1}^L(U_m,V_l)\in \mathcal{F}\right]\right\}$$

$$\approx \mathbb{P}[{ı}_{U;V}(U;V)>\log ML]$$

$$\mathbb{P}\left[\bigcap _{m=1}^M\{(U_m,V)\notin \mathcal{F}\}\right]$$

$$\le P_{UV}({\mathcal{F}}^c)+\mathbb{P}[{ı}_{U;V}(U;V)\ge \log M-\gamma ]$$

$$+e^{-\exp (\gamma )}$$

$${\left(1-\frac{p\alpha }{M}\right)}^M\le 1-p+e^{-\alpha }$$

$$\bigcap _{m=1}^M\bigcap _{l=1}^L\{(U_m,V_l)\notin \mathcal{F}\}.$$

$$\mathbb{P}\left[{\cap }_{m=1}^M{\cap }_{l=1}^L\{(U_m,V_l)\notin \mathcal{F}\}\right]$$

$$\le P_{UV}({\mathcal{F}}^c)+\mathbb{P}\left[\exp ({ı}_{U;V}(U;V))\ge \frac{ML}{\exp (\gamma )}-\delta \right]$$

$$+\frac{\min \{M,L\}-1}{\delta }+e^{-\exp (\gamma )}.$$

$$\mathbb{P}\left[\bigcap _{m=1}^M\bigcap _{l=1}^L\{(U_m,V_l)\notin \mathcal{F}\}\right]$$

$$+\frac{M+L}{{\left(P_{UV}({\mathcal{F}}^c)-ϵ\right)}^{2}ML}$$

$$I_{\infty }^{ϵ}$$

$$P_{U_m{U}_{\bar{m}}{V}_l{V}_{\bar{l}}}=P_U\times P_U\times P_V\times P_V,\forall m\ne \bar{m},l\ne \bar{l}.$$

$$\mathbb{P}$$

$$\le \frac{\exp (-\gamma )+M^{-1}+L^{-1}}{\mathbb{P}\left[(U,V)\in \mathcal{F},{ı}_{U;V}(U;V)\le \log ML-\gamma \right]}.$$

$$P_{U^{M}V^{L}}=\underbrace{P_{U}\times\cdots\times P_{U}}_{\mbox{\tiny{$M$ times}}}\times\underbrace{P_{V}\times\cdots\times P_{V}}_{\mbox{\tiny{$L$ times}}},$$

$$\mathbb{P}\left[\bigcap _{m=1}^M\bigcap _{l=1}^L\{(U_m,V_l)\notin \mathcal{F}\}\right]$$

$$\le {\exp }_e\left(-\frac{\mathbb{P}\left[(U,V)\in \mathcal{F},{ı}_{U;V}(U;V)\le \log ML-\gamma \right]}{4\exp (-\gamma )+2M^{-1}+2L^{-1}}\right).$$

$$\underset{\mathcal{F}:P_{UV}(\mathcal{F})\ge p}{\sup }\mathbb{P}\left[\bigcap _{m=1}^M\bigcap _{l=1}^L\{(U_m,V_l)\notin \mathcal{F}\}\right]$$

$$\le \underset{ϵ\in [0,p]}{\inf }{\exp }_e\left(-\frac{p-ϵ}{\frac{2}{M}+\frac{2}{L}+\frac{4\exp (I_{\infty }^{ϵ}(U;V))}{ML}}\right)$$

$$M_n$$

$$L_n$$

$$\underset{n\to \infty }{\lim }\frac{1}{n}\log \log \frac{1}{{\sup }_{\mathcal{F}}\mathbb{P}[{\bigcap }_{m=1}^{M_n}{\bigcap }_{l=1}^{L_n}(U_m^{(n)},V_l^{(n)})\notin \mathcal{F}]}$$

$$=\min \{R_1,R_2,R_1+R_2-I(\mathsf{U};\mathsf{V})\}.$$

$$\underset{n\to \infty }{\mathrm{liminf}}\frac{1}{n}\log M_{kn}\ge R_k,k=1,2$$

$$\underset{n\to \infty }{\mathrm{limsup}}\underset{k=1,2}{\max }\mathbb{P}[{\hat{W}}_k\ne W_k]=0.$$

$$R_1$$

$$R_2$$

$$R_1+R_2$$

$$\underset{n\to \infty }{\mathrm{liminf}}\frac{1}{n}\log \frac{1}{{\max }_{k=1,2}\mathbb{P}[{\hat{W}}_k\ne W_k]}$$

$$\displaystyle\geq\max_{\theta\in[0,1]}\min\left\{\begin{array}[]{c}E_{0}(\theta,P_{\sf U},P_{\sf Y|U})-\theta R_{1},\\ E_{0}(\theta,P_{\sf V},P_{\sf Z|V})-\theta R_{2},\\ \frac{E_{0}(\theta,P_{\sf U},P_{\sf Y|U})+E_{0}(\theta,P_{\sf V},P_{\sf Z|V})-\theta(R_{1}+R_{2}+I({\sf U;V}))}{2}\end{array}\right\}$$

$$\underset{\mathcal{F}}{\sup }\left\{P_{UV}(\mathcal{F})-\mathbb{P}\left[\bigcup _{m=1}^M\bigcup _{l=1}^L(U_m,V_l)\in \mathcal{F}\right]\right\}$$

$$=\frac{1}{2}\underset{P_{\hat{M}\hat{L}|U^M{V}^L}}{\inf }\left|P_{UV}-P_{U_{\hat{M}}{V}_{\hat{L}}}\right|$$

Full text

Keywords: Shannon theory, network information theory, covering lemmas, one-shot method, distribution simulation, randomness generation, rejection sampling, concentration inequalities, information density.

I Introduction

A cornerstone in the achievability proofs of many problems in rate-distortion theory and network information theory is the asymptotic covering lemma (see e.g. [1, Lemma 9.1][2, Lemma 3.3]), which gives sufficient conditions for finding among a given number of independently generated random variables a pair that will be jointly typical.

A non-asymptotic single-shot111A single-shot formulation refers to the non-asymptotic setting where the random variables are not necessarily vectors with i.i.d. coordinates. Of course, the main contribution of the paper lies in the new arguments for bounding the covering and the sampling error, which are independent of the type of formulation we adopt. version of the unilateral covering lemma222In contrast to the standard literature, we add the qualifier “unilateral” to differentiate the standard covering lemma with the mutual covering lemma. was formulated by Verdú [3]: Given $P_{UV}$, let $U_{1},\dots,U_{M}\sim P_{U}$, $V\sim P_{V}$, and suppose that $V,U_{1},\dots,U_{M}$ are independent. If $\mathcal{F}\subseteq\mathcal{U}\times\mathcal{V}$ is such that $P_{UV}(\mathcal{F})$ is large, then

[TABLE]

must be large as well, provided that $M$ is sufficiently large. The key in Verdú’s single-shot non-asymptotic covering lemma [3] is to express the condition on the size of $M$ in terms of a bound on the information spectrum of the pair $(U,V)$, namely, $M$ is large with respect to $\exp(\eta)$ where $\eta$ is such that $\mathbb{P}\left[\imath_{U;V}(U;V)>\eta\right]$ is small.

The conventional asymptotic version can be recovered if we take $P_{UV}$ to be $P_{\sf UV}^{\otimes n}$, and $\mathcal{F}$ to be the typical set. Then, by the central limit theorem, for (1) to be bounded away from 0 and 1, one must take $M=\exp(nI({\sf U;V})+O(\sqrt{n}))$. The mutual covering lemma [4] (see also [2, Lemma 8.1]), which finds applications such as Marton’s coding scheme for broadcast channels and multiple-description coding, may be thought of as a generalization where one finds a pair $(U_{m},V_{l})\in\mathcal{F}$ from $U_{1},\dots,U_{M}\sim P_{U}$, $V_{1},\dots,V_{L}\sim P_{V}$.

The standard proof of the unilateral covering lemma [2, Lemma 3.3] does not apply immediately to the mutual covering case because of the more complicated dependence structure between the pairs. The original proof of the mutual covering lemma [4] uses a “second-moment method”, an idea widely used in graph theory (see e.g. [5]). An alternative approach based on channel resolvability was recently given in [6], which gives strictly tighter exponential bounds in certain regimes. A survey of previous covering lemmas is given in Section II.

In this paper, we prove a stronger mutual covering lemma with new arguments (Lemma 5), and demonstrate its power in old and new applications of the covering lemma. Strictly improving previous mutual covering bounds, the new bound is sharp in the following two regimes:

1. “Typical case $\mathcal{F}$”: For “regular” $P_{UV}$ (in particular, for the stationary memoryless cases where $P_{UV}=P_{\sf UV}^{\otimes n}$), we have333Unless specified, the bases in $\log$ and $\exp$ are arbitrary but matching.

[TABLE]

See (14) for the formal statement. In particular, in the stationary memoryless case with $M_{n}$ and $L_{n}$ growing exponentially in the blocklength $n$, and

[TABLE]

fixed, the covering error for a “typical” $\mathcal{F}$ (that is, $P_{\sf UV}^{\otimes n}(\mathcal{F})$ is bounded away from 0 and 1) must be doubly exponentially small in $n$.

2. “Worst case $\mathcal{F}$”: Again for “regular” $P_{UV}$, we have

[TABLE]

where $(U,V)\sim P_{UV}$, and the information density is $\imath_{U;V}(u;v):=\log\frac{\operatorname*{d}P_{UV}}{\operatorname*{d}(P_{U}\times P_{V})}(u;v)$. See (28) for the formal statement. In the stationary memoryless case where (3) holds, the right side of (4) vanishes exponentially (rather than doubly exponentially) and captures the correct exponent of the left side. This implies that the supremum in (4) is achieved by some $\mathcal{F}$ such that $P_{\sf UV}^{\otimes n}(\mathcal{F})=o(1)$, since otherwise the result contradicts the double exponential convergence for the “typical case $\mathcal{F}$”.

One might ask whether the left side of (2) or the left side (4) is the “right” notion of the covering error. Interestingly, we demonstrate two applications where the two quantities respectively play a fundamental role:

1. Achievable error exponent in the broadcast channel. The estimate in (2) is more relevant to the achievability proof of coding theorems, which is the original motivation for the mutual covering lemma in [4]. We present a Gallager-type error exponent for the broadcast channel using Marton’s coding scheme. The doubly exponential decay in mutual covering has the implication that the covering error does not contribute to the error exponent in the broadcast channel. In contrast, previous covering lemmas give exponential bounds on the covering error, leading to strictly worse error exponents for the broadcast channel.

2. Joint distribution simulation via selection. Given $P_{UV}$, let $U_{1},\dots,U_{M}\sim P_{U}$, $V_{1},\dots,V_{L}\sim P_{V}$. Suppose that we want to approximate $P_{UV}$ by the distribution of $(U_{\hat{M}},V_{\hat{L}})$, where the indices $\hat{M}\in\{1,\dots,M\}$ and $\hat{L}\in\{1,\dots,L\}$ are selected upon observing all the random variables $U^{M}$ and $V^{L}$. A duality between covering and sampling recently observed in [7] shows that the approximation error in total variation is precisely characterized by the left side of (4). Based on this duality observation, we derive the exact error exponent as well as the second-order rates (for a nonvanishing error) of joint distribution simulation of stationary memoryless sources. For contrast, we prove that a simple weighted selection rule (similar to the likelihood encoder [8][9]) is fundamentally incapable of yielding the exact error exponent (Section X).

Distribution simulation has proved to be fertile ground for non-asymptotic achievability bounds [8][9][10][11][12][13][14][15][16]. For example, in the likelihood encoder approach [9], the encoder selects a codeword such that a target joint distribution with the source is achieved, even though they are independently generated. Non-interactive distribution generation, a topic in information theory and theoretical computer science [17][18][19], may appear similar to the joint distribution simulation problem mentioned above. However, the joint distribution simulation setup we consider is actually quite different, in the sense of being “fully interactive” ($\hat{M}$ is selected upon viewing both $U^{M}$ and $V^{L}$, instead of just $U^{M}$), and moreover the output must be one of the samples rather than a function of the observed sequence.

As alluded above, in both the “worst case” and the “typical case”, in order to get the exact expression for the convergence rates, the main technical challenge is to prove new and tight bounds for mutual covering. The main mutual covering lemma we prove suffices for both cases. The proof idea is to lower bound the probability that a real valued random variable is positive, just as in [2, Lemma 8.1], but with two important innovations:

• •

The original proof of mutual covering [2] bounded the probability that the sum of the indicator functions of covering events is positive. Here we use a weighted444In the asymptotic setting, the weights of the non-zero indicators are almost equal, so the weighted sum is approximately a scaled version of the number of typical events. sum instead, which results in a more compact (and slightly stronger) bound than the one-shot bound obtained by directly considering the plain sum of indicators [20]. Furthermore, the extension to the multivariate setting is straightforward.

• •

While the original proof of mutual covering [2] bounded the deviation of the sum of indicator functions from its mean via Chebyshev’s inequality, we bound the deviation using a one-sided version of the Talagrand inequality [21, Theorem 8.6] in addition to Chebyshev’s inequality. Using the Talagrand inequality, in the asymptotic setting ($U^{(n)}\sim P_{\sf U}^{\otimes n}$, $V^{(n)}\sim P_{\sf V}^{\otimes n}$) we observe that the probability of failure to cover converges to zero doubly exponentially in $n$ provided that $P_{\sf UV}^{\otimes n}(\mathcal{F})$ is bounded away from zero. This is a substantial improvement on the convergence rates of previous bounds.

The mutual covering bounds in this paper are easily extendable from the bivariate to the multivariate case, which is useful for $m$-user broadcast channel and multiple descriptions [2].

The remainder of the paper is organized as follows: Section II surveys previously published covering lemmas; Section III lists the main results of the paper, including new bounds for covering and the applications in network information theory and in sampling. Section IV discusses some extensions to multivariate and conditional settings. Section V onward are devoted to the proofs of the results.

II Previous Single-Shot Covering Bounds

In this section we survey previous single-shot covering lemmas, starting with the unilateral setting after which we discuss two mutual covering lemmas (for an earlier survey of non-asymptotic covering lemmas, see also [22]).

Lemma 1** (Unilateral covering lemma).**

[3]**. Let $P_{UV}$ be given. Let $U_{1},\dots,U_{M}\sim P_{U}$ and $V\sim P_{V}$ be independent. For any $\gamma>0$ and $\mathcal{F}\subseteq\mathcal{U}\times\mathcal{V}$,

[TABLE]

where $(U,V)\sim P_{UV}$.

The proof uses the inequality

[TABLE]

for $M,\alpha>0$ and $0\leq p\leq 1$, which is a standard tool in the achievability proof in rate distortion theory. In mutual covering, we no longer have a single random variable which we hope will be “covered” by at least one of $M$ independent realizations; instead we have $L$ independent realizations and the failure to cover event is

[TABLE]

The purpose of the mutual covering lemma is to find an upper bound to the probability of covering failure as a function of $\mathcal{F}$, and of $P_{UV}(\mathcal{F}^{c})$, the probability of failure under a nominal joint probability measure.

Lemma 2** (Mutual covering lemma).**

[6]** Let $P_{UV}$ be given. Let $U_{1},\dots,U_{M}$ and $V_{1},\dots,V_{L}$ be independent. For any $\delta,\gamma>0$ and $\mathcal{F}\subseteq\mathcal{U}\times\mathcal{V}$,

[TABLE]

A more “structured” but essentially similar proof based on $E_{\gamma}$-resolvability yields a bound similar to (7); see [6][12]. Taking $M=1$ and $\delta\downarrow 0$ in (7), we recover the one-shot unilateral covering lemma in [3]. A pleasing property of (7) is that it is linear in both probability terms, and hence the bound can be applied to the achievability proof of the broadcast channel in a similar manner as [3]. Despite the symmetry of (7), in the proof the roles of $U$ and $V$ are asymmetric, hence not easily extended to the multivariate case.

The original idea for the asymptotic mutual covering in [4] based on calculating the expected number of typical pairs and bounding the deviation by Chebyshev’s inequality can be also applied to the one-shot setting as shown in the following result.

Lemma 3** (Mutual covering lemma).**

[20]**. Given $P_{UV}$, $\mathcal{F}$, $(U_{1},\dots,U_{M})$ and $(V_{1},\dots,V_{L})$ as before, for any $\epsilon\in(0,1)$ we have

[TABLE]

*where the smooth mutual information (see e.g. [23]) is defined as *

[TABLE]

Yassaee et al. [24] proposed a general method to one-shot achievabilities based on likelihood encoders/decoders (that sample from the posterior rather than declaring the argument that maximizes it) and Jensen’s inequality. In [8], Yassaee applied this approach to the covering problems. In Theorem 7 we reproduce a bound presented by Yassaee in the conference presentation of [8] (not included in [8]).

Remark 1**.**

None of the mutual covering lemmas above is tight enough to show (2). In the stationary memoryless setting (3), these bounds only show that the left side of (2) is exponentially small in the block length $n$ (rather than doubly exponential as in (2)).

Remark 2**.**

Lemma 1 establishes (4) for $M=1$ or $L=1$ (by choosing $\gamma=0.001\log ML$, say). However, the existing mutual covering lemmas are not sufficient for establishing (4) for general $M$ and $L$. In particular, (7) establishes (4) only when $\min\left\{\frac{1}{L},\frac{1}{M}\right\}$ is negligible compared to $\mathbb{P}[\imath_{U;V}(U;V)\geq\log ML]$ (by taking $\gamma=0.001\log ML$ and $\delta=0.5(ML)^{0.999}$, say).

III Summary of the Main Results

III-A New Bounds for Mutual Covering

The proof of our new mutual covering lemma relies on two tools not used before in this context, namely, a weighted sum trick and a Talagrand concentration inequality. Instead of examining the full combined power of these tools, we first give a simpler version assuming a weaker quadruple-wise independence structure among the random variables, in a similar vein as (8). This limited independence setting is useful in the analysis of linear codes in which the codewords are generated from a random matrix and are not mutually independent.

Lemma 4** (Limited independence).**

Given $P_{UV}$, assume that $(U_{1},\cdots,U_{M},V_{1},\cdots,V_{L})$ satisfy

[TABLE]

Then for any $\mathcal{F}\subseteq\mathcal{U}\times\mathcal{V}$ and $\gamma>0$,

[TABLE]

The proof of Lemma 4 uses the weighted sum trick and is given in Section V.

Under a stronger independence structure, we replace the Chebyshev inequality in the proof of Lemma 4 with the Talagrand inequality, and get a tighter bound sufficient to yield both (2) and (4). This leads to the following lemma, which is the main result of the paper and is proved in Section V.

Lemma 5** (Full independence).**

Given $P_{UV}$, suppose that $(U_{1},\cdots,U_{M},V_{1},\cdots,V_{L})$ has the joint distribution

[TABLE]

then for any event $\mathcal{F}$ and $\gamma>0$,

[TABLE]

Remark 3**.**

The probability of failure to cover $\mathcal{F}$ can be optimized over all couplings $P_{UV}$ with marginals $P_{U}$ and $P_{V}$. This allows us to find a good joint distribution resulting in vanishing failure probability, which need not be the case for an arbitrary coupling $P_{UV}$.

III-B Double Exponential Convergence of the Covering Error

With the smooth mutual information defined in (9), for any $p\in[0,1]$ we have

[TABLE]

which follows from Lemma 5 by taking $\gamma=\log ML-I_{\infty}^{\epsilon}(U;V)$. Thus in an asymptotic setting where $\frac{1}{n}\imath_{U^{(n)};V^{(n)}}(U^{(n)};V^{(n)})$ satisfies the law of large numbers, the smooth mutual information can be approximated by the mutual information, and we see that the $\leq$ part of (2) holds. In particular, in the stationary memoryless case, we can derive the exact doubly exponential exponent (by finding an asymptotically matching converse):

Theorem 1**.**

Fix555When we derive single-letter expressions in the stationary memoryless settings from bounds in the single-shot settings, we will use the superscript $(n)$, as in $U^{(n)}$, to indicate an $n$-vector in $\mathcal{U}^{n}$; san-serif letters, as in $P_{\sf UV}$, are used in per-letter distributions. $P_{\sf UV}$, $R_{1},R_{2}>0$, $\epsilon\in(0,1)$. Let $U^{(n)}_{1},\dots,U^{(n)}_{M}\sim P^{\otimes n}_{\sf U}$ and $V^{(n)}_{1},\dots,V^{(n)}_{L}\sim P_{\sf V}^{\otimes n}$ be independent, where666Here and elsewhere, as usual the rounding of the right sides of (15) and (16) is not explicitly indicated.

[TABLE]

Then

[TABLE]

where the supremum is over all $\mathcal{F}\subseteq\mathcal{U}^{n}\times\mathcal{V}^{n}$ such that $P_{UV}^{\otimes n}(\mathcal{F})\geq 1-\epsilon$.

The proof of Theorem 1 is given in Section VI. As is well-known, doubly exponential convergence phenomenon plays a role in the analysis of various information-theoretic problems. (A very partial list of) examples include: error exponent in lossy compression (benefiting from the doubly exponential convergence in unilateral covering) [1]; second-order rate of Gelfand-Pinsker coding [25, (60)]; error exponent of random channel codes [26, (30)] exponent in channel synthesis [27] (building on the concentration of the soft-covering error probability, a.k.a. strong soft-covering lemma [12, Theorem 31]).

III-C Application in Error Exponents of Broadcast Channels

A classical application of the mutual covering lemma is the achievability proof for the broadcast channel [4],[2]. In the single-shot setting, consider a broadcast channel with marginal random transformations $P_{Y|X}$ and $P_{Z|X}$. An $(M_{1},M_{2})$-code consists of the following:

• •

Encoder maps an equiprobable message $(W_{1},W_{2})\in\{1,2,\dots,M_{1}\}\times\{1,2,\dots,M_{2}\}$ to an element in $\mathcal{X}$;

• •

Decoder 1 maps an element in $\mathcal{Y}$ to $\hat{W}_{1}\in\{1,2,\dots,M_{1}\}$. Similarly Decoder 2 maps an element in $\mathcal{Z}$ to $\hat{W}_{2}$.

The goal is to design the encoder and decoders to minimize the error probabilities $\mathbb{P}[\hat{W}_{k}\neq W_{k}]$, $k=1,2$.

Switching to the discrete memoryless setting, we are given the per-letter random transformations $P_{\sf Y|X}$ and $P_{\sf Z|X}$, and make the substitutions $P_{Y|X}\leftarrow P_{\sf Y|X}^{\otimes n}$ and $P_{Z|X}\leftarrow P_{\sf Z|X}^{\otimes n}$ in the above definition, where $n$ denotes the blocklength. A rate pair $(R_{1},R_{2})\in(0,\infty)^{2}$ is said to be achievable if there exists a sequence of $(n,M_{1n},M_{2n})$ codes such that

[TABLE]

and

[TABLE]

A two-auxiliary simplification of Marton’s inner bound [28] (reproduced in [2, Theorem 8.3]) states that $(R_{1},R_{2})$ is achievable if there exists a discrete distribution $P_{\sf UV}$ and a function $x\colon\mathcal{U}\times\mathcal{V}\to\mathcal{X}$ such that

[TABLE]

We briefly recall the proof strategy (informal): Generate $U$ and $V$ codebooks at rates $I({\sf U;Y})$ and $I({\sf V;Z})$ respectively. Sacrifice rates $I({\sf U;Y})-R_{1}$ and $I({\sf V;Z})-R_{2}$ in the two codebooks so that given any message, a jointly typical pair of codewords can be found, in view of (22) and the mutual covering lemma. Then a joint typicality rule ensures the $U$-codeword (resp. $V$-codeword) to be found at the Decoder 1 (resp. Decoder 2). We remark that (20)-(22) is known to be not tight for certain broadcast channels; the full Marton bound [28] contains a third auxiliary, an equivalent form of which was obtained by Liang and Kramer [29] (reproduced in [2, Theorem 8.4]) by introducing a common message and applying rate splitting. The proof by Liang and Kramer uses a conditional version of the mutual covering lemma, which we discuss in Section IV.

In this section we investigate the achievable error exponent in the broadcast channel, that is, a lower bound on $\liminf_{n\to\infty}\frac{1}{n}\log\frac{1}{\max_{k=1,2}\mathbb{P}[\hat{W}_{k}\neq W_{k}]}$. Our result can be seen as an error exponent counterpart of (20)-(22). Liang’s three-auxiliary inner bound [29][2, Theorem 8.4] uses a conditional version of mutual covering lemma. We discuss a single-shot version of such a conditional mutual covering lemma later in Lemma 6.

From the description above of the proof of Marton’s inner bound, one would expect that the error exponent for Decoder 1 comes from taking the minimum between the error exponent in the mutual covering part and the error exponent for the channel from $\mathcal{U}$ to $\mathcal{Y}$; one might naively imagine that former plays a nontrivial role. The surprising double exponential convergence phenomenon in Section III-B shows that this is not the case. This strictly improves the error exponent obtained by previous mutual covering bounds.

Theorem 2**.**

Consider a discrete memoryless broadcast channel with (per-letter) marginal random transformations $P_{\sf Y|X}$ and $P_{\sf Z|X}$. Let $P_{\sf UV}$ be an arbitrary discrete distribution, and $x\colon\mathcal{U}\times\mathcal{V}\to\mathcal{X}$ an arbitrary function. Let $(R_{1},R_{2})\in[0,\infty)^{2}$. Then there exists a code at rates $(R_{1},R_{2})$ achieving the following error exponent

[TABLE]

where $E_{0}(\theta,P_{\sf U},P_{\sf Y|U}):=\log\left(\sum_{y}\left(\sum_{u}P_{\sf U}(u)P_{\sf Y|U}^{\frac{1}{1+\theta}}(y|u)\right)^{1+\theta}\right)$ denotes Gallager’s function [30], and $P_{\sf Y|U}$ is induced by $P_{\sf Y|X}$, $P_{\sf UV}$ and $x(\cdot)$.

Note that by taking $\theta\to 0$, we have $E_{0}(\theta,P_{\sf U},P_{\sf Y|U})=\theta I({\sf U;Y})+o(\theta)$, and (26) can be matched to (20)-(22). We remark that slightly sharper versions are obtained in the proof of Theorem 2; we present a slightly weakened but simpler version in (26).

III-D Joint Distribution Simulation via Covering

Next, we investigate a new application of mutual covering in the setup of joint distribution simulation, in which we desire to select among $ML$ pairs of samples, generated according to some arbitrary distribution, a pair distributed according to (or closely in total variation distance) a pre-specified joint distribution. The simple relation between covering and distribution simulation, observed in [7], leads us to the following covering-sampling duality result, which shows the existence of a sampling scheme without explicit constructions, using bounds on tail probabilities, such as covering lemmas and concentration inequalities.

Theorem 3**.**

Suppose that $|\mathcal{U}\|\mathcal{V}|<\infty$, and let $P_{UV}$ be given. Consider any $U_{1},\dots,U_{M}$ and $V_{1},\dots,V_{L}$, not necessarily independent and can have any marginal distribution. Then

[TABLE]

where $\hat{M}\in\{1,\dots,M\}$ and $\hat{L}\in\{1,\dots,L\}$ are indices selected upon observing $U^{M}$ and $V^{L}$.

We give a simple proof of Theorem 3 in Section VIII, which differs than the proof in [7] based on linear programming duality. Of course, these proofs are essentially minimax duality results and are related.

Remark 4**.**

Although the cardinality of the alphabets does not appear in (27), we make such an assumption because the proof given in Section VIII uses Hall’s marriage theorem which requires finite alphabets (and is known to fail for some general alphabets). We also remark that $P_{\hat{M}\hat{L}|U^{M}V^{L}}$ can be constructed using off-the-shelf bipartite graph matching algorithms (see e.g. [31, Section 13.3]).

III-E Error Exponent and Second-Order Rate in Joint Distribution Simulation

In the special case of the joint distribution simulation setup in which the samples are independent and generated according to the desired marginals, we can leverage Lemma 5 and Theorem 3, to show the following information spectrum bound for the achievability of joint distribution simulation.

Theorem 4**.**

Given $P_{UV}$, let $U_{1},\dots,U_{M}\sim P_{U}$, any $p\in(0,1)$, and $V_{1},\dots,V_{L}\sim P_{V}$ be independent. If $M,L\geq\frac{3}{p}\ln\frac{1}{1-p}$ then

[TABLE]

The proof of Theorem 4 is given in Section IX-A. Ignoring the nuisance parameters, when $M$ and $L$ are large, the right side of (28) is essentially the information spectrum tail

[TABLE]

For example, for stationary memoryless $P_{UV}$, the first term in the $\max$ dictates the exponential behavior of the right side of (28).

Our proof via the duality result Theorem 3 is a novel contribution, since other more straightforward sampling schemes do not seem to achieve the exact exponent. For example, inspired by the likelihood encoder, [24], [8] proposed the a simple sampling scheme, which we refer to as the weighted sampler. In Section X however, we show that an analysis of the weighted sampler using the Jensen’s inequality argument of [24] gives an achievability bound of

[TABLE]

which is good enough to show that mutual information is the (first-order) fundamental limit but is exponentially worse than (29). Such exponential looseness cannot be overcome with a tighter bound: we can in fact determine the exact exponent for the weighted sampler in a certain regime, which is strictly worse than the exponent achieved by an optimal sampler in Theorem 4; see Section X for details on the analysis of the weighted sampler.

The following result is a converse counterpart to Theorem 4.

Theorem 5**.**

Given $P_{UV}$, let $U_{1},\dots,U_{M}\sim P_{U}$ and $V_{1},\dots,V_{L}\sim P_{V}$, all independent. Then for any random variables $\hat{M}\in\{1,\dots,M\}$ and $\hat{L}\in\{1,\dots,L\}$,

[TABLE]

The proof of Theorem 5 is given in Section IX-B, which also uses the duality result in Theorem 3.

In the stationary memoryless case, Theorem 4 and Theorem 5 imply that (29) is a tight approximation of the error in terms of the error exponent or the second-order analysis. In particular, we obtain the exact error exponent and the second-order rates in the nonvanishing error regime in the following corollaries.

Corollary 1**.**

Fix a discrete memoryless source $P_{\sf UV}$, $R_{1},R_{2}>0$. Let $U^{(n)}_{1},\dots,U^{(n)}_{M}\sim P^{\otimes n}_{\sf U}$ and $V^{(n)}_{1},\dots,V^{(n)}_{L}\sim P_{\sf V}^{\otimes n}$ be independent, where

[TABLE]

Then

[TABLE]

where

[TABLE]

denotes the Rényi divergence of order $\alpha$ (see e.g. [32]).

Proof sketch.

From the Chernoff bound we obtain

[TABLE]

where $P_{U^{(n)}V^{(n)}}:=P_{\sf UV}^{\otimes n}$. Moreover it is well-known in large deviation theory that this bound is asymptotically tight upon optimizing $\alpha$. The claim then follows from Theorem 4, since choosing any $0<\epsilon<p<1$ independent of $n$, we see that (29) is an exponentially tight approximation. ∎

Corollary 2**.**

Fix a discrete memoryless source $P_{\sf UV}$, and $R_{1},R_{2}>0$. Let $U^{(n)}_{1},\dots,U^{(n)}_{M_{n}}\sim P^{\otimes n}_{\sf U}$ and $V^{(n)}_{1},\dots,V^{(n)}_{L_{n}}\sim P_{\sf V}^{\otimes n}$ be independent, where $M_{n}$ and $L_{n}$ are positive integers such that

[TABLE]

for some $A\in\mathbb{R}$. Then

[TABLE]

where ${Q}(\cdot)$ denotes the complementary Gaussian CDF function:

[TABLE]

and the mutual varentropy is

[TABLE]

Proof sketch.

Pick any $\epsilon>Q\left(\frac{A}{V({\sf U;V})}\right)$ and $p\in(\epsilon,1)$, and apply Theorem 4. Observe that $\log M_{n}L_{n}-I_{\infty}^{\epsilon}(U^{(n)};V^{(n)})=\Omega(\sqrt{n})$ by the central limit theorem. Hence the first term in the max in (28) is the dominant term (the second term vanishes faster than exponentially). The claim follows by applying the central limit theorem to the first term in the max in (28). ∎

IV Extension to Multivariate Settings

The asymptotic multivariate covering lemma [2, Lemma 8.2] is a generalization of the mutual covering counterpart to the case of $k\geq 3$ codebooks. Given (single-letter distributions) $P_{{\sf V}_{1}\dots\sf V_{k}}$, suppose the rate of the $V_{k}$-codebook is $R_{k}$, then a typical tuple occurs with high probability if for every $\mathcal{S}\subseteq\{1,2,\dots,k\}$,

[TABLE]

Just as in the asymptotic case, the one-shot mutual covering lemmas (Lemma 4 and Lemma 5) can be extended to the multivariate setting by the same arguments. In some applications such as multiple description coding [2, Chapter 13] and broadcast channel with a common message [2, Theorem 8.4], a conditional version of (39) is useful, which states that given $P_{{\sf ZV}_{1}\dots\sf V_{k}}$, suppose the rate of the $V_{k}$-codebook is $R_{k}$ where the codewords are generated independently conditioned on $Z$, then a typical tuple occurs with high probability if

[TABLE]

see [2, Lemma 8.2]. We prove the following single-shot multivariate covering lemma, which implies (40) in the stationary memoryless setting.

Lemma 6**.**

Let $P_{ZV_{1}\cdots V_{k}}$ be a given joint distribution. Let $Z\sim P_{Z}$, and conditioned on $Z=z$, let $(V_{1}^{M_{1}},\cdots,V_{k}^{M_{k}})$ be mutually independent, and

[TABLE]

*then for any $\mathcal{F}\subseteq\mathcal{Z}\times\mathcal{V}_{1}\times\cdots\times\mathcal{V}_{k}$ and any $\gamma\in\mathbb{R}$, *

[TABLE]

where

[TABLE]

in which $V_{\mathcal{S}}:=(V_{i}:i\in\mathcal{S})$, and $\imath(Z,V_{\mathcal{S}})$ is a shorthand for777For probability measures $P$ and $Q$ on the same probability space, the relative information is $\imath_{P\|Q}(.):=\log\dfrac{\mathsf{d}P}{\mathsf{d}Q}(.)$.

[TABLE]

The proof of Lemma 6 is relegated to Section XI.

Remark 5**.**

Assume that $Z$ is constant in Lemma 6. Using steps similar to those in the proof of Theorem 1, it can be shown that the error probability (42) decays double exponentially in the blocklength. More precisely, assuming $M_{in}=\exp(nR_{i})$ and setting $P_{V_{i}}\leftarrow P^{\otimes n}_{{\sf V}_{i}}$ and $P_{V_{1}\cdots V_{k}}\leftarrow P^{\otimes n}_{{\sf V}_{1}\cdots{\sf V}_{k}}$ and denoting the left side of (42) by $P_{e}(\mathcal{F})$, its exact doubly exponential decay is given by

[TABLE]

where the supremum is over all $\mathcal{F}$ with $P^{\otimes n}_{{\sf V}_{1}\cdots{\sf V}_{k}}(\mathcal{F})\geq 1-\epsilon$, where $\epsilon\in(0,1)$ is independent of $n$. Note that the expression in $\min$ equals the inequality gap in (39). In particular, this shows that if strict inequality is achieved in (39) then the probability of non-covering is doubly exponentially small. However, the same is not true for general (not constant) $Z$: in the discrete memoryless setting with $P_{ZV_{1}\dots V_{k}}\leftarrow P_{{\sf ZV}_{1}\dots\sf V_{k}}^{\otimes n}$, we see that $P_{e}(\mathcal{F})$ decreases exponentially under (40), with the exponent $\min_{Q_{\sf Z}}D(Q_{\sf Z}\|P_{\sf Z})$ where the minimum is subject to the constraint

[TABLE]

V Proofs of Single-Shot Covering Lemmas

V-A Proof of Lemma 4

The blueprint follows El Gamal-van der Meulen’s proof of mutual covering lemma in the asymptotic case [4] with a slight but important change. Fix the subset of pairs $\mathcal{F}$ and let

[TABLE]

a random variable which can be viewed as a weighted sum of the indicator functions that the pairs belong to $\mathcal{G}$. In contrast, the proof in [4] uses non-weighted sum representing the total number of times the pairs belong to $\mathcal{G}$. We have

[TABLE]

where

[TABLE]

The remainder of the proof is devoted to showing that the variance of $S$ satisfies

[TABLE]

To that end, it is convenient to denote

[TABLE]

In view of the i.i.d. assumption in (10), we can write

[TABLE]

Note that here and after, the computations of the second-order statistics (variances) only used the partial independence (10), and full independence is not necessary. Next, we bound the four terms in the right side of (58) separately. First, we have

[TABLE]

where (61) is due to the change of measure and (62) follows from the definition of $\mathcal{G}$.

Next, consider

[TABLE]

where (66) follows from (10), (67) follows from

[TABLE]

and (68) follows by change of measure. Similarly, we have

[TABLE]

Finally, (10) implies

[TABLE]

Substituting (62), (68)–(71) into (58) yields (54).

Remark 6** (Rationale for the weighted sum).**

To use concentration inequalities to bound the probability of a non-negative random variable being zero, one can expect that the mean should be bounded away from zero. If we use a non-weighted sum (i.e. $S^{\prime}:=\frac{1}{ML}\sum_{m=1}^{M}\sum_{l=1}^{L}1\left\{(U_{m},V_{l})\in\mathcal{F}\right\}$), then $\mathbb{E}[S^{\prime}]=(P_{U}\times P_{V})(\mathcal{F})$ which is usually small in comparison to (53) for a good coupling $P_{UV}$.

V-B Proof of Lemma 5

In high dimensional probability, we often encounter situations where limited independence only gives a polynomial decay of the tail probability (e.g. Chebyshev’s inequality), whereas a full independence condition yields an exponential upgrade of the decay of the tail probability (e.g. sub-Gaussian concentration). This phenomenon also happens in our problem. We derive a sub-Gaussian bound for the probability in the right side of (50) by means of the following concentration inequality. Such concentration inequalities with one-side Lipschitz conditions were originally due to Talagrand [33][34]. However, a “modern” proof using Marton’s transportation method can be found in [21, Theorem 8.6].

Theorem 6** (Talagrand).**

Let $X_{1},\cdots,X_{n}$ be independent and assume that $f\colon\mathcal{X}^{n}\to\mathbb{R}$ is such that

[TABLE]

for some functions $c_{i}\colon\mathcal{X}^{n}\to\mathbb{R}$. Then, for all $t\geq 0$,

[TABLE]

Denote

[TABLE]

We proceed to specialize Theorem 6 to

• •

$n=M+L$,

• •

$f(u^{M},v^{L})=\sum_{m=1}^{M}\sum_{\ell=1}^{L}y_{m\ell}(u_{m},v_{l})$,

• •

$t=\mathbb{E}[f(U^{M},V^{L})]=\mathbb{E}[S]=\mathbb{P}[(U,V)\in\mathcal{G}]$,

• •

$c_{m}(u^{M},v^{L})=\sum_{\ell=1}^{L}y_{m,\ell}(u_{m},v_{l}),\quad\forall m\in\{1,\dots,M\},$

• •

$c_{-\ell}(u^{M},v^{L})=\sum_{m=1}^{M}y_{m,\ell}(u_{m},v_{l}),\quad\forall\ell\in\{1,\dots,L\}.$

We can verify that (72) is satisfied since

[TABLE]

Invoking (62) and (68), we get, for each $m$,

[TABLE]

Consequently,

[TABLE]

By the same argument,

[TABLE]

Finally, Theorem 6 yields

[TABLE]

as we wanted to show.

Remark 7**.**

The McDiarmid inequality (e.g. [35, Theorem 3.3.8]) has previously been applied to several information-theoretic problems including the strong soft-covering problem [12, Theorem 31]. However, the McDiarmid inequality bounds the variance proxy in terms of the $L^{\infty}$ norm of the random variables $\sum_{i}\left\|c_{i}^{2}(X^{n})\right\|_{\infty}$, hence it is too weak to yield any meaningful mutual covering bound. The Talagrand inequality improves the $L^{\infty}$ bound to a $L^{2}$ bound (see the denominator in (73)), while only bounding one side of the tail probability; but the one-sided bound is all we need for our purposes. The improvement to $L^{2}$ is only available when one considers the lower tail only.

Next we show a corollary of Lemma 5, whose asymptotic version was originally proved in [36] using Suen’s correlation inequality. Consider a bipartite regular graph associated with an $n$-type $P_{\sf UV}$, in which the left vertices represent the sequences $u^{n}$ with type $P_{\sf U}$ and the right vertices $v^{n}$ with type $P_{\sf V}$. The left and right sides have $A_{n}=2^{nH({\sf U})+o(n)}$ and $B_{n}=2^{nH({\sf V})+o(n)}$ vertices, respectively. The vertex $u^{n}$ is connected to $v^{n}$, if the pair $(u^{n},v^{n})$ has type $P_{\sf UV}$. The number of edges is $E_{n}=2^{nH({\sf UV})+o(n)}$.

Corollary 3**.**

If we draw equiprobably and independently (with replacement) $M_{n}$ and $L_{n}$ vertices from the left and right, respectively, the probability of drawing no connected pairs is upper bounded by

[TABLE]

Proof.

Let ${\sf U}^{n}$ and ${\sf V}^{n}$ be equiprobably distributed on the set of sequences with a type $P_{\sf UV}$. The claim follows by applying Lemma 5 with $U\leftarrow{\sf U}^{n}$, $V\leftarrow{\sf V}^{n}$, $\gamma\leftarrow\log\frac{M_{n}L_{n}E_{n}}{A_{n}B_{n}}$, $\mathcal{F}$ the set of sequences with type $P_{\sf UV}$, and $P_{UV}$ the equiprobable distribution on the $P_{\sf UV}$-type class. ∎

We remark that Suen’s correlation inequality [37] (see [38] for an improved version by Janson), while a useful tool in graph theory, cannot be used in place of the Talagrand inequality in our proof of the sharp mutual covering lemma: Suen’s inequality provides a lower bound on the probability that the sum of a collection of (possibly correlated) Bernoulli random variables is positive. In contrast, our proof of the sharp mutual covering lemma (which uses the weighted sum-trick) concerns the sum of real-valued (not necessarily Bernoulli) random variables.

VI Proof of the Double Exponential Convergence in Covering

In this section we give a proof of Theorem 1. To that end, we split the proof of (17) into the proof of the corresponding inequalities:

VI-A $\geq$* in (17)*

Assuming that $\mathcal{F}$ satisfies $P_{\sf UV}^{\otimes n}(\mathcal{F})\geq 1-\epsilon$, by Lemma 5 we have

[TABLE]

where $P_{U^{(n)}V^{(n)}}=P_{\sf UV}^{\otimes n}$. In view of assumptions (15) and (16), we can choose

[TABLE]

where $\tau\in(0,\infty)$ is arbitrary. Since the numerator in the fraction in (81) converges to $1-\epsilon$, we conclude that

[TABLE]

Therefore, $\geq$ holds in (17) since $\tau>0$ is arbitrary.

VI-B $\leq$* in (17)*

To prove this direction, we need to construct $\mathcal{F}$ for which the probability of covering failure is large enough. If $\mathcal{\bar{U}}\subseteq\mathcal{U}^{n}$ satisfies

[TABLE]

then $\mathcal{F}:=\mathcal{\bar{U}}\times\mathcal{V}^{n}$ also has large probability

[TABLE]

and

[TABLE]

Therefore,

[TABLE]

By symmetry, we have also shown that the left side of (17) is upper bounded by $R_{2}$.

It remains to show that whenever

[TABLE]

the left side of (17) is upper bounded by $R_{1}+R_{2}-I({\sf U;V})$. For sufficiently large $n$ we can pick $\mathcal{F}:=\{(u^{n},v^{n})\colon\,\sqrt{n}A\leq\sum_{i=1}^{n}\imath_{\sf U;V}(u_{i};v_{i})-nI({\sf U;V})\leq\sqrt{n}B\}$ for appropriately chosen $A,B\in\mathbb{R}$ independent of $n$ such that

[TABLE]

Then

[TABLE]

where (93) and (94) follow from Jensen’s inequality and the union bound, respectively. Note that $\lim_{n\to\infty}M_{n}\mathbb{P}[(U^{(n)}_{1},V^{(n)}_{1})\in\mathcal{F}]=0$ which follows from (89) and (91). Thus,

[TABLE]

as desired.

VII Proof of Broadcast Channel Error Exponent

In this section we prove Theorem 2 by first showing a stronger single-shot bound.

Lemma 7**.**

Consider a broadcast channel with marginal random transformations $P_{Y|X}$ and $P_{Z|X}$. Given any positive integers $M_{1}$, $M_{2}$, $N_{1}$, $N_{2}$, a joint distribution $P_{UV}$, functions

[TABLE]

and $\gamma\in\mathbb{R}$, $\theta_{1},\theta_{2}\in[0,1]$, there exists an $(M_{1},M_{2})$-code such that

[TABLE]

where $\mathcal{F}$ is defined as the set of $(u,v)\in\mathcal{U}\times\mathcal{V}$ satisfying

[TABLE]

with

[TABLE]

in which $P_{U,V,Y,\bar{U}}(u,v,y,\bar{u}):=P_{UV}(u,v)P_{Y|X=x(u,v)}(y)P_{U}(\bar{u})$, and

[TABLE]

in which $P_{U,V,Z,\bar{V}}(u,v,z,\bar{v}):=P_{UV}(u,v)P_{Z|X=x(u,v)}(z)P_{V}(\bar{v})$.

Note that the function $\phi_{k}$ depends on $\theta_{k}$, $k=1,2$. The functions $\tt g$ and $\tt h$ are also allowed to depend on $\theta_{k}$. There is no confusion in omitting the argument $\theta_{k}$ since $\theta_{k}$ is fixed in Lemma 7.

Proof Lemma 7.

Construct codebooks $\mathcal{C}=(U_{mn})_{1\leq m\leq M_{1},\,1\leq n\leq N_{1}}$ and $\mathcal{D}=(V_{mn})_{1\leq m\leq M_{2},\,1\leq n\leq N_{2}}$ where each codeword is independently generated according to $P_{U}$ and $P_{V}$ respectively.

Encoder: Given $(W_{1},W_{2})=(m_{1},m_{2})$, select (if any) $(K_{1},K_{2})=(n_{1},n_{2})$ such that $(U_{m_{1}n_{1}},V_{m_{2}n_{2}})\in\mathcal{F}$. Then the encoder sends $X=x(U_{m_{1}n_{1}},V_{m_{2}n_{2}})$. If there are more than one such pair of $(n_{1},n_{2})$, the encoder can pick any one of them; if there is no such pair, the encoder fails and outputs any element in $\mathcal{X}$.

Decoders: Upon receiving $Y$, Decoder 1 outputs

[TABLE]

with arbitrary tiebreaking. Decoder 2 outputs $(\hat{W}_{2},\hat{K}_{2})$ using a similar rule.

Error analysis: By the symmetry in the codebook construction, we can assume without loss of generality that $(W_{1},W_{2})=(1,1)$ and we will compute all probabilities below conditioned on this event. By Theorem 5,

[TABLE]

To analyze the failure probability at Decoder 1, note that for any $(u,v)\in\mathcal{F}$ and $y\in\mathcal{Y}$,

[TABLE]

where (106) is the union bound; (107) follows because from the codebook construction, $U_{mn}\sim P_{U}$ is independent of $(U_{1K_{1}},V_{1K_{2}},Y)$, for each $m\in\{2,\dots,M_{1}\}$ and $n\in\{1,\dots,N_{1}\}$. Since probability does not exceed 1, for $\theta_{1}\in[0,1]$ we obtain

[TABLE]

Next, note that $x(U_{1K_{1}},V_{1K_{2}})$ is the input signal for the broadcast channel. Integrating both sides of (109) with respect to $P_{Y|X=x(u,v)}$ we obtain

[TABLE]

where the last step used (101) and the assumption that $(u,v)\in\mathcal{F}$. Then the proof for Decoder 1 is completed by noting that

[TABLE]

and invoking (105). The proof for Decoder 2 is similar. ∎

Proof of Theorem 2.

The proof follows by specializing the single-shot bound Lemma 7 to the discrete memoryless setting: Let $S_{1},S_{2}$ be any positive numbers such that

[TABLE]

With the substitutions

[TABLE]

we see that $\mathcal{F}$ defined by (101) satisfies $\lim_{n\to\infty}P_{\sf UV}^{\otimes n}(\mathcal{F})=1$, and hence the first term on the right side of (113) vanishes doubly exponentially in $n$, and

[TABLE]

Let

[TABLE]

and restrict to $\theta_{1}=\theta_{2}=\theta$ (this restriction is simply to make the final expression simpler; it is possible to obtain a slightly stronger bound allowing $\theta_{1}$ and $\theta_{2}$ to differ). We see that there exists positive $S_{1},S_{2}$ satisfying (114) as long as

[TABLE]

Rearranging, we see that the following $E$ is achievable:

[TABLE]

The proof of the theorem follows by further weakening (126): By Jensen’s inequality,

[TABLE]

Furthermore, with ${\tt h}(u,y):=\exp\left(\frac{\imath_{\sf U;Y}(u;y)}{1+\theta}\right)$, the right side of (128) equals $E_{0}(\theta,P_{\sf U},P_{\sf Y|U})$. Similar simplifications applies to $\mathbb{E}[\phi_{2}({\sf U,V})]$, and the theorem is proved. ∎

VIII Proof of the Duality between Covering and Distribution Simulation

This section proves Theorem 3, invoking a combinatorial result, Hall’s marriage lemma. In fact, we show a more general claim:

Lemma 8**.**

Let $\mathcal{Z}$ be finite. Suppose that $Z_{1},\dots,Z_{N}$ are arbitrarily distributed random variables on $\mathcal{Z}$. Then, for any $P_{Z}$ on $\mathcal{Z}$,

[TABLE]

where the random variable $\hat{N}\in\{1,\dots,N\}$.

Proof.

The $\leq$ part is obvious since for any $\mathcal{F}$ and any $P_{\hat{N}|Z^{N}}$,

[TABLE]

The $\geq$ part can be shown via Hall’s marriage theorem, reproduced after the proof. We first prove the result assuming that $P_{Z^{N}}$ and $P_{Z}$ are $K$-type distributions (i.e. all the probabilities in the distribution are multiples of $1/K$) for an arbitrary integer $K$; the general result will then follow by taking $K\to\infty$ and an approximation argument.

Let $\epsilon$ denote the left side of (129), which is a multiple of $1/K$. Construct a bipartite graph with $K$ left vertices and $K(1+\epsilon)$ right vertices. Each $z^{N}$ is “split” into $KP_{Z^{N}}(z^{N})$ left vertices, that is, we use $KP_{Z^{N}}(z^{N})$ identical vertices to represent $z^{N}$, so that all the $K$ left vertices have equal probability $1/K$. Similarly, $z\in\mathcal{Z}$ is “split” into $KP_{Z}(z)$ right vertices. Moreover, construct $\epsilon K$ right vertices, each representing a redundant symbol which can be set to equal any element in $\mathcal{Z}$. The redundant symbols will represent the error events where we are allowed to produce any symbol. There is an edge between $(z_{1},\dots,z_{N})$ and $z$ if the latter is a coordinate of the former, and there is also an edge between each redundant symbol and each left vertex. An example is shown in Fig. 1. In this example, The sequence distribution is $P_{Z^{3}}(1,2,3)=\frac{2}{3}$ and $P_{Z^{3}}(1,1,3)=\frac{1}{3}$; the target distribution $P_{Z}$ is equiprobable on $\{1,2,3\}$. The lines indicate the permissible selection rules. For example, if the sequence $(x_{1},x_{2},x_{3})=(a,b,b)$, then the output can either be $a$ or $b$, hence there are lines connected to $a$ and $b$. After drawing such a bipartite graph, we solve a graph matching problem, and the solution are the bold lines.

By Hall’s marriage theorem (reproduced after the proof), we claim that there exists a matching that covers the left vertices if for any $\mathcal{S}\subseteq\mathcal{Z}^{N}$,

[TABLE]

Indeed, for any $\mathcal{W}$ a subset of left vertices, let $\mathcal{S}$ be the set of all $z^{N}$ such that $z^{N}$ corresponds to some vertex in $\mathcal{W}$. Then $|\mathcal{W}|\leq KP_{Z^{N}}(\mathcal{S})$ (not equality since $\mathcal{W}$ may only contain some, but not all, vertices corresponding to a sequence in $\mathcal{S}$). The neighborhood of $\mathcal{W}$ in the bipartite graph has cardinality $K\left(\epsilon+P_{Z}\left(\bigcup_{z^{N}\in\mathcal{S}}\{z\colon z\in z^{N}\}\right)\right)$. Now for any given $\mathcal{S}$, let

[TABLE]

We see that

[TABLE]

hence (131) holds by the definition of $\epsilon$. The matching then determines a rule of choosing a right vertex (and hence choosing $\hat{Z}$) upon observing $(z_{1},\dots,z_{N})$, such that $\frac{1}{2}|P_{Z_{\hat{N}}}-P_{Z}|$ is bounded by the probability of the redundant right vertices, which is $\epsilon$.

Finally we complete the approximation argument: for general $P_{Z^{N}}$ and $P_{Z}$, we can approximate them within any $\delta\in(0,1)$ total variation distance by some $K$-type distributions $P_{\tilde{Z}^{N}}$ and $P_{\tilde{Z}}$.

[TABLE]

By the preceding argument we find some $\hat{N}$ such that $\frac{1}{2}|P_{\tilde{Z}_{\hat{N}}}-P_{\hat{Z}}|\leq\epsilon+2\delta$. Applying the same selection rule to $P_{Z^{N}}$ and using the data processing inequality of the total variation distance, we have $\frac{1}{2}|P_{Z_{\hat{N}}}-P_{\hat{Z}}|\leq\epsilon+3\delta$. By the triangle inequality, $\frac{1}{2}|P_{Z_{\hat{N}}}-P_{Z}|\leq\epsilon+4\delta.$ The proof is completed since $\delta$ can be made arbitrarily small by choosing large enough $K$. ∎

Hall’s Marriage Theorem**.**

[39]** Consider a bipartite graph with bipartite sets $\mathcal{X}$ and $\mathcal{Y}$. There exists a matching that covers $\mathcal{X}$ (i.e. an injective map from $\mathcal{X}$ to $\mathcal{Y}$ such that any $x\in\mathcal{X}$ is mapped to an adjacent $y\in\mathcal{Y}$) if and only if for any $\mathcal{W}\subseteq\mathcal{X}$, the neighborhood of $\mathcal{W}$ (i.e. the set of elements in $\mathcal{Y}$ adjacent to some element in $\mathcal{X}$) has at least the cardinality of $\mathcal{W}$.

IX Proof of Simulation Error Bounds

This Section proves Theorem 4 and Theorem 5.

IX-A Achievability: Theorem 4

The achievability follows from the covering-sampling duality (Theorem 3). In the case of $P_{UV}(\mathcal{F})\geq p$ we can apply (14), so it remains to bound the $P_{UV}(\mathcal{F})<p$ case, which is accomplished by the following information spectrum bound.

Lemma 9**.**

Fix $P_{UV}$, $p\in[0,1)$, and $M,L\geq\frac{6}{p}\ln\frac{1}{1-p}$. Let $U_{1},\dots,U_{M}\sim P_{U}$ and $V_{1},\dots,V_{L}\sim P_{V}$ be independent. Then

[TABLE]

Proof.

Upon choosing $\gamma=\log\left(\frac{3}{p}\ln\frac{1}{1-p}\right)$, we obtain from Lemma 5 that

[TABLE]

where we have used the fact that $e^{-\frac{t}{p}\ln\frac{1}{1-p}}\leq 1-t$ for $t\in[0,p]$. ∎

Note that in the stationary memoryless case, $\liminf_{n\to\infty}P_{\sf UV}^{\otimes n}(\mathcal{F}_{n})>0$ implies that the covering error (the first term in the supremum in (135)) is doubly exponentially small. Therefore in the stationary memoryless case, if the blocklength $n$ is large enough, (135) continues to hold when the supremum is relaxed to all $\mathcal{F}$.

IX-B Converse: Theorem 5

Proof.

Take $\mathcal{F}:=\left\{\imath_{U;V}(U;V)>\log ML\right\}$. We have

[TABLE]

where (140) follows from [12, Proposition 13.1], which allows one to upper bound the supremum of (139) over $\mathcal{F}$ by the information spectrum. ∎

X Proof of Sub-optimality of Weighted Sampling

Section III-E shows the exact error exponent of joint distribution simulation, where the achievability part relies on a duality with the covering problem, without giving an explicit construction. In this section, we consider the performance of a simple weighted sampler, which was inspired by a likelihood encoder introduced by Yassaee et al. [24] in the context of achievability proofs in network information theory. Given the observations $U^{M}=u^{M}$ and $V^{L}=v^{L}$, the sampler chooses $(\hat{M},\hat{L})=(m,l)$ with probability

[TABLE]

In other words, the probability that a sample is selected is proportional to the ratio of the target distribution to the underlying distribution. The rationale of such a weighted sampler is reminiscent of, though not equivalent to, importance sampling in statistics. The purpose of analyzing such a weighted sampling scheme is two-fold:

• •

The weighted sampler is shown to give a strictly suboptimal error exponent. This justifies the duality approach in Section III-E based on covering lemmas as a singular contribution.

• •

The converse proof for the weighted sampler gives another application of Talagrand’s concentration inequality (Theorem 6).

X-A One-shot Achievability Bound

Theorem 7**.**

Fix $P_{UV}$. Let $U_{1},\dots,U_{M}\sim P_{U}$ and $V_{1},\dots,V_{L}\sim P_{V}$ be independent. Let $(U_{\hat{M}},V_{\hat{L}})$ be selected by the weighted sampling scheme. Then

[TABLE]

where $(U,V)\sim P_{UV}$.

Proof.

For each $(u,v)$, define

[TABLE]

For any event $\mathcal{F}$,

[TABLE]

where (145) follows from symmetry, (146) follows from Jensen’s inequality, and (147) follows from change of measure. Then

[TABLE]

where (149) follows by bounding the quantity in the expectation by 1 when $\imath_{U;V}(U;V)>\log ML-\gamma$, and by $\exp(-\gamma)$ otherwise. ∎

X-B Achievable Error Exponent for Weighted Sampling

The following result uses Theorem 7 to analyze the asymptotic decay of the total variation distance between the target joint probability measure and that achieved by the weighted sampler when each of the $M$ and $L$ observations are vectors of length $n$.

Theorem 8**.**

Fix $P_{\sf UV}$ on $\mathcal{U}\times\mathcal{V}$ and $R_{1},R_{2}>0$ such that $R_{1}+R_{2}\geq I({\sf U;V})$. For each $n$, define

[TABLE]

Let $U^{(n)}_{1},\dots,U^{(n)}_{M_{n}}\sim P_{\sf U}^{\otimes n}$ and $V^{(n)}_{1},\dots,V^{(n)}_{L_{n}}\sim P_{\sf V}^{\otimes n}$ be independent. Let $(U^{(n)}_{\hat{M}},V^{(n)}_{\hat{L}})\in\mathcal{U}^{n}\times\mathcal{V}^{n}$ be the pair of $n$-tuples selected by the weighted sampling scheme which aims to simulate $P_{\sf UV}^{\otimes n}$. Then,

[TABLE]

Proof.

We first observe that the single-shot bound (142) can be relaxed to the following

[TABLE]

where (154) used the simple inequality $\frac{1}{1+t}\leq t^{-\rho}$ for $t>0,\rho\in[0,1]$. The claim then follows by specializing to the stationary memoryless setting $P_{U^{(n)}V^{(n)}}\leftarrow P_{\sf UV}^{\otimes n}$. ∎

Remark 8**.**

Define the tilted distribution

[TABLE]

and denote

[TABLE]

where the information density $\imath_{\sf U;V}$ is defined with $P_{\sf UV}$. Optimizing the expression (152) we obtain the following solution

• •

If $0\leq R_{1}+R_{2}\leq R^{(1)}$, we can express the right side of (152) equivalently as

[TABLE]

where $\rho^{\star}\in[0,1]$ is the solution to

[TABLE]

• •

If $R_{1}+R_{2}>R^{(1)}$,

[TABLE]

Remark 9**.**

We did not use (143) in the proof of Theorem 8 because the step from (142) to (143) is not exponentially tight in the stationary memoryless setting. According to the analysis in Theorem 8, an exponentially tight approximation of (142) is

[TABLE]

In fact, by the Cramer’s large deviation theorem [40] in the asymptotic setting, we have the following exponentially tight approximation of (162)888We write $f(n)\doteq g(n)$ if $\frac{1}{n}\log f(n)-\frac{1}{n}\log g(n)\to 0$, $n\to\infty$.

[TABLE]

where in (163) we have used Sion’s min-max theorem, since the inside expression is convex in $\rho$ and linear in $\gamma$. Finally (164) is (155).

X-C A Converse on the Error Exponent of Weighted Sampling

Theorem 9**.**

Equality in (152) (see also (160)) holds if $R_{1}$ and $R_{2}$ satisfy

[TABLE]

where $P^{(2)}_{\sf UV}$ is the tilted distribution defined in (156).

Note that unless $I({\sf U;V})=0$, we have $R^{(1)}-D_{2}(P_{\sf UV}\|P_{\sf U}\times P_{\sf V})=D(P_{\sf UV}^{(2)}\|P_{\sf UV})>0$, and the $(R_{1},R_{2})$ region in Theorem 9 is nonempty. From the analysis in (158)-(161), we see that $\rho=1$ is achieved in the optimization in (152) for $(R_{1},R_{2})$ in this region, and hence the exponent in (152) is strictly worse than (34). This shows that the weighted sampler yields a fundamentally suboptimal exponent (not merely due to a weak achievability proof).

Remark 10**.**

In the context of channel coding, the likelihood decoder selects a codeword with probability proportional to the likelihood (analogous to the rule (141)). For rates below the capacity, a certain “$\alpha$-likelihood” decoder achieves the optimal random coding exponent when $\alpha\in[\alpha_{\tt c},\infty]$ where $\alpha_{\tt c}\in[0,1]$ depends on the given rate [41][42][43]. $\alpha=1$ corresponds to likelihood decoder and $\alpha=\infty$ corresponds to the maximum likelihood decoder. However, the situation for the current distribution simulation problem is entirely different: if we select $(m,n)$ according to an “$\alpha$-likelihood” rule, i.e. with probability proportional to $\exp(\alpha\imath_{U;V}(u_{m};v_{l}))$, then we will not be able to simulate a distribution close to $P_{UV}$ even with infinitely many codewords unless $\alpha=1$, hence we should never choose $\alpha\neq 1$. In contrast, in the channel coding problem the error probability is monotonically decreasing in $\alpha$, and $\alpha=\infty$ is optimal [43].

Proof of Theorem 9.

To simplify notation, we first consider the single-shot setting with a given joint distribution $P_{UV}$, and the claim will follow by taking $P_{UV}\leftarrow P_{\sf UV}^{\otimes n}$.

By the same arguments as the proof of Lemma 5, we can show the concentration inequality

[TABLE]

for any $t\geq 0$, where $Y_{m,l}=\exp(\imath_{U;V}(U_{m},V_{l}))$.

Now by splitting into two events, $\sum_{m=2,l=2}^{M,L}Y_{m,l}\geq ML-t$ and its complement, we have

[TABLE]

where for brevity $\epsilon$ denotes the right side of (167) and $(U,V)\sim P_{UV}$. Thus,

[TABLE]

Note that the right side is not necessarily maximized when $\mathcal{F}$ is the whole set since the numerator can be negative. Let $\gamma:=\log ML-R^{(2)}\geq 0$, which is exponentially equivalent to the optimal $\gamma$ in (162). Set

[TABLE]

By choosing

[TABLE]

we can ensure that the fraction in the expectation in the right side of (172) is at least $\exp(-\gamma)/2$ for $(U,V)\in\mathcal{F}$. Indeed, by monotonicity it suffices to verify by substituting $\imath_{U;V}(U;V)$ with $\log ML-\gamma$, in which case the choice of $t$ in (174) ensures that the fraction equals exactly $\exp(-\gamma)/2$.

[TABLE]

Finally, consider the asymptotic setting where we take $P_{UV}\leftarrow P_{\sf UV}^{\otimes n}$, $M\leftarrow\exp(nR_{1})$, and $L\leftarrow\exp(nR_{2})$, then

[TABLE]

Therefore $\epsilon$ vanishes doubly exponentially under the assumptions (165) and (166). The right side of (175) is thus exponentially equivalent to the right side of (162). ∎

XI Proof of the Multivariate Extension

This section proves Lemma 6. The idea is to analyze the following weighted sum for any instance $z$ using the Talagrand inequality and then taking the typical with respect to $P_{Z}$,

[TABLE]

where $\imath(z,v_{1},\cdots,v_{k}):=\imath_{P_{ZV^{k}}||P_{Z}\prod_{i=1}^{k}P_{V_{i}|Z}}(z,v_{1},\cdots,v_{k})$. We proceed with the same steps as in the proof of Lemma 5. First, observe that the mean of $S(z)$ is

[TABLE]

Define

[TABLE]

Next in Theorem 6, let $f(V_{1}^{M_{1}},\cdots,V_{k}^{M_{k}})=S(z)$ and

[TABLE]

Then, the one-sided Lipshitz property (72) is satisfied since

[TABLE]

The variance proxy in Talagrand’s inequality (i.e. the denominator in the (73)) is

[TABLE]

where the equality follows from symmetry.

Next we simplify the term inside the summation for $i=1$, (the simplification for $i>1$ is similar)

[TABLE]

where (180) is again follows from symmetry.

Next for each $(m_{1},\cdots,m_{k})$ satisfying $m_{t}=1,t\in\mathcal{S}$ and $m_{t}\neq 1,t\notin\mathcal{S}$, we have

[TABLE]

where

• •

(183) follows from the chain rule for information density.

• •

(184) follows from change of measure and the fact that conditioned on $Z=z$, $\{V_{i,m_{i}}:i\notin\mathcal{S}\}\sim\prod_{i\notin\mathcal{S}}P_{V_{i}|Z=z}$.

• •

(185) follows from the definition of $\mathcal{G}$ in (43).

• •

(186) follows from change of measure and the definition of $\imath(z,v_{1},\cdots,v_{k})$ in (44).

Invoking (181) and (186), we get

[TABLE]

Consequently, the variance proxy (179) is upper bounded by

[TABLE]

Finally, taking $t=\mathbb{E}[f(V_{1}^{M_{1}},\cdots,V_{k}^{M_{k}})]$ in Theorem 6, we obtain

[TABLE]

XII Conclusion

We derived a sharp mutual covering lemma (Lemma 5) and demonstrated its tightness in two regimes which we called “typical” and “worst case”. Our main result is presented in the single-shot form for simplicity and elegance, but the asymptotic sharpness really comes from the tailor-made concentration inequality and the novel duality argument in the proof. We showed two applications where the two regimes are the most useful, respectively: broadcast channel and joint distribution simulation.

XIII Acknowledgement

This work was supported in part by NSF Grants CCF-1016625, CCF-0939370, and by the Center for Science of Information. Jingbo Liu’s work was supported by the Wallace Memorial Fellowship at Princeton University.