The Method of Types for the AWGN Channel

TL;DR

This paper provides new ways to calculate error and decoding bounds for a communication channel using a method involving finite alphabets.

Contribution

The paper introduces alternative derivations for error and decoding bounds using the method of types with sub-exponential complexity.

Findings

Sphere-packing upper bound on the optimal block error exponent is derived using the method of types.

A lower bound on the optimal correct-decoding exponent is also derived using similar techniques.

The method uses finite alphabets with sizes dependent on block length n.

Abstract

For the discrete-time AWGN channel with a power constraint, we give an alternative derivation for the sphere-packing upper bound on the optimal block error exponent and an alternative derivation for the analogous lower bound on the optimal correct-decoding exponent. The derivations use the method of types with finite alphabets of sizes depending on the block length n and with the number of types sub-exponential in n.

Full text

1. Introduction

We study reliability of the discrete-time additive white Gaussian noise (AWGN) channel with a power constraint imposed on blocks of its inputs. Consider the capacity of this channel, found by Shannon:

$$[eqn]$$

where $[eqn]$ is the channel noise variance and $[eqn]$ is the power constraint. This capacity corresponds to the maximum of the mutual information $[eqn]$ over $[eqn]$ , under the power constraint on $[eqn]$ , where w stands for the channel transition probability density function (PDF) and $[eqn]$ is the channel input PDF. Let us briefly recall the technicalities [1] of how expression (1) is obtained from the mutual information:

$$[eqn]$$

Here $[eqn]$ and $[eqn]$ , the operator $[eqn]$ denotes the expectation, and D is the Kullback–Leibler divergence between two probability densities. The maximum in (2) is attained by the Gaussian PDF with zero mean and variance $[eqn]$ , which simultaneously gives $[eqn]$ and brings the divergence $[eqn]$ to zero, which is its lowest possible value [1] [Equation (8.57)]. In this paper we consider the optimal exponents in the block error/correct-decoding probability of the AWGN channel. We propose explanations, similar to (2), both for Shannon’s sphere-packing converse bound on the optimal error exponent [2] [Equations (3), (4) and (11)] (see also [3] [Equation (47)] with [4] [Equations (7.5.34) and (7.5.32)]) and for Oohama’s converse bound on the optimal correct-decoding exponent [5] [Equation (4)].

In the case of discrete memoryless channels, the mutual information enters into the expressions for correct-decoding and error exponents through the method of types [6,7]. For the moment, without any particular meaning attached to it, let us rewrite the sphere-packing constant-composition exponent [6] [Equation (5.19)] with the PDF’s:

$$[eqn]$$

where $[eqn]$ denotes the Gaussian density with variance $[eqn]$ , which maximizes (2), and $[eqn]$ is the information rate. When $[eqn]$ is Gaussian, the minimum (3) allows an explicit solution by the method of Lagrange multipliers. The minimizing solution $[eqn]$ of (3) is Gaussian [8] [Equation (65)], Ref. [9] [Equation (327)], and we obtain that (3) is the same as Shannon’s converse bound on the error exponent [2] [Equations (3), (4) and (11)], Refs. [8,9] in the limit of a large block length:

$$[eqn]$$

Then it turns out that $[eqn]$ of (3) and the y-marginal PDF of the product $[eqn]$ [8] [Equation (63)] play the same roles in the derivation of the converse bound as w and $[eqn]$ , respectively, in the maximization (2).

In this paper, in order to derive expressions similar to (3), we extend the method of types [1] [Chapter 11.1], Ref. [10] to include countable alphabets consisting of uniformly spaced real numbers, with the help of power constraints on types. The countable alphabets depend on the block length n and the number of types satisfying the power constraints is kept sub-exponential in n. The latter idea is inspired by a different subject—of “runs” in a binary sequence. If we treat every “run” of ones or zeros in a binary sequence as a separate symbol from the countable alphabet of run-lengths, then the number of different empirical distributions of such symbols in a binary sequence of length n is equivalent to the number of different partitions of the integer n into the sum of positive integers, which is $[eqn]$ [11] [Equation (5.1.2)]. Thus, it is sub-exponential, and the method of types can be extended to that case. In our present case, however, the types are empirical distributions of uniformly quantized real numbers in quantized versions of real channel input and output vectors of length n. The quantized versions serve only for classification of channel input and output vectors and not for the communication itself. The uniform quantization step is different for the quantized versions of channel inputs and outputs, and in both cases it is chosen to be a decreasing function of n.

Via expressions similar to (2), the proposed derivations demonstrate that, in order to achieve the converse bounds on the correct-decoding and error exponents, it is necessary for the types of the quantized versions of codewords to converge to the Gaussian distribution in the characteristic function (CF), or, equivalently, in cumulative distribution function (CDF).

The contributions of the current paper are twofold: we successfully apply the method of types to derive converse bounds on both the error exponent [12] and the correct-decoding exponent [13] of the AWGN channel. This underscores the advantage of the method of types.

In Section 2 and Section 3, we describe the communication system and introduce other definitions. In Section 4 we present the main results of the paper, which consist of two theorems and a proposition. Section 5 provides an extension to the method of types. The results of Section 5 are then applied in all the sections that follow. In Section 6 we prove a converse lemma that is then used for derivation of both the correct-decoding and error exponents in Section 7 and Section 8, respectively. Section 9 connects between the PDFs and types.

Notation

Countable alphabets consisting of real numbers are denoted by $[eqn]$ , $[eqn]$ . The set of types with denominator n over $[eqn]$ is denoted by $[eqn]$ . Capital ‘ $[eqn]$ ’ denotes probability mass functions, which are types $[eqn]$ , $[eqn]$ , $[eqn]$ , $[eqn]$ . The type class and the support of a type $[eqn]$ are denoted by $[eqn]$ and $[eqn]$ , respectively. The expectation with respect to a probability distribution $[eqn]$ is denoted by $[eqn]$ . Small ‘p’ denotes probability density functions $[eqn]$ , $[eqn]$ , $[eqn]$ , $[eqn]$ . Thin letters x, y represent real values, while thick letters $[eqn]$ , $[eqn]$ represent real vectors. Capital letters X, Y represent random variables; boldface $[eqn]$ represents a random vector of length n. The conditional type class of $[eqn]$ given $[eqn]$ is denoted by $[eqn]$ . The quantized versions of variables are denoted by a superscript ‘q’: $[eqn]$ , $[eqn]$ , $[eqn]$ . Small w stands for a conditional PDF, and $[eqn]$ stands for a discrete positive measure which does not necessarily add up to 1. All information-theoretic quantities such as joint and conditional entropies $[eqn]$ , $[eqn]$ , the mutual information $[eqn]$ , $[eqn]$ , $[eqn]$ , the Kullback–Leibler divergence $[eqn]$ , $[eqn]$ , and the information rate R are defined with respect to the logarithm to a base $[eqn]$ , denoted as $[eqn]$ . It is assumed that $[eqn]$ . The natural logarithm is denoted as ln. The cardinality of a discrete set is denoted by $[eqn]$ , while the volume of a continuous region is denoted by $[eqn]$ . The complementary set of a set A is denoted by $[eqn]$ . Logical “or” and “and” are represented by the symbols ∨ and ∧, respectively. Gaussian distributions are denoted by $[eqn]$ , while $[eqn]$ stands for the discrete uniform distribution. In the Appendix B, $[eqn]$ represents the rounded down version of the PDF $[eqn]$ .

2. Communication System

We consider communication over the time-discrete additive white Gaussian noise channel with real channel inputs $[eqn]$ and channel outputs $[eqn]$ and a transition probability density

$$[eqn]$$

Communication is performed by blocks of n channel inputs. Let $[eqn]$ denote a nominal information rate. Each block is used for transmission of one out of M messages, where $[eqn]$ , for some logarithm base $[eqn]$ . The encoder is a deterministic function $[eqn]$ , which converts a message into a transmitted block, such that

$$[eqn]$$

where $[eqn]$ , for all $[eqn]$ . The set of all the codewords $[eqn]$ , $[eqn]$ , constitutes a codebook $[eqn]$ . Each codeword $[eqn]$ in $[eqn]$ satisfies the power constraint:

$$[eqn]$$

The decoder is another deterministic function $[eqn]$ , which converts the received block of n channel outputs $[eqn]$ into an estimated message or, possibly, to a special error symbol ‘0’:

$$[eqn]$$

where each set $[eqn]$ is either an open region or the empty set, and the regions are disjoint: $[eqn]$ for $[eqn]$ . Observe that the maximum-likelihood decoder with open decision regions $[eqn]$ , defined for $[eqn]$ as

$$[eqn]$$

is a special case of (6). Note that the formal definition of $[eqn]$ includes the undesirable possibility of $[eqn]$ for $[eqn]$ .

3. Definitions

For each n, we define two discrete countable alphabets $[eqn]$ and $[eqn]$ as one-dimensional lattices:

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$

For each n, we define also a discrete positive measure (not necessarily a distribution), which will approximate the channel w:

$$[eqn]$$

Denoting by $[eqn]$ a class of functions $[eqn]$ continuous on an open subset $[eqn]$ , we define

$$[eqn]$$

The set (11), defined for a given n, will be used only in the derivation of the correct-decoding exponent, while the following set of Lipschitz continuous functions will be used only in the derivation of the error exponent:

$$[eqn]$$

Note that $[eqn]$ is a convex set and also each function $[eqn]$ is bounded and cannot exceed $[eqn]$ .

With a parameter $[eqn]$ , we define the following Gaussian probability density functions [8,9]:

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$

The first property of the following lemma shows that $[eqn]$ is the y-marginal PDF of the product $[eqn]$ .

Lemma 1(Properties of (13)–(18)). The following properties hold:

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$

and for any two jointly distributed random variables $[eqn]$ , such that $[eqn]$ , $[eqn]$ , and $[eqn]$ , it holds that

$$[eqn]$$ $$[eqn]$$

Here (17) combined with (14) corresponds to [8] [Equation (64)], (15) and (20) can be found in [8] [Equation (65)], while (14), (21) correspond respectively to [9] [Equations (302) and (328)].

Proof **of Lemma 1.**The first property (19) can be verified using (14), (15), (17). Then (20) can be obtained from (15), (17), (19). Property (21) follows by (17) and (19). It can be verified from (14) that $[eqn]$ is a positive monotonically decreasing function of $[eqn]$ , such that $[eqn]$ . Then we get (22) and (23). From (21) and (22) we see that $[eqn]$ for all $[eqn]$ , which gives (24). Equality (25) can be obtained using (21). Then, using (22) and (23), we obtain (26). □

The following expressions will describe our results for the error and correct-decoding exponents:

$$[eqn]$$ $$[eqn]$$

The following identity can be obtained using (13), (15), (16), (18) and (20):

$$[eqn]$$ $$[eqn]$$

We note also that $[eqn]$ , as $[eqn]$ , which can be verified using the properties (15), (17) and (21). It can be verified that the expression inside the supremum of (27) is equivalent to the expression for the Gaussian random-coding error exponent of Gallager before the maximization over $[eqn]$ [4] [Equations (7.4.24) and (7.4.28)]. Therefore, with the supremum over $[eqn]$ , the expression (27) coincides with the converse sphere-packing bound of Shannon (4).

4. Main Results

In this section we present two theorems and a proposition. The two theorems give converse bounds on the optimal error exponent and correct-decoding exponent of the AWGN, respectively. The bounds are asymptotic in the limit of a large block length n, and are given by the expressions (27) and (28), accordingly. The proofs, leading to these expressions, are also presented in this section, while some of their technical details are encapsulated in the form of lemmas that are taken care of by the rest of the sections in this paper. In the course of the two proofs leading to (27) and (28) we obtain expressions analogous to (2), which, in exactly the same manner as the maximization of (2), allow us to draw conclusions about the asymptotically optimal codeword types, achieving the converse bounds. This is made precise in the remark below, after the proof of the second theorem. The section is concluded with the proposition that brings together the bounds of the two theorems in a parametric form.

The proof of the first theorem relies on Lemmas 13 and 19, which appear in Section 7 and Section 9, respectively.

Theorem 1(Error exponent). Let $[eqn]$ be a random variable, independent of the channel noise, and let $[eqn]$ be the random channel-input and channel-output vectors, respectively. Then

$$[eqn]$$

where $[eqn]$ is defined by (27), decoder functions g are defined by (6), and codebooks $[eqn]$ satisfy (5).

Proof. Starting from Lemma 13, we can write the following sequence of inequalities:

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$ $$[eqn]$$

where(a) holds for any $[eqn]$ by Lemma 13 with $[eqn]$ . Note also that $[eqn]$ in (31) denotes the Kullback–Leibler divergence between the probability distribution $[eqn]$ and the positive measure $[eqn]$ defined in (10), which is not a probability distribution but only approximates the channel w.(b) follows by Lemma 19 for the alphabet parameters $[eqn]$ and $[eqn]$ .(c) holds for all $[eqn]$ with the possible exception of the single point on R-axis where (32) may transition between a finite value and $[eqn]$ . To verify the equality, let us compare the infimum in (32) and the supremum over $[eqn]$ in (33) as functions of $[eqn]$ , for a given $[eqn]$ . First, it can be verified that the supremum of (33) is the closure of the lower convex envelope of the infimum of (32). Second, it can be checked by the definition of convexity, that the infimum of (32) itself is a convex (∪) function of R. Then they coincide for all values of R, except possibly for the single point where they both jump to $[eqn]$ . This property carries over to the external ‘lim sup max’ as well.(d) follows because by (24) and (26) function $[eqn]$ satisfies the conditions under the infimum of (33).(e) holds as equality inside the supremum over $[eqn]$ , separately for each $[eqn]$ . In (34) by $[eqn]$ we denote the corresponding marginal PDF of the product $[eqn]$ and use the definitions (29). Then (e) follows by the definitions of $[eqn]$ and $[eqn]$ in (13) and (16), and by their properties (15) and (20).(f) follows by the non-negativity of the divergence, and by the condition under the maximum of (34), since $[eqn]$ for $[eqn]$ .In conclusion, according to (c) we obtain that the inequality between (30) and (35), as functions of R, holds for all $[eqn]$ , except possibly for the single point $[eqn]$ , where the jump to $[eqn]$ in (35) occurs. Therefore, taking the limit as $[eqn]$ , we obtain that (30) is upper-bounded for all $[eqn]$ by

$$[eqn]$$

which is the same as (27). □

The second theorem relies on Lemmas 17 and 20, which appear in Section 8 and Section 9, respectively.

Theorem 2(Correct-decoding exponent). Let $[eqn]$ be a random variable, independent of the channel noise, and let $[eqn]$ be the random channel-input and channel-output vectors, respectively. Then

$$[eqn]$$

where $[eqn]$ is defined by (28), decoder functions g are defined by (6), and codebooks $[eqn]$ satisfy (5).

Proof. Starting from Lemma 17, for each $[eqn]$ we can choose a different parameter $[eqn]$ , such that there is equality $[eqn]$ between (74) and (28). Then by (80) we obtain

$$[eqn]$$

With the choice $[eqn]$ , the first term in the minimum can be lower-bounded as follows:

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$

where:(a) follows by Lemma 20 with $[eqn]$ .(b) holds for $[eqn]$ , because $[eqn]$ for any such $[eqn]$ , and because $[eqn]$ , where $[eqn]$ is the Gaussian PDF defined in (16).(c) holds as an identity inside the infimum by the definitions (13), (16) and (29), and properties (15) and (20).(d) holds if $[eqn]$ and $[eqn]$ , because then by (26) and (11) the function $[eqn]$ satisfies the conditions under the infimum and achieves the infimum.(e) follows by the condition under the minimum of (36) since $[eqn]$ for $[eqn]$ .In conclusion, since (37) is the lower bound for any $[eqn]$ and $[eqn]$ , we obtain

$$[eqn]$$

□

Remark 1. Observe that neither the inequality (f) in the proof of Theorem 1 nor the inequality (e) in the proof of Theorem 2 can be met with equality unless $[eqn]$ . Furthermore, neither the inequality (f) in the proof of Theorem 1 nor the inequality (b) in the proof of Theorem 2 can be met with equality unless $[eqn]$ , where $[eqn]$ is the y-marginal PDF of $[eqn]$ . This is similar to (2). Accordingly, since $[eqn]$ is Gaussian, while $[eqn]$ is a convolution of $[eqn]$ with the Gaussian PDF $[eqn]$ , the type $[eqn]$ must converge to the Gaussian distribution with zero mean and variance $[eqn]$ in CF (it follows because the expression for the characteristic function of the zero-mean Gaussian distribution also has a Gaussian form) and CDF in order to achieve the exponents of Theorems 1 and 2. In both proofs the type $[eqn]$ represents the histograms of codewords, i.e., the empirical distributions of their quantized versions.

The functions $[eqn]$ and $[eqn]$ given by (27) and (28) can be expressed parametrically [4,5,8] as follows:

Proposition 1(Parametric representations of $[eqn]$ and $[eqn]$ ). For every $[eqn]$ there exists a unique $[eqn]$ , such that

$$[eqn]$$

For every $[eqn]$ there exists a unique $[eqn]$ , such that

$$[eqn]$$

The correct-decoding exponent representation of (38) is equivalent to [5] [Equation (22)] and appears in [14] [Equations (25) and (26)], while the error exponent representation of (39) is equivalent to [4] [Equations (7.4.30) and (7.4.31)] and appears in [8] [Equations (70) and (71)] and [9] [Equations (329) and (330)]. Here we present an alternative proof of Proposition 1 in the vein of the proofs of Theorems 1 and 2.

Proof. Let us denote $[eqn]$ . Then for $[eqn]$ we can write a sandwich proof:

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$

where $[eqn]$ denotes the set of all bivariate non-degenerate Gaussian PDF’s. Here (a) follows similarly to the inequality (b) in Theorem 2; (b) is an identity; (c) follows because $[eqn]$ is Gaussian and $[eqn]$ achieves the infimum; and (d) is a lower bound on the supremum at $[eqn]$ . Finally, since the RHS of (41) is further lower-bounded by the infimum (40), we conclude that $[eqn]$ .For $[eqn]$ , besides $[eqn]$ let us define $[eqn]$ . Then

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$

Here (a) follows due to the inequality under the first infimum; (b) is an identity; (c) follows because $[eqn]$ is Gaussian and $[eqn]$ achieves the infimum; and (d) follows because $[eqn]$ ; (e) is a lower bound on the supremum at $[eqn]$ . Since the RHS of (43) is lower-bounded by the infimum (42), we obtain $[eqn]$ . From $[eqn]$ using (17) and (21) we obtain $[eqn]$ . Hence for every $[eqn]$ the parameter $[eqn]$ is unique. □

5. Method of Types

In this section we extend the method of types [1] to include the countable alphabets of uniformly spaced reals (9) by using power constraints on the types. The method of types in the form of the results of this section is then used in the rest of the paper. It allows us to establish converse bounds in terms of types in Section 6, Section 7 and Section 8 and is used in the Appendix B dedicated to the proof of Lemma 18 of Section 9, connecting between PDFs and types.

5.1. Alphabet Size

Consider all the types $[eqn]$ satisfying the power constraint $[eqn]$ . Let $[eqn]$ denote the subset of the alphabet used by these types. In particular, every letter $[eqn]$ must satisfy $[eqn]$ , while by the definition of a type we have $[eqn]$ . This gives $[eqn]$ . Then $[eqn]$ is finite and by (7) we obtain

Lemma 2(Alphabet size). * $[eqn]$ .*

5.2. Size of a Type Class

For $[eqn]$ let us define

$$[eqn]$$

Lemma 3(Support of a joint type). Let $[eqn]$ be a joint type, such that $[eqn]$ and $[eqn]$ . Then

$$[eqn]$$

The proof is given in the Appendix A.

Lemma 4(Support of a type). Let $[eqn]$ and $[eqn]$ be types, such that $[eqn]$ and $[eqn]$ . Then

$$[eqn]$$

The proof for $[eqn]$ is given in the Appendix A.

For $[eqn]$ , the parameters $[eqn]$ , $[eqn]$ replace, respectively, $[eqn]$ , $[eqn]$ .

Lemma 5(Size of a type class). Let $[eqn]$ be a joint type, such that $[eqn]$ and $[eqn]$ . Then

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$

where $[eqn]$ , $[eqn]$ , and $[eqn]$ .

Proof. Observe that the standard type-size bounds (see, e.g., [6] [Lemma 2.3], Ref. [1] [Equation (11.16)]) can be rewritten as

$$[eqn]$$

Here $[eqn]$ can be replaced with its upper bound of Lemma 3. This gives (44). The remaining bounds of (45) and (46) are obtained similarly using Lemma 4. □

Since it holds for any $[eqn]$ that $[eqn]$ , and similarly for $[eqn]$ , as a corollary of the previous lemma we also obtain

Lemma 6(Size of a conditional type class). Let $[eqn]$ be a joint type, such that $[eqn]$ and $[eqn]$ . Then for $[eqn]$ and $[eqn]$ respectively

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ , $[eqn]$ , and $[eqn]$ are defined as in Lemma 5.

5.3. Number of Types

Let $[eqn]$ be the set of all the types $[eqn]$ satisfying the power constraint $[eqn]$ . Then its cardinality can be upper-bounded as follows:

$$[eqn]$$

where (a) follows by the definition of $[eqn]$ preceding Lemma 2, (b) follows by [1] [Equation (11.6)], and (c) follows by Lemma 2. This bound is sub-exponential in n for $[eqn]$ . This can be also further improved and made sub-exponential in n for all $[eqn]$ using Lemma 4, as follows.

Lemma 7(Number of types).

$$[eqn]$$

where $[eqn]$ and $[eqn]$ .

Proof. Denoting $[eqn]$ and $[eqn]$ , we can upper-bound as follows

$$[eqn]$$

Substituting for k and ℓ their upper bounds of Lemma 2 (with $[eqn]$ ) and Lemma 4, we obtain (51). □

Similarly, let $[eqn]$ denote the set of all the joint types $[eqn]$ , such that $[eqn]$ and $[eqn]$ . Then its cardinality can be bounded as follows.

Lemma 8(Number of joint types).

$$[eqn]$$

where $[eqn]$ and $[eqn]$ .

Proof. Denoting $[eqn]$ and $[eqn]$ , we repeat the steps of (52) and use the bounds of Lemma 2 and Lemma 3 to obtain (53). □

6. Converse Lemma

In this section we prove a converse Lemma 10, which is then used both for the error exponent in Section 7 and for the correct-decoding exponent in Section 8.

In order to determine exponents in channel probabilities, it is convenient to take hold of the exponent in the channel probability density. Let $[eqn]$ be a vector of n channel inputs and let $[eqn]$ be its quantized version, with components

$$[eqn]$$

Similarly, let $[eqn]$ be a vector of n channel outputs and let $[eqn]$ be its quantized version, with $[eqn]$ for all $[eqn]$ . Then we have the following

Lemma 9(PDF exponent). Let $[eqn]$ and $[eqn]$ be two channel input and output vectors, with their respective quantized versions $[eqn]$ , such that $[eqn]$ . Then

$$[eqn]$$

Proof. The exponent can be equivalently rewritten as

$$[eqn]$$

Defining $[eqn]$ , we observe that

$$[eqn]$$

The second term on the RHS is bounded as:

$$[eqn]$$

where (a) follows because $[eqn]$ , (b) follows by Jensen’s inequality for the concave (∩) function $[eqn]$ , and (c) follows by the condition of the lemma. The third term is bounded as

$$[eqn]$$

Since the exponent with the quantized versions $[eqn]$ , in turn, can also be rewritten similarly to (55), the result of the lemma follows by (55)–(58). □

The following lemma will be used both for the upper bound on the error exponent and for the lower bound on the correct-decoding exponent.

Lemma 10(Conditional probability of correct decoding). Let $[eqn]$ be a joint type, such that $[eqn]$ , $[eqn]$ , and $[eqn]$ , and let $[eqn]$ be a codebook, such that the quantized versions (54) of its codewords $[eqn]$ , $[eqn]$ , are all of the type $[eqn]$ , that is:

$$[eqn]$$

Let $[eqn]$ be a random variable, independent of the channel noise, and let $[eqn]$ be the random channel-input and channel-output vectors, respectively. Let $[eqn]$ . Then

$$[eqn]$$

where $[eqn]$ , and $[eqn]$ , as $[eqn]$ , depending only on α, β, $[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ .

Proof. First, from the single code $[eqn]$ we create an ensemble of codes, where each member code has the same probability of error/correct-decoding as the original code $[eqn]$ . Then we upper bound the ensemble average probability of correct decoding.Considering the codebook $[eqn]$ as an $[eqn]$ matrix, we permute its n columns. This produces a set of codebooks: $[eqn]$ , $[eqn]$ . The quantized versions of all the codewords of each codebook $[eqn]$ belong to the same type class $[eqn]$ . In accordance with $[eqn]$ , we permute also the n coordinates of each $[eqn]$ of the decision regions $[eqn]$ in the definition (6) of the decoder g, obtaining open sets $[eqn]$ and creating in this way an ensemble of codes $[eqn]$ , $[eqn]$ .Let $[eqn]$ denote the random channel-input and channel-output vectors, respectively, when using the code with an index $[eqn]$ . Let $[eqn]$ and $[eqn]$ denote their respective quantized versions. Since the additive channel noise is i.i.d., permutation of components does not change the distribution of the noise vector $[eqn]$ , and we obtain

$$[eqn]$$ $$[eqn]$$

Suppose that one of the codes $[eqn]$ , $[eqn]$ , is used for communication with probability $[eqn]$ , chosen independently of the sent message J and of the channel noise. Let $[eqn]$ be the random variable denoting the index of this code. Then, using (59) and (60) we obtain

$$[eqn]$$

In what follows, we upper bound the RHS of (61) with an added condition that $[eqn]$ :

$$[eqn]$$

The total number of codes in the ensemble can be rewritten as

$$[eqn]$$

Given $[eqn]$ , the total number of all the codewords in the ensemble such that their quantized versions belong to the same conditional type class $[eqn]$ (counted as distinct if the codewords belong to different ensemble member codes or represent different messages) is given by

$$[eqn]$$

Let $[eqn]$ denote the number of the codewords in a codebook $[eqn]$ such that their quantized versions belong to $[eqn]$ . Given that $[eqn]$ , the channel output vector $[eqn]$ falls into a hypercube region of $[eqn]$ :

$$[eqn]$$

For any $[eqn]$ such that $[eqn]$ and any open region $[eqn]$ , by Lemma 9 we obtain

$$[eqn]$$ $$[eqn]$$

Then, since all the codes and messages are equiprobable, the conditional probability of the code with the index ℓ is upper-bounded as

$$[eqn]$$

For $[eqn]$ , let $[eqn]$ be the indices of all the codewords in the codebook $[eqn]$ with their quantized versions in $[eqn]$ . Given that indeed the codebook $[eqn]$ has been used for communication, similarly to (65), by (64) the conditional probability of correct decoding can be upper-bounded as

$$[eqn]$$

where the second inequality follows because the decision regions $[eqn]$ are disjoint. Summing up over all the codes, we finally obtain:

$$[eqn]$$

where (a) follows by (65) and (66), (b) follows by (62) and (63), and (c) follows by (45) of Lemma 5 and (48) of Lemma 6. □

In the next two sections, we derive converse bounds on the error and correct-decoding exponents in terms of types.

7. Error Exponent

The end result of this section is given by Lemma 13 and represents a converse bound on the error exponent by the method of types.

Lemma 11(Error exponent of mono-composition codebooks). Let $[eqn]$ be a type, such that $[eqn]$ , and let $[eqn]$ be a codebook, such that the quantized versions (54) of its codewords $[eqn]$ , $[eqn]$ , are all of the type $[eqn]$ , that is:

$$[eqn]$$

*Let $[eqn]$ be a random variable, independent of the channel noise, and let $[eqn]$ be the random channel-input and channel-output vectors, respectively. Then for any parameter $[eqn]$ *

$$[eqn]$$

where $[eqn]$ , and $[eqn]$ , as $[eqn]$ , depending only on α, β, $[eqn]$ , $[eqn]$ , and $[eqn]$ .

Proof. For a joint type $[eqn]$ with the marginal type $[eqn]$ , such that $[eqn]$ , we have also

$$[eqn]$$

Then with $[eqn]$ for any $[eqn]$ , we obtain

$$[eqn]$$

where (a) follows by Lemma 9, and (b) follows by (49) of Lemma 6 and (10). This gives

$$[eqn]$$

Now we are ready to apply Lemma 10:

$$[eqn]$$

where (a) follows by (69) and Lemma 10, and (b) holds for $[eqn]$ . □

Lemma 12(Type constraint). For any $[eqn]$ there exists $[eqn]$ , such that for any $[eqn]$ and any codeword $[eqn]$ , satisfying the power constraint (5), the quantized version of that codeword, defined by (54), satisfies the power constraint (5) within ϵ, that is with $[eqn]$ replaced by $[eqn]$ .

The proof is the same as (56)–(58).

Lemma 13(Error exponent). *Let $[eqn]$ be a random variable, independent of the channel noise, and let $[eqn]$ be the random channel-input and channel-output vectors, respectively. Then for any $[eqn]$ and $[eqn]$ there exists $[eqn]$ , such that for any $[eqn]$ *

$$[eqn]$$

where $[eqn]$ , as $[eqn]$ , depending only on the parameters α, β, $[eqn]$ , $[eqn]$ , and $[eqn]$ .

Proof. For a type $[eqn]$ let us define $[eqn]$ . Then for any n greater than $[eqn]$ of Lemma 12, there exists at least one type $[eqn]$ such that

$$[eqn]$$

where the second inequality follows by Lemma 7 applied with $[eqn]$ . Then we can use such a type for a bound:

$$[eqn]$$ $$[eqn]$$

where (a) follows by (71), and (b) holds for the random variable $[eqn]$ , independent of the channel noise, and the channel input/output random vectors $[eqn]$ with the decoder

$$[eqn]$$

where $[eqn]$ are the indices of the codewords in $[eqn]$ with their quantized versions in $[eqn]$ .It follows now from (72) that the LHS of (70) can be upper-bounded by (67) of Lemma 11 with $[eqn]$ . Substituting then (71) in place of $[eqn]$ we obtain a stricter condition under the minimum of (67), leading to an upper bound with a condition $[eqn]$ and to (70). □

8. Correct-Decoding Exponent

The end result of this section is Lemma 17, which is a converse bound on the correct-decoding exponent by the method of types.

Lemma 14(Joint type constraint). For any $[eqn]$ and $[eqn]$ , there exists $[eqn]$ , such that for any $[eqn]$ and any pair of vectors $[eqn]$ satisfying

$$[eqn]$$

the respective quantized versions $[eqn]$ and $[eqn]$ , defined as in (54), satisfy

$$[eqn]$$

The proof is the same as (56)–(58).

We use a Chernoff bound for the probability of an event when the method of types cannot be applied:

Lemma 15(Chernoff bound). Let $[eqn]$ , $[eqn]$ , be n independent random variables. Then for $[eqn]$ and $[eqn]$ :

$$[eqn]$$

Lemma 16(Correct-decoding exponent of mono-composition codebooks). Let $[eqn]$ be a type, such that $[eqn]$ , and let $[eqn]$ be a codebook, such that the quantized versions (54) of its codewords $[eqn]$ , $[eqn]$ , are all of the type $[eqn]$ , that is:

$$[eqn]$$

*Let $[eqn]$ be a random variable, independent of the channel noise, and let $[eqn]$ be the random channel-input and channel-output vectors, respectively. Then for any $[eqn]$ and $[eqn]$ there exists $[eqn]$ , such that for any $[eqn]$ *

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$

where $[eqn]$ , and $[eqn]$ , as $[eqn]$ , depending only on α, β, $[eqn]$ , $[eqn]$ , and $[eqn]$ .

Proof. We consider probabilities of two disjoint events:

$$[eqn]$$

For the first term in the maximum, we obtain:

$$[eqn]$$ $$[eqn]$$

where (a) holds with $[eqn]$ for all $[eqn]$ of Lemma 14, (b) holds because $[eqn]$ , and in (c) we use the notation $[eqn]$ for the set of all the joint types $[eqn]$ satisfying both $[eqn]$ and $[eqn]$ . By the same steps as in (68), we further obtain

$$[eqn]$$

while Lemma 10 gives

$$[eqn]$$

Now by Lemma 8 for the number of joint types and by (77)–(79), we obtain

$$[eqn]$$

where $[eqn]$ denotes the expression (75). Applying Lemma 15 to the second term in the maximum of (76) we obtain (73)–(75). □

Lemma 17(Correct-decoding exponent). *Let $[eqn]$ be a random variable, independent of the channel noise, and let $[eqn]$ be the random channel-input and channel-output vectors, respectively. Then for any $[eqn]$ and $[eqn]$ there exists $[eqn]$ , such that for any $[eqn]$ *

$$[eqn]$$ $$[eqn]$$

where $[eqn]$ and $[eqn]$ are as defined in (74) and (75), respectively, and $[eqn]$ , as $[eqn]$ , depending only on the parameters α, β, $[eqn]$ , $[eqn]$ , and $[eqn]$ .

Proof. Similarly as in [7] [Lemma 5]:

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$

where:(a) holds for $[eqn]$ of Lemma 12;(b) follows by Lemma 7 with $[eqn]$ , while $[eqn]$ is a maximizer of (83);(c) holds for the channel input/output random vectors $[eqn]$ and a code $[eqn]$ , such that $[eqn]$ with $[eqn]$ for $[eqn]$ , and $[eqn]$ with $[eqn]$ for $[eqn]$ , where $[eqn]$ are the indices of the codewords in the original codebook $[eqn]$ with their quantized versions in $[eqn]$ . Since all the codewords of $[eqn]$ have their quantized versions in $[eqn]$ , we can apply Lemma 16 with $[eqn]$ for the RHS of (83) to obtain (80) and (81). □

9. PDF to Type

Lemmas 19 and 20 of this section relate between minimums over types and over PDF’s. The next Lemma 18, which has a laborious proof, is required only in the proof of Lemma 19, used for Theorem 1.

Lemma 18(Quantization of PDF). Let $[eqn]$ be an alphabet defined as in (9), (7) with $[eqn]$ . Let $[eqn]$ be a type and $[eqn]$ , $[eqn]$ , be a collection of functions from (12), such that $[eqn]$ , $[eqn]$ , and $[eqn]$ . Then for any alphabet $[eqn]$ defined as in (9), (7) with $[eqn]$ , there exists a joint type $[eqn]$ with the marginal type $[eqn]$ , such that

$$[eqn]$$ $$[eqn]$$ $$[eqn]$$

where $[eqn]$ , $[eqn]$ , and $[eqn]$ , as $[eqn]$ , and depends only on the parameters α, β, $[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ (through (12)).

The proof is given in the Appendix B.

Lemma 19(PDF to type). Let $[eqn]$ and $[eqn]$ be two alphabets defined as in (9), (7) with $[eqn]$ and $[eqn]$ . Then for any $[eqn]$ , $[eqn]$ , and $[eqn]$ there exists $[eqn]$ , such that for any $[eqn]$ and for any type $[eqn]$ with $[eqn]$ :

$$[eqn]$$

where $[eqn]$ , as $[eqn]$ , and depends only on the parameters α, β, $[eqn]$ , $[eqn]$ , and $[eqn]$ .

Proof. For a type $[eqn]$ with a collection of $[eqn]$ such that $[eqn]$ and $[eqn]$ , we can find also an upper bound $[eqn]$ on $[eqn]$ . For example, using the Cauchy–Schwarz inequality:

$$[eqn]$$

Then by Lemma 18 there exists a joint type $[eqn]$ with the marginal type $[eqn]$ , such that simultaneously the three inequalities (84)–(86) are satisfied and it also follows by (10) and (85) that

$$[eqn]$$

Then the sum of (84) and (88) gives

$$[eqn]$$

while the difference of (84) and (86) gives

$$[eqn]$$

Note that all $[eqn]$ in the above relations are independent of the joint type $[eqn]$ and the functions $[eqn]$ . Therefore by (89), (90) and (85) we conclude, that given any $[eqn]$ for n sufficiently large for every type $[eqn]$ with the prerequisites of this lemma and every collection of $[eqn]$ that satisfy the conditions under the infimum on the LHS of (87) there exists a joint type $[eqn]$ such that simultaneously

$$[eqn]$$

and (89) holds with a uniform $[eqn]$ , i.e., independent of $[eqn]$ and $[eqn]$ . It follows that such $[eqn]$ satisfies also the conditions under the minimum on the RHS of (87) and results in the objective function of (87) satisfying (89) with the uniform $[eqn]$ . Then the minimum itself, which can only possibly be taken over a greater variety of $[eqn]$ , satisfies the inequality (87). □

Lemma 20(Type to PDF). For any $[eqn]$ and $[eqn]$ there exists $[eqn]$ , such that for any $[eqn]$ and for any type $[eqn]$ :

$$[eqn]$$

where $[eqn]$ , as $[eqn]$ , and depends only on the parameters β and $[eqn]$ .

Proof. Observe first that any collection of conditional PDF’s $[eqn]$ that satisfies the conditions under the infimum of (91) has finite differential entropies and well-defined quantities $[eqn]$ and $[eqn]$ . For any conditional type $[eqn]$ from the LHS of (91) we can define a set of histogram-like conditional PDF’s:

$$[eqn]$$

which are step functions of $[eqn]$ for each $[eqn]$ . Then $[eqn]$ , and $[eqn]$ , as defined in (11). Analogously to (56)–(58), it can be obtained that

$$[eqn]$$

Then also $[eqn]$ . Then the lemma follows. □

We use Lemmas 19 and 20 in Section 4 in the derivation of Theorems 1 and 2, respectively.