Finite-Length Bounds for Joint Source-Channel Coding with Markovian Source and Additive Channel Noise to Achieve Large and Moderate Deviation Bounds

TL;DR

This paper establishes new finite-length bounds on error probabilities for joint source-channel coding with Markovian sources and channels, providing tight results in large and moderate deviation regimes.

Contribution

It introduces novel upper and lower bounds for error probability in joint source-channel coding with Markovian processes, covering large and moderate deviation regimes.

Findings

Bounds are tight in large deviation regimes

Bounds are tight in moderate deviation regimes

Applicable to ergodic Markov sources and channels

Abstract

We derive novel upper and lower finite-length bounds of the error probability in joint source-channel coding when the source obeys an ergodic Markov process and the channel is a Markovian additive channel or a Markovian conditional additive channel. These bounds are tight in the large and moderate deviation regimes.

Tables

Table 1. TABLE I: Comparison of upper bounds of decoding error probability for joint source-channel coding in the additive channel noise case

	Tight	Finite bound	Markov	Markov	Markov	Linear
	exponent	without	channel		conditional
	in IID case	polynomial factor	noise	source	additive channel	code
[4]	No	Yes	No	No	No	No
[9]	Yes	No	No	No	No	No
[7, 6, 5]	Yes	Yes	No	No	No	No
[8] +[3]	Yes	No	Yes	Yes	No	Yes
Proposed	Yes	Yes	Yes	Yes	Yes	Yes

Table 2. TABLE II: Comparison of lower bounds of decoding error probability for joint source-channel coding in the additive channel noise case

	Tight	Finite bound	Markov	Markov	Markov
	exponent	without	channel		conditional
	in IID case	polynomial factor	noise	source	additive channel
[4]	No	Yes	No	No	No
[9]	Yes	No	No	No	No
[3]	Yes	No	Yes	Yes	No
Proposed	Yes	Yes	Yes	Yes	Yes

Table 3. TABLE III: Summary of results.

	Channel	Finite	LD	MD	Complexity
Direct	Ass. 1	Theorem 1	Theorem 5	Theorem 9	$𝒪\left(1\right)$
	Ass. 2	Theorem 3	Theorem 7 (Tight)	(Tight)	$𝒪\left(1\right)$
Converse	Ass. 1	Theorem 2	Theorem 6	Theorem 9	$𝒪\left(1\right)$
	Ass. 2	Theorem 4	Theorem 8 (Tight)	(Tight)	$𝒪\left(1\right)$

Equations

$$H_{1-\theta }(P_{XY}|Q_Y):=\frac{1}{\theta }\log \sum _{x,y}{P}_{XY}(x,y{)}^{1-\theta }{Q}_Y(y{)}^{\theta }.$$

$$H_{1-\theta }^{\downarrow }(X|Y)$$

$$H_{1-\theta }^{\uparrow }(X|Y)$$

$$P_Y^{1-\theta }(y):=\frac{[{\sum }_x{P}_{XY}(x,y{)}^{1-\theta }{]}^{\frac{1}{1-\theta }}}{{\sum }_{y^{\prime }}[{\sum }_x{P}_{XY}(x,y^{\prime }{)}^{1-\theta }{]}^{\frac{1}{1-\theta }}}.$$

$$H_{1-\theta ,1-{\theta }^{\prime }}(X|Y):=H_{1-\theta }(P_{XY}|P_Y^{1-{\theta }^{\prime }}).$$

$$D_{1+s}(P||Q):=\frac{1}{s}\log \sum _{x}P(x{)}^{1+s}Q(x{)}^{-s}.$$

$$I_{1-s}^{\downarrow }(X;Y|P_{XY}):=$$

$$I_{1-s}^{\uparrow }(X;Y|P_{XY}):=$$

$$\sum _{x}W(x,y|x^{\prime },y^{\prime })=W(y|y^{\prime }),$$

$$W_{\theta }(y|y^{\prime })=\sum _{x}W(x,y|x^{\prime },y^{\prime }{)}^{1-\theta }.$$

$$H_{1-\theta }^{W|V}(X|Y):=\frac{1}{\theta }\log {\lambda }_{\theta }^{W|V},$$

$$W(x,y|x^{\prime },y^{\prime }{)}^{1-\theta }V(y|y^{\prime }{)}^{\theta }.$$

$$H_{1-\theta }^{W,\downarrow }(X|Y):=H_{1-\theta }^{W|W_Y}(X|Y).$$

$$H_{1-\theta }^{W,\uparrow }(X|Y):=\underset{V}{\max }{H}_{1-\theta }^{W|V}(X|Y).$$

$$N_{\theta ,{\theta }^{\prime }}(y|y^{\prime }):=W_{\theta }(y|y^{\prime })W_{{\theta }^{\prime }}(y|y^{\prime }{)}^{\frac{\theta }{1-{\theta }^{\prime }}},$$

$$H_{1-\theta ,1-{\theta }^{\prime }}^W(X|Y):=\frac{1}{\theta }\log {\nu }_{\theta ,{\theta }^{\prime }}-\frac{{\theta }^{\prime }}{1-{\theta }^{\prime }}{H}_{1-{\theta }^{\prime }}^{W,\uparrow }(X|Y).$$

$$H^W(X|Y):=\underset{\theta \to 0}{\lim }{H}_{1-\theta }^{W,\downarrow }(X|Y).$$

$$V^W(X|Y):=\underset{\theta \to 0}{\lim }\frac{2[H_{1-\theta }^{W,\downarrow }(X|Y)-H^W(X|Y)]}{\theta }.$$

$$H_{1-\theta }^{W,\downarrow }(X|Y)=H^W(X|Y)+\frac{\theta }{2}{V}^W(X|Y)+o(\theta ),$$

$$H_{1-\theta }^{W,\uparrow }(X|Y)=H^W(X|Y)+\frac{\theta }{2}{V}^W(X|Y)+o(\theta )$$

$$(n-1)\theta H_{1-\theta }^{W,\downarrow }(X|Y)+{\underline{\delta }}_W(\theta )\le \theta H_{1-\theta }^{\downarrow }(X^n|Y^n)\le (n-1)\theta H_{1-\theta }^{W,\downarrow }(X|Y)+{\overline{\delta }}_W(\theta ),$$

$${\overline{\delta }}_W(\theta ):=\log v_{\theta }\cdot w_{\theta },$$

$${\underline{\delta }}_W(\theta ):=\log v_{\theta }\cdot w_{\theta }-\log \underset{x,y}{\max }{v}_{\theta }(x,y).$$

$$(n-1)\frac{\theta }{1-\theta }{H}_{1-\theta }^{W,\uparrow }(X|Y)+{\underline{\xi }}_W(\theta )\le \frac{\theta }{1-\theta }{H}_{1-\theta }^{\uparrow }(X^n|Y^n)\le (n-1)\frac{\theta }{1-\theta }{H}_{1-\theta }^{W,\uparrow }(X|Y)+{\overline{\xi }}_W(\theta ),$$

$$K_{\theta }(y|y^{\prime }):=[\sum _{x}W(x,y|x^{\prime },y^{\prime }{)}^{1-\theta }{]}^{\frac{1}{1-\theta }},$$

$$w_{\theta }(y)={\left[\sum _x{P}_{X_1{Y}_1}(x,y{)}^{1-\theta }\right]}^{\frac{1}{1-\theta }}.$$

$${\overline{\xi }}_W(\theta ):=\log v_{\theta }\cdot w_{\theta },$$

$${\underline{\xi }}_W(\theta ):=\log v_{\theta }\cdot w_{\theta }-\log \underset{y}{\max }{v}_{\theta }(y).$$

$${\overline{\xi }}_W(\theta ):=\log v_{\theta }\cdot w_{\theta },$$

$${\underline{\xi }}_W(\theta ):=\log v_{\theta }\cdot w_{\theta }-\log \underset{x}{\max }{v}_{\theta }(x).$$

$$(n-1)\theta H_{1-\theta ,1-{\theta }^{\prime }}^W(X|Y)+{\underline{\zeta }}_W(\theta ,{\theta }^{\prime })\le \theta H_{1-\theta ,1-{\theta }^{\prime }}(X^n|Y^n)\le (n-1)\theta H_{1-\theta ,1-{\theta }^{\prime }}^W(X|Y)+{\overline{\zeta }}_W(\theta ,{\theta }^{\prime })$$

$$w_{\theta ,{\theta }^{\prime }}(y):=\left[\sum _x{P}_{X_1{Y}_1}(x,y{)}^{1-\theta }\right]{\left[\sum _x{P}_{X_1{Y}_1}(x,y{)}^{1-{\theta }^{\prime }}\right]}^{\frac{\theta }{1-{\theta }^{\prime }}}.$$

$${\overline{\zeta }}_W(\theta ,{\theta }^{\prime }):=$$

Full text

I Introduction

Shannon theoretic information theory originally focuses on the asymptotic performance. Since the block length of any real code is finite, analysis with finite-blocklength is more important in a practical setting. Although the tight analysis is possible in the asymptotic regime, it is almost impossible in the finite-length regime. Hence, we usually take a strategy to find good upper and lower bounds of the decoding error probability in the finite-length regime. Since lower and upper bounds are not unique, we need several requirements for the bounds to clarify their goodness. One is the asymptotic tightness. That is, we impose the first condition that the limit of the bound attains one of the following regimes: (1) Second order, (2) Moderate deviation, and (3) Large deviation.

To satisfy the above requirement, one may use the minimum value with respect to so many parameters. If the calculation complexity for the bound is too huge, it cannot be used in a practical use because we cannot calculate the bound. To estimate the optimal performance for a given blocklength $n$, we need to impose the second condition that its calculation complexity is not so large, e.g., $O(1)$, $O(n)$, or $O(n\log n)$.

Usually, the channel coding is discussed with the message subject to the uniform distribution. However, in the real communication, the message is not necessarily subject to the uniform distribution. To resolve this problem, we often consider the channel coding with the message subject to the non-uniform distribution. Such a problem is called source-channel joint coding and has been actively studied by several researchers [9, 5, 8, 2, 4, 3].

As a simple case, we often assume that the message is subject to an independent and identical distribution. In this case, the capacity is given as the ratio of the conventional channel capacity to the entropy of the message. Recently, Wang-Ingber-Kochman [2] and Kostina-Verdú [4] discussed the second-order coefficient in this problem. In the same setting, the papers [9, 7, 6, 5] derived the exponential decay rate of the minimum decoding error probability when the information source is subject to an independent and identical distribution and the channel is a discrete memoryless channel. Now, we focus on the case when the information source obeys a Markovian process and the channel is affected by additive noise that simply obeys Markovian process. In this setting, the paper [8] derived a lower bound of the exponential decreasing rate of the minimum decoding error probability, and the paper [3] derived the moderate deviation of the same error probability. That is, their direct part [3] follows from the idea of the paper [8], and their converse part [3] follows from their new idea. However, they did not derived a finite-length bound without polynomial overhead.

The recent paper [1] discussed the channel coding when the distribution of the additive noise in the channel is decided by the channel state, and the channel state is observed by the receiver and is subject to Markovian process. Such a channel is called a conditional additive channel. For example, Gilbert-Elliot channel with state-information available at the receiver is written as a special case of the former setting, but cannot be written as a special case of the latter setting. Hence, it is needed to treat a conditional additive channel to adopt a more realistic situation. In this paper, we focus on two kinds of assumptions (Assumptions 1 and 2) for such generalized additive noise channels. Under these assumptions for channels, we address joint source-channel coding with Markovian source and conditional additive channel noise.

As summarized in Tables I and II, the contribution of this paper is the following two points. One is to derive large and moderate deviation bounds under the above general setting, which are the generalizations of the results by the papers [8, 3]. The other is to derive upper and lower bounds with computable forms of the decoding error probability that match in the large deviation regime in the above general setting while the papers [8, 3] did not give finite-length bounds in a computable form in our sense.

The remaining part of this paper is organized as follows. In Section II, we prepare several information quantities for Markovian process. Section III prepares several useful functions for finite-length analysis. Section IV explains several useful lemmas under the single shot setting. Section IV shows our main results, i.e., our finite-length bounds and large and moderate deviation bounds. Section VI gives our numerical analysis based on our finite-length bounds. Table III explain the summary of our results.

II Information Measures for two terminals

In this section, we introduce some information measures and their properties will be used in latter sections.

II-A Information measures for single-shot setting

Since this paper addresses finite-length setting and the large deviation analysis, we need the conditional Rényi entropy. When the joint distribution is given to be $P_{XY}$ the conditional Rényi entropy relative to $Q_{Y}$ is given as

[TABLE]

Dependently of the choice fo the distribution $Q_{Y}$, we have the upper and lower types of conditional Rényi entropy:

[TABLE]

where

[TABLE]

To connect these two types of conditional Rényi entropy, we often focus on the following type of conditional Rényi entropy

[TABLE]

For $P,Q\in{\cal P(X)}$, we define Rényi divergence

[TABLE]

Using Rényi divergence, we introduce two types of Rényi mutual informations

[TABLE]

II-B Information measures for transition matrix

Since this paper address the Markovian information source, we prepare several information measures given in [1] for an ergodic and irreducible transition matrix $W=\{W(x,y|x^{\prime},y^{\prime})\}_{(x,y),(x^{\prime},y^{\prime})\in({\cal X\times Y})^{2}}$ on $({\cal X}\times{\cal Y})$. For this purpose, we employ two assumptions on transition matrices, which were introduced by the paper [1].

Definition 1** (Assumption 1 (non-hidden)).**

We assume the following condition for a transition matrix $W$:

[TABLE]

for every $x^{\prime}\in{\cal X}$ and $y,y^{\prime}\in{\cal Y}$.

When this condition holds, a transition matrix $W$ is called non-hidden (with respect to ${\cal Y}$).

Definition 2** (Assumption 2).**

We assume one of the following conditions for a transition matrix $W$:

for every $\theta\in(-\infty,0)$ and $(y,y^{\prime})\in{\cal Y}\times{\cal Y}$,

[TABLE]

is well defined, i.e., the right hand side of (10) is independent of $x^{\prime}$.

When this condition holds, a transition matrix $W$ is called strongly non-hidden (with respect to ${\cal Y}$). 2. 2.

$|{\cal Y}|=1$.

When this condition holds, a transition matrix $W$ is called singleton.

Assumption 1 is acquired from (10) by substituting $\theta=0$, so Assumption 2 implies Assumption 1. When a transition matrix on $W$ satisfies Assumption 1, we define the marginal $W_{Y}$ by $W_{Y}(y|y^{\prime}):=\sum_{x}W(x,y|x^{\prime},y^{\prime})$. For the transition matrix $T$ on ${\cal Y}$, we also define ${\cal Y}^{2}_{T}:=\{(y,y^{\prime}):T(y|y^{\prime})>0\}$. Then, when another transition matrix $V$ on ${\cal Y}$ satisfies ${\cal Y}^{2}_{W_{Y}}\subset{\cal Y}^{2}_{V}$, we define

[TABLE]

where $\lambda_{\theta}^{W|V}$ is the Perron-Frobenius eigenvalue of

[TABLE]

Then, the lower type of conditional Rényi entropy for the transition matrix [1] is given as

[TABLE]

Also, when $W$ satisfies Assumption 2, the upper type of conditional Rényi entropy for the transition matrix [1] is given as

[TABLE]

Furthermore, we define the information measure which is counterpart of (5). For this purpose, we introduce the following $|{\cal Y}|\times|{\cal Y}|$ matrix:

[TABLE]

where $W_{\theta}(y|y^{\prime})$ is defined in (10). Let $\nu_{\theta,\theta^{\prime}}$ be the Perron-Frobenius eigenvalue of $N_{\theta,\theta^{\prime}}$. Then, we define the two-parameter conditional Rényi entropy [1] by

[TABLE]

For $\theta=0$, we define the conditional Rényi entropy for $W$ by

[TABLE]

Also, we define following quantity.

[TABLE]

According to [1], using (17) and (18), we obtain the following two expansions.

[TABLE]

around $\theta=0$.

Under these preparations, we have three lemmas as follows.

Proposition 3**.**

[1, lemma 9]** Suppose that a transition matrix W satisfies Assumption 1. Let $W_{\theta}(x,y):=W(x,y|x^{\prime},y^{\prime})^{1-\theta}W(y|y^{\prime})^{\theta}$ and $\it{v}_{\theta}$ be the eigenvector of $W_{\theta}^{T}$ with respect to the Perron-Frobenius eigenvalue $\lambda_{\theta}$ such that $\min_{x,y}\it{v}_{\theta}(x,y)=1$. Let $\it{w}_{\theta}(x,y)=P_{X_{1}Y_{1}}(x,y)^{1-\theta}P_{Y_{1}}(y)^{\theta}$. Then, we have

[TABLE]

where

[TABLE]

Proposition 4**.**

[1, lemma 10]** Suppose that a transition matrix W satisfies Assumption 2. Then, we have

[TABLE]

where $\overline{\xi}_{W}(\theta)$ and $\underline{\xi}_{W}(\theta)$ is defined as follows:

For the non-hidden case, we define the $|{\cal Y}|\times|{\cal Y}|$ matrix $K_{\theta}$ so that

[TABLE]

and $\it{v}_{\theta}$ be the eigenvector of $K_{\theta}^{T}$ with respect to the Perron-Frobenius eigenvalue $\kappa_{\theta}$ such that $\min_{y}\it{v}_{\theta}(y)=1$. Let $\it{w}_{\theta}$ be the $|{\cal Y}|$-dimensional vector defined by

[TABLE]

Then, $\overline{\xi}_{W}(\theta)$ and $\underline{\xi}_{W}(\theta)$ are defined as:

[TABLE]

For the singleton case, let $W_{\theta}(x):=W(x|x^{\prime})^{1-\theta}$ and $\it{v}_{\theta}$ be the eigenvector of $W_{\theta}^{T}$ with respect to the Perron-Frobenius eigenvalue $\lambda_{\theta}$ such that $\min_{x}\it{v}_{\theta}(x)=1$. Let $\it{w}_{\theta}(x)=P_{X_{1}}(x)^{1-\theta}$. Then, $\overline{\xi}_{W}(\theta)$ and $\underline{\xi}_{W}(\theta)$ are defined as:

[TABLE]

Proposition 5**.**

[1, lemmas 9 and 11]** Suppose that a transition matrix $W$ satisfies Assumption 2. Then, we have

[TABLE]

where $\overline{\zeta}_{W}(\theta,\theta^{\prime})$ and $\underline{\zeta}_{W}(\theta,\theta^{\prime})$ are defined as follows:

For the non-hidden case with respect to ${\cal Y}$, let $\it{v}_{\theta,\theta^{\prime}}$ be the eigenvector of $N_{\theta,\theta^{\prime}}^{T}$ with respect to the Perron-Frobenius eigenvalue $\nu_{\theta,\theta^{\prime}}$ such that $\min_{y}\it{v}_{\theta,\theta^{\prime}}(y)=1$. Let $\it{w}_{\theta,\theta^{\prime}}$ be the $|{\cal Y}|$-dimensional vector defined by

[TABLE]

Then, $\overline{\zeta}_{W}(\theta,\theta^{\prime})$ and $\underline{\zeta}_{W}(\theta,\theta^{\prime})$ are defined as:

[TABLE]

for $\theta<0$ and

[TABLE]

for $\theta>0$.

For the singleton case, we define $\overline{\zeta}_{W}(\theta,\theta^{\prime})$ and $\underline{\zeta}_{W}(\theta,\theta^{\prime})$ by (29) and (30) independently of $\theta^{\prime}$.

III Functions with three terminals

III-A Functions for single shot setting

Now, to deal with joint source and channel coding, we newly introduce some functions related with three random variables $M,X$ and $Z$. For $r>0$ and $\theta\in(-\infty,1)$, we define following function.

[TABLE]

Also we define its derivative

[TABLE]

Since $U[P_{XZ},Q_{Y};r](\theta)$ is convex function, $u[P_{XZ},Q_{Y};r](\theta)$ is monotonically increasing function. Hence, we can define its inverse function $\theta[P_{XZ},Q_{Y};r](a)$ by

[TABLE]

for $\underline{a}\leq a\leq\overline{a}$, where $\underline{a}:=\lim_{\theta\rightarrow-\infty}u[P_{XZ},Q_{Y};r](\theta)$ and $\overline{a}:=\lim_{\theta\rightarrow 1}u[P_{XZ},Q_{Y};r](\theta)$.

When we define

[TABLE]

for $\underline{a}\leq a\leq\overline{a}$, the derivative is calculated to be

[TABLE]

Hence, $R[P_{XZ},Q_{Y};r](a)$ is monotonically increasing function of $\underline{a}\leq a\leq\overline{a}$. Thus, we can define the inverse function $a[P_{XZ},Q_{Y};r](R)$ by

[TABLE]

for $R[P_{XZ},Q_{Y};r](\underline{a})<R\leq rH_{0}(M)+H_{0}(X|Z)$.

III-B Functions for two transition matrices

We define similar functions for two transition matrices $W_{s}$ on ${\cal M}$ and $W_{c}$ on ${\cal X}\times{\cal Z}$. Suppose that $W_{c}$ is non-hidden with respect to ${\cal Z}$, i.e., satisfies Assumption 1.

For $r>0$ and $\theta\in(-\infty,1)$, we define

[TABLE]

Using above two functions, we define

[TABLE]

for $\underline{a}\leq a\leq\overline{a}$, where $\underline{a}:=\lim_{\theta\rightarrow-\infty}u[W_{s},W_{c},\downarrow;r](\theta)$ and $\overline{a}:=\lim_{\theta\rightarrow 1}u[W_{s},W_{c},\downarrow;r](\theta)$. Moreover, we define

[TABLE]

for $R[W_{s},W_{c},\downarrow;r](\underline{a})<R\leq rH_{0}^{W_{s}}(M)+H_{0}^{W_{c},\downarrow}(X|Z)$.

Now, we suppose that $W_{c}$ satisfies Assumption 2. For $r>0$ and $\theta,\theta^{\prime}\in(-\infty,1)$, we define

[TABLE]

When $\theta=\theta^{\prime}$ we also define for $r>0$ and $\theta\in(-\infty,1)$,

[TABLE]

Using above two functions, we define

[TABLE]

for $\underline{a}\leq a\leq\overline{a}$, where $\underline{a}:=\lim_{\theta\rightarrow-\infty}u[W_{s},W_{c},\uparrow;r](\theta)$ and $\overline{a}:=\lim_{\theta\rightarrow 1}u[W_{s},W_{c},\uparrow;r](\theta)$. Moreover, we define

[TABLE]

for $R[W_{s},W_{c},\uparrow;r](\underline{a})<R\leq rH_{0}^{W_{s}}(M)+H_{0}^{W_{c}\uparrow}(X|Z)$.

IV SINGLE SHOT SETTING

IV-A Problem formulation

We first present the problem formulation by the single shot setting. Assume that the message $M$ takes values in ${\cal M}$ and is subject to the distribution $P_{M}$. For a channel $W_{Y|X}(y|x)$ with input alphabet ${\cal X}$ and output alphabet ${\cal Y}$, a channel code $\phi=(\mathsf{e},\mathsf{d})$ consists of one encoder $\mathsf{e}:{\cal M}\to{\cal X}$ and one decoder $\mathsf{d}:{\cal Y}\to{\cal M}$. The average decoding error probability is defined by

[TABLE]

For notational convenience, we introduce the minimum error probability under the above condition:

[TABLE]

IV-B Direct part

IV-B1 General case

We introduce several lemmas for the case when ${\cal M}$ is the set of messages to be sent, $P_{M}$ is the distribution of the messages, and $W_{Y|X}$ is the channel from ${\cal X}$ to ${\cal Y}$.

We have the following single-shot lemma for the direct part.

Proposition 6**.**

[10, Lemma 3.8.1]** For any constant $c>0$ and for any $P_{X}\in{\cal P(X)}$, there exists a code $\phi=(\mathsf{e},\mathsf{d})$ such that

[TABLE]

where $\bar{W}_{Y}(y):=\sum_{x}P_{X}(x)W_{Y|X}(y|x)$ and $P_{X}\times W_{Y|X}(y,x):=P_{X}(x)W_{Y|X}(y|x)$.

From above Proposition, we obviously have following corollary.

Corollary 1**.**

[TABLE]

Proof.

Since the proof of this lemma is crucial for our proof of the next novel lemma, we give a proof of this lemma as follows. We prove this lemma by using the random coding method. For the code $\phi=(\mathsf{e},\mathsf{d})$, we independently choose $\mathsf{e}(m)\in{\cal X}$ subject to $P_{X}$. Define $D_{m}:=\{y|P_{M}(m)W_{Y|X}(y|\mathsf{e}(m))\geq c\bar{W}_{Y}(y)\}$ and define decoding region of message $m$ as $D_{m}^{\prime}:=D_{m}\setminus(\cup_{m^{\prime}\neq m}D_{m^{\prime}})$. The error probability of this code can be evaluated as:

[TABLE]

Taking the average for the random choice, the first term is

[TABLE]

and the second term is

[TABLE]

Combining (59), (60) and (62), we have

[TABLE]

Consequently, there must exist at least one deterministic code $\phi$ satisfying

[TABLE]

∎

From the above proof, we also find the following single-shot lemma for the direct part.

Lemma 1**.**

For any constant $c>0$ and for any distribution $P_{X}\in{\cal P(X)}$, we have

[TABLE]

where $1_{M}$ is a counting measure on ${\cal M}$. The choice $c=1$ gives the minimum upper bound.

We also have following lemma.

Lemma 2**.**

[TABLE]

*Proof of Lemma 1:  * From (61) in the proof of Proposition 6, we can evaluate the second term of (60) as

[TABLE]

So, we obtain (65).

Next, we prove that the right hand side of (65) is minimized when $c=1$. For any $c>0$, we can evaluate the right hand side of (65) as:

[TABLE]

*Proof of Lemma 2:  * For any $s\in(0,1)$, we have

[TABLE]

However, even when $M$ is subject to the uniform distribution, the upper bound (66) is not so tight. In the uniform case, the Gallager bound is tighter than the upper bound (66). So, modifying the derivation of the Gallager bound, we derive joint source and channel coding version of the Gallager bound as follows.

Lemma 3**.**

For any distribution $P_{X}\in{\cal P}({\cal X})$, we have

[TABLE]

for any $s\in[0,1/2]$.

Proof.

For encoder, we independently choose $\mathsf{e}(i)\in{\cal X}$ subject to $P_{X}$, and for decoder, we define decoding region of the message $i$ as

[TABLE]

And we also define

[TABLE]

Then, for any $0\leq s\leq 1$ and $0\leq t\leq 1$,

[TABLE]

and error probability can be represented by

[TABLE]

So that,

[TABLE]

Taking the average for the random choice, we have

[TABLE]

By setting $t=\frac{s}{1-s}$ in (73), we have

[TABLE]

Hence, we have

[TABLE]

(75) means that there must exist at least one deterministic code $\phi$ satisfying

[TABLE]

Since $0\leq t\leq 1$, $s$ is restricted to $0\leq s\leq\frac{1}{2}$. So we obtain (67). ∎

IV-B2 Conditional additive case

Now, we proceed to the case when the channel is conditional additive. Assume that ${\cal X}$ is a module and ${\cal Y}$ is given as ${\cal X}\times{\cal Z}$. Then, the channel $W$ is called conditional additive [1] when there exists a joint distribution $P_{XZ}$ such that

[TABLE]

Then we can simplify (58). We have following lemma.

Lemma 4**.**

When the channel is conditional additive channel, it follows that

[TABLE]

Proof.

By setting that $P_{X}$ is the uniform distribution and choosing the random variables $X=X^{\prime}$ and $Y=XZ$ to the right hand side of (58), we have

[TABLE]

where $P_{Z}(z):=\sum_{x}P_{XZ}(x,z)$. Hence, (58) can be simplified to

[TABLE]

∎

Also we can simplify (66) and (67). We have following lemma.

Lemma 5**.**

When the channel is conditional additive channel, it follows that

[TABLE]

and

[TABLE]

Proof.

Firstly, we prove (80). $e^{sH^{\downarrow}(X|Y)}$ is represented as:

[TABLE]

Assume that ${\cal Y}={\cal X}\times{\cal Z}$ and its random variable is $Y=XZ$. Setting $P_{XY}=P_{X}\times W_{XZ|X}$, $P_{Y}(y)=P_{Z}(z):=\sum_{x}P_{XZ}(x,z)$ and $P_{X}$ is uniform distribution, we have

[TABLE]

Substituting (83) to (66), we have (80).

And also we have

[TABLE]

Substituting (84) to (67), we have (81). ∎

IV-C Converse part

IV-C1 General case

Firstly, combining the idea of meta converse [13] and [14, Lemma 4] and the general converse lemma for the joint source and channel coding [10, Lemma 3.8.2], we obtain the following lemma for the single shot setting. The following lemma is the same as [10, Lemma 3.8.2] when $Q_{Y}$ is $\bar{W}_{Y}$.

Lemma 6**.**

For any constant $c>0$, any code $\phi=(\mathsf{e},\mathsf{d})$ and any distribution $Q_{Y}$ on ${\cal Y}$, we have

[TABLE]

Proof.

First, we set

[TABLE]

and for each $(m,x)\in({\cal M,X})$, define

[TABLE]

Also, for decoder $\psi$ and each $m\in{\cal M}$, we define

[TABLE]

In addition, we define $P_{X|M}$ so that

[TABLE]

Using this, we define

[TABLE]

Then,

[TABLE]

The last equality follows since the error probability can be written as

[TABLE]

We notice here that

[TABLE]

for $y\in{\cal B}(m,x)$. By substituting this into (92), the first term of (92) is

[TABLE]

which implies (85). ∎

IV-C2 Conditional additive case

Now, we proceed to the conditional additive case given in (77). Applying (85) to the conditional additive case, we obtain following lemma.

Lemma 7**.**

For arbitrary distribution $Q_{Z}\in{\cal P(Z)}$, we have

[TABLE]

Proof.

For some $Q_{Z}\in{\cal P}({\cal Z})$, we substitute

[TABLE]

to (85). Then, the first term of the right hand side of (93) is

[TABLE]

So, we obtain (93).

∎

Similar to [1, Theorem 5], using the monotonicity of Rényi divergence, we obtain another type of converse lemma.

Lemma 8**.**

We set $R:=\log|{\cal X}|$. Then, it holds that

[TABLE]

where

[TABLE]

Proof.

In this proof, we use the notation defined in (96)-(98).

For arbitrary $\rho\in\mathbb{R}$, we define following new distributions.

[TABLE]

Using these, we define following joint distribution.

[TABLE]

For arbitrary code $\phi=(\mathsf{e},\mathsf{d})$, we define

[TABLE]

And also, when the source distribution is $P_{M,\rho}$ and the channel is conditional additive channel $W_{XZ|X,\rho}$ defined by

[TABLE]

we define

[TABLE]

Then, for any $s>0$, by the monotonicity of the Rényi divergence, we have

[TABLE]

Thus, we have

[TABLE]

For the Rényi divergence, we have

[TABLE]

In addition, substituting $P_{M}=P_{M}(m)^{1-\rho}e^{-\rho H_{1-\rho}(M)}$ and $P_{XZ}=P_{XZ,{\rho}}(x,z)^{1-\rho}e^{-\rho H_{1-\rho}(P_{XZ}|Q_{Z})}$ into (93), we have

[TABLE]

For any $\sigma\geq 0$, the first term of right hand side of (108) can be evaluated as:

[TABLE]

Thus, by setting $c$ so that

[TABLE]

we have

[TABLE]

For the Rényi divergence in (110), we have

[TABLE]

So, we have

[TABLE]

Combining (106), (107) and (111), we obtain (94).

Now, we restrict the range of $\rho$ so that $\theta(a(R))<\rho<1$, and take

[TABLE]

we obtain the second inequality. ∎

V $n$-fold Markovian conditional additive channel

V-A Formulation for general case

Firstly, we give general notations for channel coding when the message obeys Markovian process. We assume that the set of messages is ${\cal M}^{k}$. Then, we assume that the message $M^{k}=(M_{1},\ldots,M_{k})\in{\cal M}^{k}$ is subject to the Markov process with the transition matrix $\{W_{s}(m|m^{\prime})\}_{m,m^{\prime}\in{\cal M}}$. We denote the distribution for $M^{k}$ by $P_{M^{k}}$.

Now, we consider very general sequence of channels with the input alphabet ${\cal X}^{n}$ and the output alphabet ${\cal Y}^{n}$. In this case, the transition matrix as $\{W_{Y^{n}|X^{n}}(y^{n}|x^{n})\}_{x^{n}\in{\cal X}^{n},y^{n}\in{\cal Y}^{n}}$. Then, a channel code $\phi=(\mathsf{e},\mathsf{d})$ consists of one encoder $\mathsf{e}:{\cal M}^{k}\to{\cal X}^{n}$ and one decoder $\mathsf{d}:{\cal Y}^{n}\to{\cal M}^{k}$. Then, the average decoding error probability is defined by

[TABLE]

For notational convenience, we introduce the error probability under the above condition:

[TABLE]

When there is no possibility for confusion, we simplify it to $\mathrm{P}_{\mathrm{j}}(k,n)$. Instead of evaluating the error probability $\mathrm{P}_{\mathrm{j}}(n,k)$ for given $n,k$, we are also interested in evaluating

[TABLE]

for given $0\leq\varepsilon\leq 1$.

V-B Formulation for Markovian conditional additive channel

In this section, we address an $n$-fold Markovian conditional additive channel [1]. That is, we consider the case when the joint distribution for the additive noise obeys the Markov process. To formulate our channel, we prepare notations. Consider the joint Markovian process on ${\cal X}\times{\cal Z}$. That is, the random variables $X^{n}=(X_{1},\ldots,X_{n})\in{\cal X}^{n}$ and $Z^{n}=(Z_{1},\ldots,Z_{n})\in{\cal Z}^{n}$ are assumed to be subject to the joint Markovian process defined by the transition matrix $\{W_{c}(x,z|x^{\prime},z^{\prime})\}_{x,x^{\prime}\in{\cal X},z,z^{\prime}\in{\cal Z}}$. We denote the joint distribution for $X^{n}$ and $Z^{n}$ by $P_{X^{n},Z^{n}}$. Now, we assume that ${\cal X}$ is a module, and consider the channel with the input alphabet ${\cal X}^{n}$ and the output alphabet $({\cal X}\times{\cal Z})^{n}$. The transition matrix for the channel $W_{X^{n},Z^{n}|\tilde{X}^{n}}$ is given as

[TABLE]

for $z^{n}\in{\cal Z}^{n}$ and $x^{n},\tilde{x}^{n}\in{\cal X}^{n}$. Also, we denote $\log|{\cal X}|$ by $R$. In the following discussion, we use the channel capacity $C:=\log|{\cal X}|-H^{W_{c}}(X|Z)$, which is shown in [1]. In this case, we denote the average error probability $\mathrm{P}_{\mathrm{j}}[\phi|k,n|W_{s},W_{X^{n},Z^{n}|X^{n}}]$ and the minimum average error probability $\mathrm{P}_{\mathrm{j}}(k,n|W_{s},W_{X^{n},Z^{n}|X^{n}})$ by $\mathrm{P_{jca}}[\phi|k,n|W_{s},W_{c}]$ and $\mathrm{P_{jca}}(k,n|W_{s},W_{c})$, respectively. Then, we denote the maximum size $\mathrm{K}(n,\epsilon|W_{s},W_{Y^{n}|X^{n}})$ by $\mathrm{K_{ca}}(n,\epsilon|W_{s},W_{c})$. When we have no possibility for confusion, we simplify them to by $\mathrm{P_{jca}}[\phi|k,n]$, $\mathrm{P_{jca}}(k,n)$, and $\mathrm{K_{ca}}(n,\epsilon)$, respectively.

In the following discussion, we assume Assumption 1 or 2 for the joint Markovian process described by the transition matrix $\{W_{c}(x,z|x^{\prime},z^{\prime})\}_{x,x^{\prime}\in{\cal X},z,z^{\prime}\in{\cal Z}}$. The paper [1] derives the single-letterized channel capacity under Assumption 1. Among author’s knowledge, the class of channels satisfying Assumption 1 is the largest class of channels whose channel capacity is known. When ${\cal Z}$ is singleton and the channel is the noiseless channel given by identity transition matrix $I$, our problem is the source coding with Markovian source. In this case, the memory size is equal to the cardinality $|{\cal X}|^{k}$, we denote the minimum error probability $\mathrm{P_{jca}}(k,n|W_{s},I_{X|X})$ by $\mathrm{P}_{\mathrm{s}}(k,n|W_{s})$.

V-C Finite-length bound

V-C1 Assumption 1

Now, we assume Assumption 1. Combining Proposition 3 and (80) of Lemma 5, we have an upper bound of the minimum error probability as follows.

Theorem 1** (Direct Bound).**

When Assumption 1 holds, setting $R=\log|{\cal X}|$, we have

[TABLE]

where

[TABLE]

Combining Proposition 3 and (94) of Lemma 8, we have a lower bound of the minimum error probability as follows.

Theorem 2** (Converse bound).**

When Assumption 1 holds, setting $R=\log|{\cal X}|$, we have

[TABLE]

where

[TABLE]

and where

[TABLE]

Proof.

We first substitute $P_{XZ}=P_{X^{n}Z^{n}}$ $Q_{Z}=P_{Z^{n}}$ to (94) of Lemma 8 and use Proposition 3. Then, we restrict the range of $\rho$ as $\theta(a(R))<\rho<1$ and set $\sigma=\frac{\rho-\theta(a(R))}{1-\rho}$. Then, we have the claim of the Theorem. ∎

V-C2 Assumption 2

Next, we assume Assumption 2. Combining Proposition 4 and (81) of Lemma 5, we have an upper bound of the minimum error probability as follows.

Theorem 3** (Direct Bound).**

When Assumption 2 holds, setting $R=\log|{\cal X}|$, we have

[TABLE]

where

[TABLE]

Combining Proposition 5 and (94), we have a lower bound of the minimum error probability as follows.

Theorem 4** (Converse Bound).**

When Assumption 2 holds, setting $R=\log|{\cal X}|$, we have

[TABLE]

where

[TABLE]

and where

[TABLE]

Proof.

We first substitute $P_{XZ}=P_{X^{n}Z^{n}}$ $Q_{Z}=P_{Z^{n}}^{(1-\theta(a(R)))}$ to (94) of Lemma 8 and use Proposition 4 and 5. Then, we restrict the range of $\rho$ as $\theta(a(R))<\rho<1$ and set $\sigma=\frac{\rho-\theta(a(R))}{1-\rho}$. Then, we have the claim of the Theorem. ∎

Remark 7**.**

Although the paper [3] derived a different finite-length converse bound as Lemma 3 of [3], their bound contains so large polynomial factor that their bound cannot yield good numerical evaluation as ours.

V-D Large deviation bounds

In this section, for some constant $r>0$, we fix the coding rate $\frac{k}{n}$ to be $r$ by using the real number $R:=\log|{\cal X}|$.

V-D1 Assumption 1

Now, we assume Assumption 1. Using Theorem 1, we can upper bound the exponent of the minimum error probability as follows. By setting $k=nr$, taking logarithm and normalizing by $n$ both sides of (117), we obtain following theorem.

Theorem 5** (Direct Bound).**

Assume that Assumption 1 holds and set $R=\log|{\cal X}|$. When the rate $r$ satisfies $rH^{W_{s}}(M)+H^{W_{c},\downarrow}(X|Z)<R$, we have

[TABLE]

where $E_{1,j}(r)$ is error exponent function defined as

[TABLE]

Remark 8**.**

This theorem is a conditional additive version of [8, Proposition 1].

Using Theorem 2, we can lower bound exponent of the minimum error probability as follows. By setting $k=nr$, we obtain following theorem.

Theorem 6** (Converse Bound).**

Assume that Assumption 1 holds and set $R=\log|{\cal X}|$. When the rate $r$ satisfies $rH^{W_{s}}(M)+H^{W_{c},\downarrow}(X|Z)<R<rH_{0}^{W_{s}}(M)+H_{0}^{W_{c},\downarrow}(X|Z)$, we have

[TABLE]

where $\overline{E}_{1,j}(r)$ is error exponent function defined as

[TABLE]

where

[TABLE]

Remark 9**.**

This theorem is a conditional additive version of [8, Theorem 2].

Proof.

From Theorem 2, we have

[TABLE]

where $u(\cdot):=u[W_{s},W_{c},\downarrow;r](\cdot)$. ∎

This part will be done similar to [1, Theorem 21]. In this case, the direct part bound does not coincide with the converse part bound, in general. To derive the exact value of the exponent, we need a stronger assumption.

V-D2 Assumption 2

Next, we assume Assumption 2, which is stronger than Assumption 1. Using Theorem 3, we can upper bound the exponent of the minimum error probability as follows. By setting $k=nr$, taking logarithm and normalizing the both side of (125), we obtain following theorem.

Theorem 7** (Direct Bound).**

Assume that Assumption 2 holds and set $R=\log|{\cal X}|$. When the rate $r$ satisfies $rH^{W_{s}}(M)+H^{W_{c},\uparrow}(X|Z)<R$, we have

[TABLE]

where $E_{2,j}$ is an error exponent function defined as

[TABLE]

Using Theorem 4, we can lower bound the exponent of the minimum error probability as follows. By setting $k=nr$, we obtain following theorem.

Theorem 8** (Converse Bound).**

Assume that Assumption 2 holds and set $R=\log|{\cal X}|$. When the rate $r$ satisfies $rH^{W_{s}}(M)+H^{W_{c},\uparrow}(X|Z)<R<rH_{0}^{W_{s}}(M)+H_{0}^{W_{c},\uparrow}(X|Z)$, we have

[TABLE]

where $\overline{E}_{2,j}(r)$ is an error exponent function defined as

[TABLE]

where

[TABLE]

Proof.

From Theorem 4, we have

[TABLE]

where $u_{\theta(a(R))}(\cdot):=u[W_{s},W_{c},\theta(a(R));r](\cdot)$ and $u^{\uparrow}(\cdot):=u[W_{s},W_{c},\uparrow;r](\cdot)$. ∎

Corollary 2**.**

Combining the above theorems, we obtain the exact expression of the exponent of the minimum error probability when we define the critical rate $R_{cr}$ as

[TABLE]

For $R\leq R_{cr}$, we can rewrite the upper bound in Theorem 8 as

[TABLE]

Thus, the lower bound in Theorem 7 coincides with the upper bound in Theorem 8. So we have

[TABLE]

Remark 10**.**

Now, we consider the case when ${\cal Z}$ is singleton and the transition matrix $W_{c}$ of the additive noise is the identity matrix $I$, which is the same as the data compression with Markovian source. Since $C=\log|{\cal X}|$, we have

[TABLE]

which is the same as the result of [1, Theorem 12].

V-E Moderate deviation bound

Next, we proceed to the moderate deviation regime, in which, the coding rate $r_{n}$ behaves as $r_{n}:=\frac{k}{n}=\frac{C}{H^{W_{s}}}-\delta n^{-t}$ with $t\in(0,\frac{1}{2})$. Then, the minimum error probability can be evaluated as follows.

Theorem 9**.**

Assume that Assumption 1 holds. Then, for arbitrary $t\in(0,\frac{1}{2})$ and $\delta>0$, it holds that

[TABLE]

Remark 11**.**

Theorem 9 is conditional additive channel version of [3, Theorem 1].

Proof.

From Theorem 1, we obtain

[TABLE]

By (19), Taylor expansions of $H_{1-s}^{W_{s}}(M)$ and $H_{1-s}^{W_{c},\downarrow}(X|Z)$ in the neighborhood of $s=0$ are

[TABLE]

Substituting these expansions into (156), we obtain

[TABLE]

Now, we set $s^{\prime}:=\frac{\delta n^{-t}H^{W_{s}}(M)}{\frac{C}{H^{W_{s}}(M)}V^{W_{s}}(M)+V^{W_{c}}(X|Z)}$ which satisfies $s\in[0,1]$ for enough large $n$. Then, we have

[TABLE]

that is,

[TABLE]

On the other hands, by choosing $\rho=\frac{n^{-t}\delta}{\frac{1}{H^{W_{s}}(M)}\{\frac{C}{H^{W_{s}}(M)}V^{W_{s}}(M)+V^{W_{c}}(X|Z)\}}$, Theorem 2 implies that

[TABLE]

∎

Now, we consider the case when ${\cal Z}$ is singleton and the transition matrix $W_{c}$ of the additive noise is the identity matrix $I$. When $k=\frac{C}{H^{W_{s}}(M)}n-\frac{C}{H^{W_{s}}(M)^{2}}(\frac{C}{H^{W_{s}}(M)})^{-t}{\delta}^{\prime}n^{1-t}$, the minimum error probability $\mathrm{P}_{\mathrm{s}}(k,n|W_{s})$ is characterized as follows. Setting $\delta=\frac{C}{H^{W_{s}}(M)^{2}}(\frac{C}{H^{W_{s}}(M)})^{-t}{\delta}^{\prime}$ i.e., $k=\frac{C}{H^{W_{s}}(M)}n-\frac{C}{H^{W_{s}}(M)^{2}}(\frac{C}{H^{W_{s}}(M)})^{-t}{\delta}^{\prime}n^{1-t}$, the minimum error probability $\mathrm{P}_{\mathrm{s}}(k,n|W_{s})$ and using $C=\log|{\cal X}|$, we obtain

[TABLE]

This result coincides with [1, Theorem 11].

VI Numerical Example

Finally, to demonstrate the advantage of our finite-length bounds, we numerically evaluate the achievability bound in Theorem 3 and the converse bound in Theorem 4. Due to the efficient construction of our bounds, we could calculate both bounds with huge size $n=1\times 10^{6}$ because the calculation complexity behaves as $O(1)$.

We employ the following parametrization $W(p,q)$ for the binary transition matrix:

[TABLE]

We consider the case when $W_{s}=W_{c}=W(0.1,0.2)$. The optimal transmission rate $\frac{C}{H^{W_{s}}(M)}$ and the dispersion ${\frac{1}{(H^{W_{s}}(M))^{2}}\left[\frac{C}{H^{W_{s}}(M)}V^{W_{s}}(M)+V^{W_{c}}(X|Z)\right]}$ are calculated to be 0.807317 and 6.12809, respectively. Also, the exponent $E(0.75)$ is calculated to be 0.0002826, which is approximated by $E_{md}(0.75n,n)/n=0.0002680$.

When $n=10000$, Fig. 1 calculates the upper and lower bounds of $-\log\mathrm{P}_{\mathrm{j}}(k,n)$ based on Theorems 3 and 4. Also, it shows the comparison them with the approximations $nE(k/n)$ and $E_{md}(k,n)$ by Theorems 7 and 9. Fig. 2 addresses the quantity $-\frac{1}{n}\log\mathrm{P}_{\mathrm{j}}(0.75n,n)$ in the same way.

Acknowledgments

MH is very grateful to Professor Vincent Y. F. Tan and Professor Shun Watanabe for helpful discussions and comments. The works reported here were supported in part by a MEXT Grant-in-Aid for Scientific Research (B) No. 16KT0017, the Okawa Research Grant and Kayamori Foundation of Informational Science Advancement.