The adaptive zero-error capacity for a class of channels with noisy feedback

TL;DR

This paper investigates the zero-error capacity of discrete memoryless channels with noisy feedback, establishing bounds and conditions under which the capacity equals the zero-undetected error capacity of the forward channel.

Contribution

It extends the understanding of zero-error capacity to channels with noisy feedback, providing bounds and conditions for capacity equality.

Findings

Lower bound on zero-error capacity with noisy feedback equals zero-undetected error capacity

Capacity bounds are tight under certain conditions

Noisy feedback can be effectively characterized in terms of zero-undetected error capacity

Abstract

The adaptive zero-error capacity of discrete memoryless channels (DMC) with noiseless feedback has been shown to be positive whenever there exists at least one channel output "disprover", i.e. a channel output that cannot be reached from at least one of the inputs. Furthermore, whenever there exists a disprover, the adaptive zero-error capacity attains the Shannon (small-error) capacity. Here, we study the zero-error capacity of a DMC when the channel feedback is noisy rather than perfect. We show that the adaptive zero-error capacity with noisy feedback is lower bounded by the forward channel's zero-undetected error capacity, and show that under certain conditions this is tight.

Equations

$$W^n(y^n|x^n)=\prod _{j=1}^{n}W(y_j|x_j),x^n\in {\mathcal{X}}^n,y^n\in {\mathcal{Y}}^n.$$

$$\mathcal{F}:\mathcal{M}\to {\mathcal{X}}^n,\mathcal{G}:{\mathcal{Y}}^n\to \mathcal{M}.$$

$${\lambda }_m^{(n)}=Pr(\mathcal{G}(y^n)\ne m|X^n=c^{(n)}(m)),$$

$${\lambda }^{(n)}=\underset{m\in \mathcal{M}}{\max }{\lambda }_m^{(n)},P_e^{(n)}=\frac{1}{|\mathcal{M}|}\sum _{m\in \mathcal{M}}{\lambda }_m^{(n)}.$$

$${\mathcal{F}}_{0u}:\mathcal{M}\to {\mathcal{X}}^n,$$

$$M(y^n)=\{m\in \mathcal{M}:W^n(y^n|c_{ou}^{(n)}(m))>0\}.$$

$${\mathcal{G}}_{0u}(y^n)=\{\begin{array}{ll}M(y^n)& \text{mbox}if|M(y^n)|=1\\ \mathcal{E}& \text{mbox}if|M(y^n)|>1.\end{array}$$

$${\lambda }_m=Pr({\mathcal{G}}_{0u}(y^n)=\mathcal{E}|X^n=c_{0u}^{(n)}(m)).$$

$${\mathcal{F}}_j:$$

$${\mathcal{G}}_j:$$

$${\mathcal{H}}_j:$$

$$C_{0fa}=\{\begin{array}{ll}C& \text{mbox}if{C}_{0u}>0\\ 0& \text{mbox}otherwise,\end{array}$$

$$\bar{N}$$

$$\underset{n\to \infty }{\lim }\bar{R}$$

$$C_{0fa}^{noisy}\ge C_{0u}^{(f)}\text{mbox}if{C}_{0u}^{(f)}>0\text{mbox}and{C}_{0u}^{(b)}>0,$$

Full text

I Introduction

Shannon determined that the zero-error capacity, denoted by $C_{0}$, of a point-to-point channel whose channel $W(y|x)$ has confusability graph $G_{X|Y}$ is positive if and only if there exist two inputs that are “non-confusable” [1]. Equivalently, it is non-zero if and only if the independence number of $G_{X|Y}$ is strictly greater than 1.

Shannon’s condition for positive zero-error capacity $C_{0}$ is restrictive; that for positive zero-error capacity in the presence of perfect output feedback is less so. In the set of slides [2], Massey showed that it is possible to communicate at a non-zero rate with zero-error over a DMC with noiseless feedback if, and only if, there exists at least one channel output that is reachable from some but not all the channel inputs. Such a channel output is called a “disprover”. Not only does the existence of a disprover allow for positive rates, but Massey showed that with perfect feedback, the adaptive zero-error capacity of channels attains the small-error Shannon capacity $C$. Note that the adaptive zero-error capacity allows for adaptive and variable-length codewords rather than blockcodes. Shannon only considered block codes for zero-error feedback channels in [1].

The binary erasure channel (BEC) and the Z-channel are examples of channels whose zero-error capacity $C_{0}$ without feedback is equal zero, but, as both contain a disprover, have zero-error capacity equal to their Shannon capacity (positive in general) in the presence of perfect feedback. In order to achieve such zero-error rates, an adaptive communication scheme is used in which the transmitter repeatedly sends a message until it sees that it has been correctly received.

While the zero-error capacity in the presence of feedback has not been extensively studied beyond the slides of Massey [2], as we will show, it has strong connections with the zero-undetected-error capacity [3] with noiseless feedback [4]. Two types of communication errors occur: i) erasure errors, when the decoder is unable to uniquely decode any message, and ii) undetected-errors, when the decoder uniquely decodes an erroneous message. The zero-undetectable error capacity $C_{0u}$, first considered by Forney [3], denotes the maximal number of inputs that can be transmitted to ensure that the probability of an undetectable error is exactly zero. Forney derived a lower bound for the zero-undetected-capacity $(C_{0u})$ of a channel, which he showed is positive if, and only if, this channel contains a disprover. Later on, a tighter lower bound on $C_{0u}$ was derived by Ahlswede [5], which was shown to be tight for two classes of channels in [6] and [7]. Finally, in [4] it was shown that the zero-undetected-error capacity for a channel with noiseless feedback, denoted by $C_{0uf}$, is equal to the small-error Shannon capacity $C$ if the channel contains at least one disprover. Note that in general $C_{0}\leq C_{0u}\leq C_{0uf}\leq C$.

Contribution. In this paper we focus on zero-error communication for a general DMC with feedback. In Theorem 1, we detail the proof of a result outlined by Massey in his slides [2] for the zero error capacity of a channel with noiseless feedback, $C_{0fa}$. In Theorem 2, our main result, we consider noisy (rather than noiseless) feedback, and using an adaptive zero-error scheme, we prove that the adaptive zero-error capacity of the channel with noisy feedback, $C_{0fa}^{noisy}$, is at least the zero-undetected-error capacity of the forward channel $C_{0u}^{(f)}$. Theorem 2 further outlines a class of channels for which this lower bound is tight.

II Definitions

Let $x_{i}^{j}:=(x_{i},x_{i+1},\dots,x_{j})$ when $i\leq j$ and $|x_{i}^{j}|=j-i+1$ denote its size. For simplicity we write $x^{n}=x_{1}^{n}$. Let $\mathcal{B}=\{0,1\}$ be the binary set, and $\mathcal{M}$ be the message set.

Channels. A channel $({\cal X},{\cal Y},W)$ is used to denote a generic DMC with finite input alphabet $\mathcal{X}$, finite output alphabet $\mathcal{Y}$, and transition probability $W(y|x)$. We write $W^{n}$ to denote the channel corresponding to $n$ uses of $W$:

[TABLE]

We consider channels with feedback, with a forward channel $(\mathcal{X}_{(f)},\mathcal{Y}_{(f)},W_{(f)})$ (subscript ${(f)}$) and a backward channel $(\mathcal{X}_{(b)},\mathcal{Y}_{(b)},W_{(b)})$ (for feedback, subscript ${(b)}$).

Small error capacity $C$ without feedback. A $\mathcal{C}(\mathcal{M},n)$ code for DMC $W$ with message set $\mathcal{M}$ without feedback, consists of a message set ${\cal M}$ of size $2^{nR}$, for $R$ the rate and $n$ the blocklength, and encoding and decoding functions $\mathcal{F}$ and $\mathcal{G}$ respectively:

[TABLE]

Let $c^{(n)}(m)$ denote a codeword corresponding to message $m$, i.e. $c^{(n)}(m)=F(m)$ and let

[TABLE]

be the conditional probability of error given that message $m$ was sent. The maximum and average, respectively, probabilities of error for a $\mathcal{C}(\mathcal{M},n)$ are defined as

[TABLE]

The small error capacity $C$ for channel $W$ is defined as the largest number $R$ such that there exists a sequence of $\mathcal{C}(\mathcal{M},n)$ codes such that $\lambda^{(n)}$ tends to [math] as $n\rightarrow\infty$.

Zero-undetected error code and capacity $C_{0u}$ [4]. A zero-undetected-error code of block-length $n$, denoted by $\mathcal{C}_{0u}(\mathcal{M},n)$, again consists of a message set ${\cal M}$, an encoding function

[TABLE]

that encodes messages $m$ to $c_{ou}^{(n)}(m)$, and a decoding function $G_{0u}$ described as follows. Let $M(y_{1}^{n})$ denote the set of probable messages corresponding to a received output $y_{1}^{n}$

[TABLE]

The decoder declares an erasure, denoted by $\mathcal{E}$, if there exist more than one possible message that could have yielded output $y^{n}$, i.e. $|M(y^{n})|>1$. A zero-undetected-error decoder function is then defined as

[TABLE]

A valid zero-undetected-error code must have no undetected errors, hence the maximal error probability is given only by the probability of erasures as

[TABLE]

The zero-undetected capacity $C_{0u}$ for channel $W$ is defined as the largest rate $R$ such that there exist a sequence of $\mathcal{C}_{0u}(\mathcal{M},n)$ codes that $\max_{m\in{\cal M}}\lambda_{m}$ tends to [math] as $n\rightarrow\infty$.

Adaptive zero-error capacity with feedback, $C_{0fa}$. An adaptive zero-error code with feedback ${\cal C}_{0fa}({\cal M})$ for a DMC with a forward channel $(\mathcal{X}_{(f)},\mathcal{Y}_{(f)},W_{(f)})$ and a backward channel $(\mathcal{X}_{(b)},\mathcal{Y}_{(b)},W_{(b)})$ as in Fig. 1 consists of a message set ${\cal M}$, a set of encoding ${\cal F}$, feedback ${\cal G}$, and decoding functions ${\cal H}$ respectively for $j=1,2,\cdots$

[TABLE]

where, for every $m\in{\cal M}$, after some $N_{m}$ channel uses, $H_{N_{m}}(Y_{(f)}^{N_{m}})=m$ (i.e. zero-error after $N_{m}$ channel uses). At each channel use $j$, the transmitter sends ${\cal F}_{j}(m,y_{(b)}^{j-1})$ over the forward channel $W_{(f)}$, which is received and added to the sequence of received signals $y_{(f)}^{j}$. The receiver takes this sequence and transmits ${\cal G}_{j}(y_{(f)}^{j-1})$ back over the backwards channel, where it is received as $y_{(b),j}$. Let $D_{m}\geq N_{m}$ be the number of channel uses needed for the message $m$ to be transmitted and decoded with zero error before the next message starts, and is a random variable. Define the expected delay per information bit, with expectation taken over messages $m$ and channel instances, as $\bar{D}:=E[D_{m}]/\log|{\cal M}|$. The zero-error adaptive feedback capacity of the channel in Fig. 1 is then given by the largest expected rate defined as $\bar{R}:=\frac{\log_{2}|{\cal M}|}{E[D_{m}]}$ such that there exists an adaptive zero-error code with feedback with expected delay $\bar{D}<\infty$.

We use the notation $C_{0fa}$ and $C_{0fa}^{noisy}$ to distinguish the zero-error adaptive feedback capacities when the feedback is noiseless and noisy, respectively.

III Adaptive zero-error communication for channels with feedback

One way of ensuring zero-error communication in a channel with perfect feedback is to keep repeating a message until it is correctly received (the transmitter can verify correct reception from the perfect output feedback). One needs to then calculate the average rate and delay incurred. In the following, we present such communication schemes for channels with perfect (Theorem 1) and noisy (Theorem 2) feedback. We denote $x_{i}^{j}\bumpeq x^{\prime}$ if there exists at least one $k\in[i,j]$ such that $x_{k}=x^{\prime}$ (e.g. $1101\bumpeq 0$). Let $[x_{i}]^{l}$ denote a sequence of $l$ repetitions of letter $x_{i}$ in some alphabet ${\cal X}$, $[x_{i}]^{l}=(x_{i},x_{i},\dots,x_{i}),|[x_{i}]^{l}|=l$.

III-A Complete, noiseless feedback

When complete, noiseless feedback is available, Massey [2] suggested a method for achieving zero-error at an expected rate approaching the small error or Shannon capacity of the forward channel, $C$. We outline a Theorem that we attribute to Massey below. Let $\gamma_{n}=o(n)$, and $\gamma_{n}\rightarrow\infty$ as $n\rightarrow\infty$ (e.g. $\gamma_{n}=\log(n)$). Since the backward channel is noiseless, we omit subscript (f) for the forward channel and use $(\mathcal{X},\mathcal{Y},W)$.

Theorem 1** (Massey (Elaborated) [2])**

The adaptive zero-error capacity $C_{0fa}$ for a DMC channel $({\cal X},{\cal Y},W)$ with noiseless feedback is

[TABLE]

where $C$ denotes the Shannon capacity of the channel $({\cal X},{\cal Y},W)$, and $C_{0u}$ denotes its zero-undetected error capacity.

Proof  If $C_{0u}=0$ then by [3], channel $W$ does not have a disprover, i.e. for every $x\in\mathcal{X},y\in\mathcal{Y},W(y|x)>0$. Thus, no matter which sequence is sent the receiver is unable to decide anything with zero error and $C_{0fa}=0$.

When $C_{0u}>0$, we may assume that the DMC $W$ contains at least one disprover. Equivalently, there exists at least one triple $(x_{c},x_{e},y_{c})\in(\mathcal{X}\times\mathcal{X}\times\mathcal{Y})$ such that $W(y_{c}|x_{e})=0$ and $W(y_{c}|x_{c})>0$.

The converse proof is trivial using $C_{0fa}\overset{(1)}{\leq}C_{f}\overset{(2)}{=}C$, where (1) follows as $C_{f}$ denotes the small-error capacity of the channel with perfect feedback, which is always an outer bound to the more restrictive zero-error setting, and (2) follows from Shannon’s result that perfect feedback does not increase the small error capacity of a channel.

For the achievability, let $\mathcal{C}(\mathcal{M},n)$ be a capacity achieving code for the DMC $W$ whose maximal probability of error $\lambda^{(n)}$ tends to zero and whose rate approaches the Shannon capacity $C$ as block length $n\rightarrow\infty$. Note that the output block $y^{n}$ is available in real time at the transmitter due to the presence of perfect feedback. The transmitter can thus mimic the receiver’s decoding rule and determine whether the receiver obtained the correct message. It then tells the receiver this by sending $\gamma_{n}$ copies of either $x_{c}$ (if correct) or $x_{e}$ (if erroneous) through the noisy $W$. Since the receiver can only receive a $y_{c}$ from an $x_{c}$ (definition of a disprover), once it receives at least one $y_{c}$ it realizes that its decoded message is correct, and zero-error communication is achieved. We note that variable $I$ in the Algorithm 1 is used to synchronize the transmitted and receivers, i.e. indicates when a new message will start.

To calculate the average rate and delay achieved, note that the probability that a message is correctly received with zero error is the probability that the message was correctly received at the receiver after seeing the codeword of length $n$, and then the receiver seeing at least one correct indicator (i.e. seeing one $y_{c}$) in a block of length $\gamma_{n}$. Hence after $n+\gamma_{n}$ channel uses, the probability of correctly decoding message $m$ is p_{n,m}=(1-\lambda_{m}^{(n)})\bigg{[}1-\big{(}1-W(y_{c}|x_{c})\big{)}^{\gamma_{n}}\bigg{]}. Viewing this as a probability of success, the number of codeword re-transmissions needed to correctly receive message $m$ and wait for the transceivers to synchronize and start a new message is hence a geometric random variable $L_{n}(m)$ with $E[L_{n}(m)]=1/p_{n,m}$.

Hence, the delay incurred to correctly decode message $m$ is $N_{m}=(n+\gamma_{n})\cdot L_{n}(m)$ and hence the expected delay (not yet normalized by the number of bits) is

[TABLE]

where in (1), the outer expectation is with respect to the uniform distribution over messages $m\in{\cal M}$ and the inner expectation is over the channels, and where $(2)$ is because message $m$ is uniform over ${\cal M}$. Hence, as $n\rightarrow\infty$

[TABLE]

where $(2)$ follows as we are using a Shannon capacity achieving code $\mathcal{C}(\mathcal{M},n)$, and by definitions of $\gamma_{n}$ and $p_{n,m}$. The expected delay incurred is $\bar{D}:=\lim_{n\rightarrow\infty}E[N_{m}]/\log|{\cal M}|=\lim_{n\rightarrow\infty}\frac{n+\gamma_{n}}{nC}=\frac{1}{C}<\infty$ as needed.

III-B Noisy feedback

When the feedback channel is noisy, the above scheme no longer works as i) the transmitter does not have perfect access to the received signal, and hence cannot mimic the decoding process. It is thus harder to ensure zero error; and ii) synchronizing the transmitter and receiver becomes more challenging as both channels are noisy. How can the receiver know when a codeword is new versus when it is repeated? When feedback is noiseless, the synchronization issue can be completely resolved at the transmitter. With noisy feedback, we propose a new synchronization technique.

In [2] an adaptive zero-error communication scheme for DMC with noisy feedback was proposed. The synchronized feedback assisted transmitter and receiver are described using Algorithms 2 and 3, respectively. In these, $s_{t},s_{r}\in\mathcal{B}$ are the current states of the transmitter and receiver respectively. When equal, both transmitter and receiver are working on transmitting a new message; when different, the receiver has decoded the message but the transmitter does not know this yet due to the noisy feedback channel.

Theorem 2

The adaptive zero-error capacity of a forward DMC $W_{(f)}$ with noisy feedback DMC $W_{(b)}$ shown in Fig. 1, denoted by $C_{0fa}^{noisy}$, satisfies

[TABLE]

where $C_{0u}^{(f)}$ and $C_{0u}^{(b)}$ denote the zero-undetected error capacities of the forward and backward links. If furthermore, for some positive functions $A(\cdot)$ and $B(\cdot)$ and some capacity-achieving input distribution $Q^{*}$, $W_{(f)}(y|x)=A(x)B(y)$ holds whenever $Q^{*}(x)W_{(f)}(y|x)>0$, then $C_{0fa}=C^{(f)}$.

Proof  Since $C_{0u}^{(b)}$ is positive, there exists at least one triple $(x^{\prime}_{c},x^{\prime}_{e},y^{\prime}_{c})\in(\mathcal{X}_{(b)}\times\mathcal{X}_{(b)}\times\mathcal{Y}_{(b)})$ such that $W_{(b)}(y^{\prime}_{c}|x^{\prime}_{e})=0$ and $W_{(b)}(y^{\prime}_{c}|x^{\prime}_{c})>0$. Note that we require $C_{0u}^{(f)}$ to be positive as well, else no zero-error communication can take place at all, not even with perfect feedback.

For achievability of (4), take a zero-undetected-error capacity achieving code $C_{0u}(\mathcal{M},n)$ for channel $W_{(f)}$ whose maximal erasure probability tends to zero and whose rate approaches $C_{0u}^{(f)}$. Note that the first message bit $b_{1}$ out of the bit stream of length $k,b_{1}^{k}$ (that is encoded) carries the transmitter’s state variable $s_{t}$, used for synchronization.

To transmit message $m\in{\cal M}$, codeword $c_{0u}^{(n)}(m)$ is sent through $W_{(f)}$. Upon receiving $y^{n}\in\mathcal{Y}_{(f)}^{n}$, the zero-undetected-error decoder is used to obtain an estimate of the message. Since the probability of undetected-error is equal to zero, the only type of error that might occur is an erasure ($|M(y^{n})|>1$, see (1)). If there is an erasure, according to Algorithm 3, the receiver informs the transmitter by sending $\gamma_{n}$ repetitions of the letter $x^{\prime}_{e}$ (i.e. it sends $[x^{\prime}_{e}]^{\gamma_{n}}$). Since $W_{(b)}(y^{\prime}_{c}|x^{\prime}_{e})=0$, it is impossible to receive $y^{\prime}_{c}$ at the transmitter through the noisy feedback channel. Thus, the transmitter – not seeing any $y_{c}^{\prime}$ – again transmits $c_{0u}^{(n)}(m)$.

A message is re-transmitted until the following happens. In the first iteration that $|M(y^{n})|=1$, the receiver sets $\hat{m}=M(y^{n})=b_{2}^{k}$ (recalling that the first bit $b_{1}$ carries the state $s_{t}$ and not the message), and knows with probability 1 that this is the correct message (i.e. zero-error in decoding the message by definition of a zero undetected error code). The challenge now is to tell the transmitter, through the noisy channel, that it has received the message and hence that the transmitter can move on to a new message. This is done through a careful protocol for keeping the binary states $s_{t}$ and $s_{r}$ at the transmitter and receiver synchronized. They start off synchronized to $b_{1}$. Once the receiver sees $|M(y^{n})|=1$, and looks at the decoded message, before sending confirmation that it received the message, the receiver switches its internal state, i.e. $s_{r}=s_{r}^{c}=b_{1}^{c}$ (the complement of the first bit) . Then, it conveys correct decoding by repeating $x^{\prime}_{c}$ $\gamma_{n}$ times through the feedback channel $W_{(b)}$. Two things can now happen:

1. If the letter $y^{\prime}_{c}$ is not received at the transmitter, then the transmitter sends back the same message and the process repeats. At this stage then, $s_{t}=b_{1}$ while $s_{r}=b_{1}^{c}$. This process repeats until the decoder uniquely decodes $|M(y^{n})|=1$ AND the state bits match. If the state bits do not match, the receiver does not update the decoded message.

2. If it does receive $y_{c}^{\prime}$, then it knows the receiver successfully and uniquely decoded the message and hence it sets $s_{t}=s_{t}^{c}$. At this point then the transmitter and receiver states are again equal $s_{t}=s_{r}$. A new message, with the new state again as first bit, is transmitted.

As in the previous case, to calculate the average rate and delay achieved, note that the probability that a message is correctly received with zero error is the probability that the message was correctly received at the receiver after seeing the codeword of length $n$, and then the transmitter (now through a noisy channel) seeing at least one $y_{c}^{\prime}$ in a block of length $\gamma_{n}$. Hence after $n+\gamma_{n}$ channel uses, the probability of correctly decoding message $m$ is p_{n,m}=(1-\lambda_{m}^{(n)})\bigg{[}1-\big{(}1-W_{(b)}(y_{c}^{\prime}|x_{c}^{\prime})\big{)}^{\gamma_{n}}\bigg{]}. Viewing this as a probability of success, the number of codeword re- transmissions needed to transmit message $m$ is hence a geometric random variable $L_{n}(m)$ with $E[L_{n}(m)]=1/p_{n,m}$. The analysis of the achieved average rate and delay is identical to Theorem 1, except that we now use the backward $W_{(b)}(y_{c}^{\prime}|x_{c}^{\prime})$ in the definition of $p_{n,m}$, and the code we use is an undetected error capacity achieving code, in which case the rate tends to $C_{0u}^{(f)}$ as $n\rightarrow\infty$.

To show that our bound is tight for the class of channels stated below (4), note that Csiszár and Narayan showed that if $C_{0u}^{(f)}>0$, and if the conditions after (4) hold then the zero-undetected capacity becomes equal to small error Shannon capacity ($C_{0u}^{(f)}=C^{(f)}$). Thus, for these channels, we can easily prove that $C_{0fa}^{noisy}\geq C^{(f)}$. This is tight, as we always have $C_{0fa}^{noisy}\leq C_{0fa}\leq C^{(f)}$.

IV Conclusion

A major difference between our adaptive-zero-error communication schemes with noiseless versus noisy feedback is that the verification sequence (i.e. transmitter and receiver agreeing the receiver has decoded it successfully) is sent by the transmitter in the noiseless case whereas it is sent by the receiver in the noisy case. In the former, the perfect feedback allows us to approach rates up to $C$ as undetected errors can be caught by the transmitter. In the latter, due to the noisy feedback, our scheme must backoff from $C$ to $C_{0u}$ in order to ensure that no undetected errors occur, as they cannot be corrected by the transmitter under our scheme.

Acknowledgment

The work of the authors was partially supported by NSF under award 1645381. The contents of this article are solely the responsibility of the authors and do not necessarily represent the official views of the NSF.