Successive-Cancellation Decoding of Linear Source Code

TL;DR

This paper analyzes the error probabilities of various decoding methods for source codes with side information, demonstrating the effectiveness of successive-cancellation decoding and its stochastic variant for polar source codes.

Contribution

It establishes the effectiveness of successive-cancellation and stochastic successive-cancellation decoding methods for source codes, including polar codes, in the asymptotic error probability sense.

Findings

Successive-cancellation decoding is effective for source codes with side information.

Stochastic successive-cancellation decoding is also effective for polar source codes.

Error probability approaches zero as code length increases.

Abstract

This paper investigates the error probability of several decoding methods for a source code with decoder side information, where the decoding methods are: 1) symbol-wise maximum a posteriori decoding, 2) successive-cancellation decoding, and 3) stochastic successive-cancellation decoding. The proof of the effectiveness of a decoding method is reduced to that for an arbitrary decoding method, where `effective' means that the error probability goes to zero as $n$ goes to infinity. Furthermore, we revisit the polar source code showing that stochastic successive-cancellation decoding, as well as successive-cancellation decoding, is effective for this code.

Equations

$$\chi (\mathrm{S})\equiv \{\begin{array}{ll}1,& \text{if the statement S is true}\\ 0,& \text{if the statement S is false}.\end{array}$$

$${\hat{\varphi }}_i({\mathbf{c}}_1,\mathbf{y})\equiv \arg \underset{x_i}{\max }{\mu }_{X_i|{\mathbf{C}}_1\mathbf{Y}}(x_i|{\mathbf{c}}_1,\mathbf{y}).$$

$$\mathrm{Prob}(\hat{\mathbf{\varphi }}(A\mathbf{X},\mathbf{Y})\ne \mathbf{X})\le n\mathrm{Prob}(\mathbf{\varphi }(A\mathbf{X},\mathbf{Y})\ne \mathbf{X}),$$

$$\mathrm{Prob}(\hat{\mathbf{\varphi }}(A\mathbf{X},\mathbf{Y})\ne \mathbf{X})$$

$$=\mathrm{Prob}({\hat{\varphi }}_i(A\mathbf{X},\mathbf{Y})\ne X_i\text{for some}i)$$

$$\le \sum _{i=1}^n\mathrm{Prob}({\hat{\varphi }}_i(A\mathbf{X},\mathbf{Y})\ne X_i)$$

$$=\sum _{i=1}^n\mathrm{Prob}(\arg \underset{x_i}{\max }{\mu }_{X_i|{\mathbf{C}}_1\mathbf{Y}}(x_i|A\mathbf{X},\mathbf{Y})\ne X_i)$$

$$\le \sum _{i=1}^n\mathrm{Prob}({\varphi }_i(A\mathbf{X},\mathbf{Y})\ne X_i)$$

$$\le \sum _{i=1}^n\mathrm{Prob}(\mathbf{\varphi }(A\mathbf{X},\mathbf{Y})\ne \mathbf{X})$$

$$\le n\mathrm{Prob}(\mathbf{\varphi }(A\mathbf{X},\mathbf{Y})\ne \mathbf{X}),$$

$$\mathbf{\varphi }({\mathbf{c}}_1,\mathbf{y})\equiv \{\begin{array}{ll}\hat{\mathbf{x}}& \text{if there is a unique}\hat{\mathbf{x}}\in {\mathcal{T}}_{\mathbf{X}|\mathbf{Y},\epsilon }(\mathbf{y})\\ \text{‘error’}& \text{otherwise},\end{array}$$

$${\mathcal{T}}_{\mathbf{X}|\mathbf{Y},\epsilon }(\mathbf{y})\equiv \left\{\mathbf{x}:|\log {\mu }_{\mathbf{X}|\mathbf{Y}}(\mathbf{x}|\mathbf{y})-H(\mathbf{X}|\mathbf{Y})|\le n\epsilon \right\}$$

$$\mathbf{\varphi }({\mathbf{c}}_1,\mathbf{y})$$

$$=\arg \underset{\mathbf{x}}{\max }{\mu }_{\mathbf{X}{\mathbf{C}}_1\mathbf{Y}}(\mathbf{x},{\mathbf{c}}_1,\mathbf{y})$$

$$=\arg \underset{\mathbf{x}:A\mathbf{x}={\mathbf{c}}_1}{\max }{\mu }_{\mathbf{X}\mathbf{Y}}(\mathbf{x},\mathbf{y})$$

$$=\arg \underset{\mathbf{x}:A\mathbf{x}={\mathbf{c}}_1}{\max }{\mu }_{\mathbf{X}|\mathbf{Y}}(\mathbf{x}|\mathbf{y}),$$

$$Q(A\mathbf{x},B\mathbf{x})=\mathbf{x}\text{for all}\mathbf{x}\in {\mathcal{X}}^n.$$

$$\mathbf{\psi }({\mathbf{c}}_1,\mathbf{y})\equiv Q(\mathbf{f}({\mathbf{c}}_1,\mathbf{y})).$$

$$\mathrm{Prob}(\mathbf{\psi }(A\mathbf{X},\mathbf{Y})\ne \mathbf{X})=\mathrm{Prob}(\mathbf{f}({\mathbf{C}}_1,\mathbf{Y})\ne ({\mathbf{C}}_0,{\mathbf{C}}_1)).$$

$$\mathrm{Prob}(\mathbf{\psi }(A\mathbf{X},\mathbf{Y})\ne \mathbf{X})$$

$$=\sum _{\mathbf{x},\mathbf{y}}{\mu }_{\mathbf{X}\mathbf{Y}}(\mathbf{x},\mathbf{y})\chi (\mathbf{\psi }(A\mathbf{x},\mathbf{y})\ne \mathbf{x})$$

$$=\sum _{\mathbf{x},\mathbf{y},{\mathbf{c}}_0,{\mathbf{c}}_1}{\mu }_{\mathbf{X}\mathbf{Y}}(\mathbf{x},\mathbf{y})\chi (A\mathbf{x}={\mathbf{c}}_1)\chi (B\mathbf{x}={\mathbf{c}}_0)$$

$$\cdot \chi (\mathbf{\psi }({\mathbf{c}}_1,\mathbf{y})\ne \mathbf{x})$$

$$=\sum _{\mathbf{x},\mathbf{y},{\mathbf{c}}_0,{\mathbf{c}}_1}{\mu }_{\mathbf{X}\mathbf{Y}}(\mathbf{x},\mathbf{y})\chi (A\mathbf{x}={\mathbf{c}}_1)\chi (B\mathbf{x}={\mathbf{c}}_0)$$

$$\cdot \chi (\mathbf{f}({\mathbf{c}}_1,\mathbf{y})\ne Q^{-1}(\mathbf{x}))$$

$$=\sum _{\mathbf{x},\mathbf{y},{\mathbf{c}}_0,{\mathbf{c}}_1}{\mu }_{\mathbf{X}\mathbf{Y}}(\mathbf{x},\mathbf{y})\chi (A\mathbf{x}={\mathbf{c}}_1)\chi (B\mathbf{x}={\mathbf{c}}_0)$$

$$\cdot \chi (\mathbf{f}({\mathbf{c}}_1,\mathbf{y})\ne (A\mathbf{x},B\mathbf{x}))$$

$$=\sum _{\mathbf{x},\mathbf{y},{\mathbf{c}}_0,{\mathbf{c}}_1}{\mu }_{\mathbf{X}\mathbf{Y}}(\mathbf{x},\mathbf{y})\chi (A\mathbf{x}={\mathbf{c}}_1)\chi (B\mathbf{x}={\mathbf{c}}_0)$$

$$\cdot \chi (\mathbf{f}({\mathbf{c}}_1,\mathbf{y})\ne ({\mathbf{c}}_0,{\mathbf{c}}_1))$$

$$=\sum _{{\mathbf{c}}_0,{\mathbf{c}}_1,\mathbf{y}}{\mu }_{{\mathbf{C}}_0{\mathbf{C}}_1\mathbf{Y}}({\mathbf{c}}_0,{\mathbf{c}}_1,\mathbf{y})\chi (\mathbf{f}({\mathbf{c}}_1,\mathbf{y})\ne ({\mathbf{c}}_0,{\mathbf{c}}_1)),$$

$$=\mathrm{Prob}(\mathbf{f}({\mathbf{C}}_1,\mathbf{Y})\ne ({\mathbf{C}}_0,{\mathbf{C}}_1)),$$

$${\mu }_{{\mathbf{C}}_0{\mathbf{C}}_1\mathbf{Y}}({\mathbf{c}}_0,{\mathbf{c}}_1,\mathbf{y})\equiv {\mu }_{\mathbf{X}\mathbf{Y}}(Q({\mathbf{c}}_1,{\mathbf{c}}_0),\mathbf{y})$$

$${\hat{c}}_i\equiv \{\begin{array}{ll}{f}_i({\hat{c}}_1^{i-1},\mathbf{y})& \text{if}i\in {\mathcal{I}}_0\\ c_i& \text{if}i\in {\mathcal{I}}_1\end{array}$$

$$f_i(c_1^{i-1},\mathbf{y})\equiv \arg \underset{c_i}{\max }{\mu }_{C_i|C_1^{i-1}\mathbf{Y}}(c_i|c_1^{i-1},\mathbf{y}),$$

$${\mu }_{C_i|C_1^{i-1}\mathbf{Y}}(c_i|c_1^{i-1},\mathbf{y})\equiv \frac{{\sum }_{c_{i+1}^n}{\mu }_{{\mathbf{C}}_0{\mathbf{C}}_1\mathbf{Y}}({\mathbf{c}}_0,{\mathbf{c}}_1,\mathbf{y})}{{\sum }_{c_i^n}{\mu }_{{\mathbf{C}}_0{\mathbf{C}}_1\mathbf{Y}}({\mathbf{c}}_0,{\mathbf{c}}_1,\mathbf{y})}$$

$$\mathrm{Prob}(\mathbf{f}({\mathbf{C}}_1,\mathbf{Y})\ne ({\mathbf{C}}_0,{\mathbf{C}}_1))$$

$$\le \sum _{i\in {\mathcal{I}}_0}\mathrm{Prob}(f_i({\mathbf{C}}_1^{i-1},\mathbf{Y})\ne C_i).$$

$${\mathcal{E}}_i\equiv \left\{(\mathbf{c},\mathbf{y}):\begin{array}{rl}& f_j(c_1^{j-1},\mathbf{y})=c_j\text{for all}j\in \{1,\dots ,i-1\}\\ & f_i(c_1^{i-1},\mathbf{y})\ne c_i\end{array}\right\}$$

$$\mathrm{Prob}(\mathbf{f}({\mathbf{C}}_1,\mathbf{Y})\ne ({\mathbf{C}}_0,{\mathbf{C}}_1))$$

$$=\mathrm{Prob}(({\mathbf{C}}_0,{\mathbf{C}}_1,\mathbf{Y})\in \mathcal{E})$$

$$\le \sum _{i=1}^n\mathrm{Prob}(({\mathbf{C}}_0,{\mathbf{C}}_1,\mathbf{Y})\in {\mathcal{E}}_i)$$

$$=\sum _{i\in {\mathcal{I}}_0}\mathrm{Prob}(({\mathbf{C}}_0,{\mathbf{C}}_1,\mathbf{Y})\in {\mathcal{E}}_i),$$

Full text

1. symbol-wise maximum a posteriori decoding,
2. successive-cancellation decoding, and
3. stochastic successive-cancellation decoding. The proof of the effectiveness of a decoding method is reduced to that for an arbitrary decoding method, where ‘effective’ means that the error probability goes to zero as $n$ goes to infinity. Furthermore, we revisit the polar source code showing that stochastic successive-cancellation decoding, as well as successive-cancellation decoding, is effective for this code.

I Introduction

Successive-cancellation (SC) decoding is one of the elements constituting the polar code introduced by Arıkan [1]. This paper investigates the error probability of SC decoding for a source code with decoder side information by extending the results in [2, 12] to general linear source codes. It is shown that if for a given encoder there is a decoder such that the block error probability is $o(1/n)$, then the block error probability of an SC decoder for the same encoder is $o(1)$. Furthermore, we introduce stochastic successive-cancellation (SSC) decoding and show that it is equivalent to the constrained-random-number generator introduced in [7]. It is shown that if for a given encoder there is a decoder such that the block error probability is $o(1)$, then the block error probability of an SC decoder for the same encoder is $o(1)$. It is also shown that the error probability of the symbol-wise maximum a posteriori decoding of a linear source code and the SSC decoder of the polar source code goes to zero as the block length goes to infinity.

It should be noted that the results of this paper can be applied to the channel coding as introduced in [2, 10, 12]. In particular, the syndrome decoding is the case when a channel is additive, a parity check matrix corresponds to a source encoding function, the syndrome of a channel output corresponds to a codeword of the source code without decoder side information, and the kernel of the parity check matrix forms the channel inputs, that is, the codewords for a channel code.

Throughout the paper, we use the following notations. For random variable $U$, let $\mathcal{U}$ be the alphabet of $U$, $\mu_{U}$ be the distribution of $U$, and $\mu_{U|V}$ be the conditional distribution of $U$ for a given random variable $V$. Let $H(U|V)$ be the conditional entropy of $U$ for a given $V$, where we assume that the base of $\log$ is the cardinality $|\mathcal{U}|$ of $\mathcal{U}$. A column vector is denoted by a boldface letter $\boldsymbol{u}$, where its dimension depends on the context. We define $u_{i}^{j}\equiv(u_{i},\ldots,u_{j})$, where $u_{i}^{j}$ is the null string when $i>j$. Let $\chi(\cdot)$ be a support function defined as

[TABLE]

II Symbol-wise Maximum A Posteriori Decoding

First, we revisit symbol-wise maximum a posteriori (SMAP) decoding, which is used for the conventional decoding of a low density parity check code. Although the symbol error rate (the Hamming distance between a source output and its reproduction divided by the block length $n$) is discussed with symbol-wise maximum a posteriori decoding, we focus on the block error probability (an error occurs when a source output and its reproduction are different, that is, the Hamming distance is positive) throughout this paper.

Let $(A,\boldsymbol{\phi})$ be a pair consisting of a source encoder $A:\mathcal{X}^{n}\to\mathcal{X}^{l}$ and a decoder $\boldsymbol{\phi}:\mathcal{X}^{l}\times\mathcal{Y}^{n}\to\mathcal{X}^{n}$ with side information. Let $\boldsymbol{c}_{1}\equiv A\boldsymbol{x}$ be the codeword of a source output $\boldsymbol{x}\in\mathcal{X}^{n}$. The decoder $\widehat{\boldsymbol{\phi}}\equiv\{\widehat{\phi}_{i}\}_{i=1}^{n}$ is constructed by using functions reproducing the $i$-th coordinate as

[TABLE]

It should be noted that when $(\boldsymbol{X},\boldsymbol{Y})\equiv(X^{n},Y^{n})$ is memoryless and $A$ is a sparse matrix we can use the sum-product algorithm to obtain an approximation of $\mu_{X_{i}|\boldsymbol{C}_{1}\boldsymbol{Y}}(x_{i}|\boldsymbol{c}_{1},\boldsymbol{y})$.

We have the following theorem.

Theorem 1

The error probability of the code $(A,\widehat{\boldsymbol{\phi}})$ is bounded as

[TABLE]

where the right hand side of this inequality goes to zero as $n\to\infty$ when $\mathrm{Prob}(\boldsymbol{\phi}(A\boldsymbol{X},\boldsymbol{Y})\neq\boldsymbol{X})=o(1/n)$.

Proof:

Let $\phi_{i}(\boldsymbol{c}_{1},\boldsymbol{y})$ be the $i$-th coordinate of $\boldsymbol{\phi}(\boldsymbol{c}_{1},\boldsymbol{y})$. Then we have

[TABLE]

where the first inequality comes from the union bound, the second inequality comes from the fact that the maximum a posteriori decision minimizes the error probability, and the third inequality comes from the fact that $\phi_{i}(\boldsymbol{c}_{1},\boldsymbol{y})\neq x_{i}$ implies $\boldsymbol{\phi}(\boldsymbol{c}_{1},\boldsymbol{y})\neq\boldsymbol{x}$. ∎

It is known that, when $l/n>H(X^{n}|Y^{n})/n$ there is an encoding function $A:\mathcal{X}^{n}\to\mathcal{X}^{l}$ such that error probability $\mathrm{Prob}(\boldsymbol{\phi}(A\boldsymbol{X},\boldsymbol{Y})\neq\boldsymbol{X})$ is close to zero for all sufficiently large $n$ [5, 11], where we can use one of the following decoders:

• •

the typical set decoder defined as

[TABLE]

where

[TABLE]

is a conditional typical set,

• •

the maximum a posteriori probability decoder111The right hand side of the third equality of (2) might be called the maximum-likelihood decoder. defined as

[TABLE]

where the third equality comes from the fact that $\mu_{\boldsymbol{X}\boldsymbol{C}_{1}\boldsymbol{Y}}(\boldsymbol{x},\boldsymbol{c}_{1},\boldsymbol{y})=\mu_{\boldsymbol{X}\boldsymbol{Y}}(\boldsymbol{x},\boldsymbol{y})$ when $A\boldsymbol{x}=\boldsymbol{c}_{1}$ and $\mu_{\boldsymbol{X}\boldsymbol{C}_{1}\boldsymbol{Y}}(\boldsymbol{x},\boldsymbol{c}_{1},\boldsymbol{y})=0$ when $A\boldsymbol{x}\neq\boldsymbol{c}_{1}$.

The following sections show upper bounds of the error probability for several decoders in terms of the error probability of a code $(A,\boldsymbol{\phi})$, where $\boldsymbol{\phi}$ is an arbitrary decoder. It should be noted that we can use one of the decoders mentioned above. We can reduce the effectiveness of the decoders to that of an arbitrary decoder, where ‘effective’ means that the error probability goes to zero as $n$ goes to infinity. For example, [9, 10] show that a decoder using a constrained-random-number generator is effective by showing that the maximum a posteriori probability decoder is effective.

III Decoding Extended Codeword

Let $A:\mathcal{X}^{n}\to\mathcal{X}^{l}$ be an encoder of a source code with decoder side information. Here, we assume that, for a given $A$ there is a function $B:\mathcal{X}^{n}\to\mathcal{X}^{n-l}$ and a bijection $Q:\mathcal{X}^{n}\to\mathcal{X}^{n}$ such that

[TABLE]

In particular, this condition is satisfied when $A$ is a full-rank matrix. We define the bijection $[A,B]:\mathcal{X}^{n}\to\mathcal{X}^{n}$ as $[A,B]\boldsymbol{x}\equiv(A\boldsymbol{x},B\boldsymbol{x})$.

Let $\mathcal{I}_{0}$ and $\mathcal{I}_{1}$ be a partition of $\mathcal{N}\equiv\{1,\ldots,n\}$, that is, they satisfy $I_{0}\cap\mathcal{I}_{1}=\emptyset$ and $\mathcal{I}_{0}\cup\mathcal{I}_{1}=\mathcal{N}$. We call $\mathcal{I}_{0}$ and $\mathcal{I}_{1}$ ordered when $\mathcal{I}_{1}=\{1,\ldots,l\}$ and $\mathcal{I}_{0}=\{l+1,\ldots,n\}$. For a vector $\boldsymbol{c}\equiv(c_{1},\ldots,c_{n})\in\mathcal{X}^{n}$, define $\boldsymbol{c}_{0}\in\mathcal{X}^{n-l}$ and $\boldsymbol{c}_{1}\in\mathcal{X}^{l}$ so that $c_{i}$ is a symbol in $\boldsymbol{c}_{b}$ when $i\in\mathcal{I}_{b}$ for every $b\in\{0,1\}$. In the following, we assume that $A\boldsymbol{x}=\boldsymbol{c}_{1}$ and $B\boldsymbol{x}=\boldsymbol{c}_{0}$, where corresponding index sets $\mathcal{I}_{1}$ and $\mathcal{I}_{0}$ may not be ordered in the bijection $[A,B]$. We call $(\boldsymbol{c}_{0},\boldsymbol{c}_{1})$ the extended codeword of $\boldsymbol{c}_{1}$. In the following, we denote $\boldsymbol{c}=(\boldsymbol{c}_{0},\boldsymbol{c}_{1})$ omitting the dependence on $(\mathcal{I}_{0},\mathcal{I}_{1})$.

Let $\boldsymbol{f}:\mathcal{X}^{l}\times\mathcal{Y}^{n}\to\mathcal{X}^{n}$ be a function that reproduces the extended codeword by using the side information. For a codeword $\boldsymbol{c}_{1}\in\mathcal{X}^{l}$ and side information $\boldsymbol{y}\in\mathcal{Y}^{n}$, the source decoder $\boldsymbol{\psi}$ with side information is defined as

[TABLE]

In the context of the polar source codes, $\boldsymbol{c}_{0}$ corresponds to unfrozen symbols and $Q$ corresponds to the final step of SC decoding. We have the following lemma for a general case.

Lemma 1

Let $\boldsymbol{C}_{1}\equiv A\boldsymbol{X}$ and $\boldsymbol{C}_{0}\equiv B\boldsymbol{X}$. Then we have

[TABLE]

Proof:

We have

[TABLE]

where the third equality comes from the fact that $Q$ is bijective, and in the sixth equality we define

[TABLE]

and use the fact that for all $\boldsymbol{c}_{0}$ and $\boldsymbol{c}_{1}$ there is a unique $\boldsymbol{x}$ satisfying $A\boldsymbol{x}=\boldsymbol{c}_{1}$ and $B\boldsymbol{x}=\boldsymbol{c}_{0}$. ∎

In the following, we investigate the decoding error probability for an extended codeword.

IV Successive-Cancellation Decoding

This section investigates the error probability of the (deterministic) SC decoding. For a source encoder $A:\mathcal{X}^{n}\to\mathcal{X}^{l}$, let $B$, $Q$, $\boldsymbol{C}_{0}$, and $\boldsymbol{C}_{1}$ be defined as in the previous section.

For a codeword $\boldsymbol{c}_{1}\in\mathcal{X}^{l}$ and side information $\boldsymbol{y}\in\mathcal{Y}^{n}$, the output $\widehat{\boldsymbol{c}}\equiv\boldsymbol{f}(\boldsymbol{c}_{1},\boldsymbol{y})$ of an SC decoder $\boldsymbol{f}$ is defined recursively as

[TABLE]

by using functions $\{f_{i}\}_{i\in\mathcal{I}_{0}}$ defined as

[TABLE]

which is known as the maximum a posteriori decision rule, where $\mu_{C_{i}|C_{1}^{i-1}\boldsymbol{Y}}$ is the conditional probability defined as

[TABLE]

by using $\mu_{\boldsymbol{C}_{0}\boldsymbol{C}_{1}\boldsymbol{Y}}$ defined by (6).

To simplify the notation, we define $f_{i}(\widehat{c}_{1}^{i-1},\boldsymbol{y})\equiv c_{i}$ when $i\in\mathcal{I}_{1}$ although $c_{i}$ does not depend on $\widehat{c}_{1}^{i-1}$ and $\boldsymbol{y}$. We have the following lemma.

Lemma 2

[TABLE]

Proof:

As with the proof in [1], we can express the block error events $\boldsymbol{f}(\boldsymbol{c}_{1},\boldsymbol{y})\neq(\boldsymbol{c}_{0},\boldsymbol{c}_{1})$ as $\mathcal{E}\equiv\bigcup_{i=1}^{n}\mathcal{E}_{i}$, where

[TABLE]

is an event where the first decision error in SC decoding occurs at stage $i$. The decoding error probability for a extended codeword is evaluated as

[TABLE]

where the first inequality comes from the union bound, the second equality comes from the fact that $f_{i}(\boldsymbol{C}_{1}^{i-1},\boldsymbol{Y})=C_{i}$ when $i\in\mathcal{I}_{1}$, and the last inequality comes from the fact that $(\boldsymbol{c}_{0},\boldsymbol{c}_{1},\boldsymbol{y})\in\mathcal{E}_{i}$ implies $f_{i}(c_{1}^{i-1},\boldsymbol{y})\neq c_{i}$. ∎

When the index sets $\mathcal{I}_{1}$ and $\mathcal{I}_{0}$ are not ordered like the polar source codes [2, 12], $f_{i}$ defined by (7) may not use the full information of a codeword $\boldsymbol{c}_{1}\equiv\{c_{i}\}_{i\in\mathcal{I}_{1}}$. Borrowing words from [1], $f_{i}$ treats future symbols as random variables rather than as known symbols. In other words, $f_{i}$ ignores the future symbols in a codeword $\boldsymbol{c}_{1}$. This implies that $\{f_{i}\}_{i=1}^{n}$ is different from the optimum maximum a posteriori decoder defined as

[TABLE]

The following investigates the error probability of the (deterministic) SC decoding by assuming that the index sets $\mathcal{I}_{1}$ and $\mathcal{I}_{0}$ are ordered, that is, $\mathcal{I}_{1}=\{1,\ldots,l\}$ and $\mathcal{I}_{0}=\{l+1,\ldots,n\}$. This implies that for every $i\in\mathcal{I}_{0}$, $f_{i}$ defined by (7) uses the full information of a codeword $\{c_{i}\}_{i\in\mathcal{I}_{1}}$.

Lemma 3

For a source encoder $A:\mathcal{X}^{n}\to\mathcal{X}^{l}$ and decoder $\boldsymbol{\phi}:\mathcal{X}^{l}\times\mathcal{Y}^{n}\to\mathcal{X}^{n}$ with side information, let $B$, $Q$, $\boldsymbol{C}_{0}$, and $\boldsymbol{C}_{1}$ be as defined in the previous section, where it is assumed that the index sets $\mathcal{I}_{1}$ and $\mathcal{I}_{0}$ are ordered. Then we have

[TABLE]

for all $i\in\mathcal{I}_{0}$.

Proof:

For $i\in\mathcal{I}_{0}$, let $f^{\prime}_{i}(\boldsymbol{c}_{1},\boldsymbol{y})$ be the $i$-th coordinate of the extended codeword of $Q^{-1}(\boldsymbol{\phi}(\boldsymbol{c}_{1},\boldsymbol{y}))$. Then we have the fact that

[TABLE]

for all $\boldsymbol{x}$ satisfying $A\boldsymbol{x}=\boldsymbol{c}_{1}$ and $B\boldsymbol{x}=\boldsymbol{c}_{0}$, where the second equivalence comes from the fact that $Q$ is bijective, and the third equivalence comes from (3). Then we have

[TABLE]

where the first inequality comes from Lemma 7 in the Appendix and the fact that $\boldsymbol{C}_{1}=C_{1}^{l}$, the second inequality comes from the fact that the maximum a posteriori decision rule minimizes the decision error probability, and the last inequality comes from (10). ∎

From Lemmas 1–3 and the fact that $|\mathcal{I}_{0}|\leq n$, we have the following theorem, which implies that SC decoding is effective when for a given encoding function $A$ there is an effective decoding function $\boldsymbol{\phi}$.

Theorem 2

For a source code $(A,\boldsymbol{\phi})$ with decoder side information, error probability of the (deterministic) SC decoding $\boldsymbol{\psi}$ is bounded as

[TABLE]

where the right hand side of this inequality goes to zero as $n\to\infty$ when $\mathrm{Prob}(\boldsymbol{\phi}(A\boldsymbol{X},\boldsymbol{Y})\neq\boldsymbol{X})=o(1/n)$.

It should be noted again that the index sets $\mathcal{I}_{1}$ and $\mathcal{I}_{0}$ are ordered, while they are not ordered in the original polar source code. In contrast, we can use an arbitrary function $B$ that satisfies the assumption and rearrange the index sets $\mathcal{I}_{1}$ and $\mathcal{I}_{0}$ so that they are ordered, while they are fixed in the original polar source code.

V Stochastic Successive-Cancellation Decoding

This section introduces stochastic successive-cancellation (SSC) decoding, which is known as randomized rounding in the context of polar codes.

When $i\in\mathcal{I}_{0}$, we replace $f_{i}$ defined in (7) by the stochastic decision rule generating $c_{i}$ randomly subject to the probability distribution $\{\mu_{C_{i}|C_{1}^{i-1}\boldsymbol{Y}}(c^{\prime}_{i}|c_{1}^{i-1},\boldsymbol{y})\}_{c^{\prime}_{i}\in\mathcal{X}}$ for a given $(c_{1}^{i-1},\boldsymbol{y})$. Let $F_{i}$ be the stochastic decision rule described above. Let $\boldsymbol{F}$ be the stochastic decoder by using $F_{i}$ instead of $f_{i}$ when $i\in\mathcal{I}_{0}$. We denote the stochastic decoder corresponding to (4) by $\boldsymbol{\Psi}$. An analysis of the error probability will be presented in the next section.

VI Implementation of Successive-Cancellation Decoding

In this section, we assume that $A$ is a full-rank $l\times n$ (sparse) matrix. Without loss of generality, we can assume that the right part of $A$ is an invertible $l\times l$ matrix. This condition is satisfied for an arbitrary full-rank matrix $A$ by using a permutation matrix $S$, where $AS$ satisfies the condition, and the codeword can be obtained as $A\boldsymbol{x}=AS[S^{-1}\boldsymbol{x}]$.

Let $B$ be an $[n-l]\times n$ matrix, where the left part of $B$ is an invertible $[n-l]\times[n-l]$ matrix. Then we have the fact that by concatenating row vectors of $B$ to $A$, we obtain the invertible $n\times n$ matrix $[A,B]$, that is, $[A,B]$ is bijective. By using $A$ and $B$, we can construct a successive-cancellation decoder that reproduces an extended codeword with $\mathcal{I}_{1}=\{1,\ldots,l\}$ and $\mathcal{I}_{0}=\{l+1,\ldots,n\}$.

Here, let us assume that the left part of $B$ is the $[n-l]\times[n-l]$ identity matrix and the right part of $B$ is the $[n-l]\times l$ zero matrix. It should be noted that a similar discussion is possible when the identity matrix is replaced by a permutation matrix.

Since the left part of $B$ is the $[n-l]\times[n-l]$ identity matrix, then, for all $i\in\{0,\ldots,l-1\}$, the $(i,i)$-element of $[A,B]$ is $1$, which is the only positive element in $i$-th row of $[A,B]$. Then we have the fact that

[TABLE]

which implies $\boldsymbol{C}_{0}=X_{1}^{n-l}$.

First, we reduce the conditional probability $\mu_{C_{i}|C_{1}^{i-1}\boldsymbol{Y}}(c_{i}|c_{1}^{i-1},\boldsymbol{y})$ defined by (8). For $i\in\{l+1,\ldots,n\}$ and $j\equiv i-l$, we have

[TABLE]

where the third equality comes from the fact that $\mathcal{I}_{1}=\{1,\ldots,l\}$ and the fourth equality comes from Lemma 8 in the Appendix and the fact that $C_{l+j}=X_{j}$ for all $j\in\{1,\ldots,n-l\}$. By substituting $c_{l+1}^{i}=x_{1}^{j}$, we have

[TABLE]

for $i\in\{l+1,\ldots,n\}$ and $j\equiv i-l$. It should be noted that the right hand side of the second equality appears in the constrained-random-number generation algorithm [7, Eq. (41)]222In [7, Eq. (41)], $\mu_{X_{j}|X_{1}^{j-1}}$ should be replaced by $\mu_{X_{j}|X_{1}^{j-1}\boldsymbol{Y}}$.. This implies that the constrained-random-number generator can be considered as an SSC decoding $\boldsymbol{\Psi}$ of the extended codeword specified in the previous section, where we have assumed that this algorithm uses the full information of the codeword $c_{1}^{l}$ for every $i\in\{l+1,\ldots,n\}$.

Next, we assume that $(X^{n},Y^{n})$ is memoryless and reduce the condition $A\boldsymbol{x}=\boldsymbol{c}_{1}$ to improve the algorithm. This idea has already been presented in [8]. Let $\boldsymbol{a}_{j}$ be the $j$-th column vector of $A$. Let $A_{1}^{j-1}$ be the sub-matrix of $A$ obtained by using $\{\boldsymbol{a}_{j^{\prime}}\}_{j^{\prime}=1}^{j-1}$ and $A_{j}^{n}$ be that obtained by using $\{\boldsymbol{a}_{j^{\prime}}\}_{j^{\prime}=j}^{n}$. At the computation of (13) for $j\in\{1,\ldots,n-l\}$, we can assume that $x_{1}^{j-1}$ has already been determined. Furthermore, we have the fact that the condition $A\boldsymbol{x}=\boldsymbol{c}_{1}$ is equivalent to $A_{j}^{n}x_{j}^{n}=\boldsymbol{c}_{1}-A_{1}^{j-1}x_{1}^{j-1}$. Then, by letting $\boldsymbol{c}^{\prime}_{1}(j)\equiv\boldsymbol{c}_{1}-A_{1}^{j-1}x_{1}^{j-1}$, we can reduce (13) as follows:

[TABLE]

It should be noted that we can obtain $A_{j}^{n}$ recursively by deleting the left-end column vector of $A_{j-1}^{n}$. We can obtain the vector $\boldsymbol{c}^{\prime}_{1}(j)$ recursively by using the relations

[TABLE]

These operations reduce the computational complexity of the algorithm. It should also be noted that the sum-product algorithm is available for the approximate computation of (14) when $A$ is a sparse matrix.

Next, we convert the reproduction of a extended codeword to the reproduction of a source output. When $j=n-l$, we have obtained the extended codeword $(\boldsymbol{c}_{1},\boldsymbol{c}_{0})$, where $\boldsymbol{c}_{0}\equiv c_{l+1}^{n}=x_{1}^{n-l}$. We can reproduce the source output $\boldsymbol{x}$ by using the relation $\boldsymbol{x}\equiv[A,B]^{-1}\boldsymbol{c}$, where $[A,B]^{-1}$ is the inverse of the concatenation of $A$ and $B$. Then we have the relations

[TABLE]

from the assumptions of $A$ and $B$. Since

[TABLE]

we obtain $x_{n-l+1}^{n}$ as

[TABLE]

where $[A_{n-l+1}^{n}]^{-1}$ is the inverse of $A_{n-l+1}^{n}$.

Finally, we summarize the decoding algorithm. We assume that $(X^{n},Y^{n})$ is memoryless, $A$ is an $l\times n$ (sparse) matrix satisfying that $A_{n-l+1}^{n}$ is an $l\times l$ invertible matrix, and $B$ is an $[n-l]\times n$ matrix satisfying that $B_{1}^{n-l}$ is an $[n-l]\times[n-l]$ identity matrix.

SC/SSC Decoding Algorithm Using Sum-Product Algorithm:

• Step 1

Let $j\leftarrow 1$ and $\boldsymbol{c}^{\prime}_{1}\leftarrow\boldsymbol{c}_{1}$.

• Step 2

Calculate the conditional probability distribution $\mu_{C_{l+j}|C_{1}^{l+j-1}\boldsymbol{Y}}$ as

[TABLE]

by using $x_{1}^{j-1}$, $y_{j+1}^{n}$, $A_{j}^{n}$, and $\boldsymbol{c}^{\prime}_{1}$, where we define $c_{l+1}^{l+j}\equiv x_{1}^{j}$. It should be noted that the sum-product algorithm can be employed to obtain an approximation of (15).

• Step 3

For the deterministic SC decoding, let $x_{j}$ be defined as

[TABLE]

For the SSC decoding, generate and record a random number $x_{j}$ subject to the distribution $\{\mu_{C_{l+j}|C_{1}^{l+j-1}\boldsymbol{Y}}(x^{\prime}_{j}|x_{1}^{j-1},\boldsymbol{y})\}_{x^{\prime}_{j}\in\mathcal{X}}$.

• Step 4

Let $\boldsymbol{c}^{\prime}_{1}\leftarrow\boldsymbol{c}^{\prime}_{1}-x_{j}\boldsymbol{a}_{j}$.

• Step 5

If $j=n-l$, then compute $x_{l+1}^{n}\equiv[A_{n-l+1}^{n}]^{-1}\boldsymbol{c}^{\prime}_{1}$, output $x_{1}^{n}$ and terminate.

• Step 6

Let $j\leftarrow j+1$ and go to Step 2.

Since the SSC decoder is equivalent to a constrained-random-number generator generating a random sequence subject to the a posteriori probability distribution $\mu_{\boldsymbol{X}|\boldsymbol{C}_{1}\boldsymbol{Y}}$ [7, Theorem 5], we have the following theorem from the fact that the error probability of a stochastic decision with an a posteriori probability distribution is at most twice that of any decision rule [9, Lemma 3].

Theorem 3

For a linear source code $(A,\phi)$ with decoder side information, the decoding error of the SSC decoding algorithm is bounded as

[TABLE]

where the right hand side of this inequality goes to zero as $n\to\infty$ when $\mathrm{Prob}(\boldsymbol{\phi}(A\boldsymbol{X},\boldsymbol{Y})\neq\boldsymbol{X})=o(1)$.

VII Analysis When Index Sets Are Not Ordered

In the previous sections, it was assumed that the index sets $\mathcal{I}_{1}$ and $\mathcal{I}_{0}$ corresponding to $\boldsymbol{c}_{1}=A\boldsymbol{x}$ and $\boldsymbol{c}_{0}=B\boldsymbol{x}$ are ordered, that is, $\mathcal{I}_{1}=\{1,\ldots,l\}$ and $\mathcal{I}_{0}=\{l+1,\ldots,n\}$. This section investigates the case when they are not ordered. The following lemma asserts that the effectiveness of the decoder is reduced to a condition where the sum of the conditional entropies corresponding to the complement of the codeword goes to zero as $n\to\infty$.

Lemma 4

Let $\boldsymbol{\psi}$ and $\boldsymbol{\Psi}$ be the SC and SSC decoding functions, respectively. Then

[TABLE]

Proof:

The first inequality is shown from Lemmas 1–3 as

[TABLE]

where the last inequality comes from the relation

[TABLE]

shown in [4]. The second inequality is shown similarly as

[TABLE]

where the second inequality comes from [9, Lemma 3]. ∎

The above lemma implies that the error probability of SC/SSC decoding is small when $\sum_{i\in\mathcal{I}_{0}}H(C_{i}|C_{1}^{i-1},\boldsymbol{Y})$ is small. The following lemma introduces quasi-polarization, where the both (18) and (19) are satisfied for all $\delta>0$ and sufficiently large $n$. It should be noted here that (18) implies that $H(C_{i}|C_{1}^{i-1})$ is close to [math] but (19) may not imply that $H(C_{i}|C_{1}^{i-1})$ is close to $1$.

Lemma 5

The condition

[TABLE]

is equivalent to the condition

[TABLE]

Proof:

Since $[A,B]$ is bijective, we have the fact that $H(A\boldsymbol{X},B\boldsymbol{X}|\boldsymbol{Y})=H(\boldsymbol{X}|\boldsymbol{Y})$. Then the condition (19) is derived from (18) as

[TABLE]

and the condition (18) is derived from (19) as

[TABLE]

∎

The following lemma asserts that we have the quasi-polarization when the SC/SSC decoding is effective in the sense that the binary entropy of the error probability is $o(1/n)$.

Lemma 6

Let $\boldsymbol{\psi}$ and $\boldsymbol{\Psi}$ be the SC and SSC decoding functions, respectively. Let $h(\theta)\equiv-\theta\log(\theta)-[1-\theta]\log(1-\theta)$ be the binary entropy function, where the base of $\log$ is $|\mathcal{X}|$. Then we have

[TABLE]

where

[TABLE]

Proof:

In the following, we show the first inequality, where we can show the second inequality similarly. Let

[TABLE]

Then we have

[TABLE]

where the first inequality comes from the Fano inequality, the second inequality comes from the fact that $f_{i}(c_{1}^{i-1},\boldsymbol{y})\neq c_{i}$ implies $\boldsymbol{\psi}(A\boldsymbol{x},\boldsymbol{y})\neq\boldsymbol{x}$ and the last inequality comes from the fact that $|\mathcal{I}_{0}|\leq n$. ∎

Remark 1

We have several interpretations of Lemmas 4 and 6. Lemma 4 asserts that the SC/SSC decoding is effective when we have the quasi-polarization and Lemma 6 asserts that the SC/SSC decoding is not effective when we do not have the sufficient333In this statement, ‘sufficient’ means that $\sum_{i\in\mathcal{I}_{0}}H(C_{i}|C^{i-1},Y^{n})=o(n)$. quasi-polarization. Conversely, Lemma 6 asserts that we have the quasi-polarization when the SC/SSC decoding is sufficiently effective and Lemma 4 asserts that we do not have the quasi-polarization when the SC/SSC decoding is not effective.

Remark 2

It is mentioned in [3, “Polarization is commonplace”] that a random permutation of the set $\{0,1\}^{n}$ is a good polarizer with a high probability. We can show a similar fact regarding a good source code $(A,\boldsymbol{\phi})$ and a matrix $B$ that introduces extended codewords, where the index sets $\mathcal{I}_{1}$ and $\mathcal{I}_{0}$ are ordered. We have a slightly tighter bound than Lemma 6 as follows:

[TABLE]

where $e$ is the base of the natural logarithm, the second inequality comes from [6, Lemma 3.12], the third inequality comes from the Fano inequality, and the fourth inequality comes from the fact that

[TABLE]

by using the relation $\log(1+\theta)\leq\theta\log e$. This means that we have quasi-polarization when $\varepsilon=o(1/n)$. In particular, when $\varepsilon$ goes to zero exponentially as $n\to\infty$, $\sum_{i=l+1}^{n}H(C_{i}|C_{1}^{i-1},\boldsymbol{Y})$ also goes to zero exponentially. It should be noted that the combination of Lemma 4 and (23) provides bounds slightly different from those provided by Theorems 2 and 3.

It is a future challenge to find the function $B$ and the order of the index sets for a general linear code where $\sum_{i\in\mathcal{I}_{0}}H(C_{i}|C_{1}^{i-1},\boldsymbol{Y})$ is small or $\sum_{i\in\mathcal{I}_{1}}H(C_{i}|C_{1}^{i-1},\boldsymbol{Y})$ is close to $H(\boldsymbol{X}|\boldsymbol{Y})$. We can expect to reduce the time complexity of SC/SSC decoding while maintaining sufficient precision of the computation for the conditional probability distribution $\mu_{C_{i}|C_{1}^{i-1}\boldsymbol{Y}}$.

VIII Stochastic Successive-Cancellation Decoding of Polar Source Code

In this section, we revisit the polar source code for a pair $(X,Y)$ of stationary memoryless source introduced in [2, 12]. For simplicity, we assume that $|\mathcal{X}|$ is a prime number. For a given positive integer $k$, let $n\equiv 2^{k}$. The source polarization transform $G$ is defined as

[TABLE]

where $\otimes k$ denotes the $k$-th Kronecker power and $S_{\mathrm{BR}}$ is the bit-reversal permutation matrix defined in [1]. Then the extended codeword $\boldsymbol{c}\in\mathcal{X}^{n}$ of a source output $\boldsymbol{x}\in\mathcal{X}^{n}$ is defined as $\boldsymbol{c}\equiv{}^{t}S_{\mathrm{BR}}{}^{t}G\boldsymbol{x}$, where both $\boldsymbol{c}$ and $\boldsymbol{x}$ are column vectors.

From [12, Theorem 4.10], we have

[TABLE]

for all $\beta\in(0,1/2)$, where $Z$ is the source Bhattacharyya parameter defined as

[TABLE]

Let $\mathcal{I}_{0}$ and $\mathcal{I}_{1}$ be defined as

[TABLE]

Then, from (25), we have the fact that the encoding rate $|\mathcal{I}_{1}|/n$ approaches $H(X|Y)$ as

[TABLE]

Furthermore, from Lemma 4, we have

[TABLE]

for all $\beta\in(0,1/2)$, where the second inequality comes from the relation

[TABLE]

shown in [2, Eq. (5)], [12, Eq. (4.11)]. This implies the well-known fact that SC decoding of the polar source code is effective [2, 12].

Similarly, we have the following theorem, which implies the effectiveness of the SSC decoding of the polar source code.

Theorem 4

[TABLE]

for all $\beta\in(0,1/2)$.

IX Concluding Remarks

It should be noted that we cannot judge from Theorems 1–3 which decoder (SMAP, SC, or SSC) performs the best when we use the same encoding function $A$. It is a future challenge to clarify the best decoder theoretically or empirically.

In Theorems 1–3, we have assumed that we can compute the conditional probability distribution defined by (15) exactly. However, the sum-product algorithm may not provide the exact computation of (15). It is a future challenge to estimate the approximation error caused by the sum-product algorithm and to introduce an alternative algorithm that provides a good approximation.

The following comments on the computational complexity of the decoding algorithms. When $A$ is a sparse matrix with the maximum row weight $w$ and we use the Fourier transform444 When $|\mathcal{X}|$ is a power of a prime $p$, the term $|\mathcal{X}|^{2}$ can be replaced by $|\mathcal{X}|\log_{p}|\mathcal{X}|$ by using the Fast-Fourier-Transform. to compute the convolutions in the sum-product algorithm, the computational complexity of SMAP decoding is $O(\iota lw[|\mathcal{X}|^{2}+|\mathcal{X}|w])$, where $\iota$ denotes the number of iterations of the sum-product algorithm. The computational complexity of SC/SSC decoding is $O(\iota[n-l]lw[|\mathcal{X}|^{2}+|\mathcal{X}|w])$. The computational complexity of the SC/SSC decoding of the polar source code is $O(|\mathcal{X}|^{2}n\log n)$ by using the recursive construction of $G$ [12, Section 4.4].

Lemma 7

For any triplet $(U,V,W)$ of random variables, we have

[TABLE]

Proof:

For all $v$, we have

[TABLE]

Then we have

[TABLE]

∎

Lemma 8

Assume that a triplet $(U,V,W)$ of random variables satisfies $U=V$. Then we have

[TABLE]

Proof:

Since $U=V$, the joint distribution of $(U,V,W)$ is given as

[TABLE]

for each $(u,v,w)$. Then we have

[TABLE]

where the last equality comes from the fact that $\mu_{VW}(v,w)\chi(u=v)=0$ when $v\neq u$. ∎

Acknowledgments

The author thanks Dr. S. Miyake, Prof. K. Iwata, and Dr. Y. Sakai for helpful discussions.