Arithmetic Coding Based Multi-Composition Codes for Bit-Level Distribution Matching

TL;DR

This paper introduces multi-composition codes and an arithmetic coding scheme to enhance distribution matching efficiency, especially for short message lengths, outperforming constant-composition methods in rate and divergence.

Contribution

It proposes a novel multi-composition coding approach with an efficient arithmetic coding scheme, improving performance over traditional CCDM for short blocks.

Findings

MCDM encodes more data than CCDM.

Lower KL divergence achieved with MCDM.

Enhanced performance for short block messages.

Abstract

A distribution matcher (DM) encodes a binary input data sequence into a sequence of symbols (codeword) with desired target probability distribution. The set of the output codewords constitutes a codebook (or code) of a DM. Constant-composition DM (CCDM) uses arithmetic coding to efficiently encode data into codewords from a constant-composition (CC) codebook. The CC constraint limits the size of the codebook, and hence the coding rate of the CCDM. The performance of CCDM degrades with decreasing output length. To improve the performance for short transmission blocks we present a class of multi-composition (MC) codes and an efficient arithmetic coding scheme for encoding and decoding. The resulting multi-composition DM (MCDM) is able to encode more data into distribution matched codewords than the CCDM and achieves lower KL divergence, especially for short block messages.

Equations

$$f_{\text{DM}}:\{0,1{\}}^k\to \mathcal{C},$$

$$\frac{1}{n}\mathbb{D}(P_{\tilde{\mathbf{A}}}\Vert P_A^n)=\frac{1}{n}\sum _{\mathbf{c}\in \mathcal{C}}\frac{1}{|\mathcal{C}|}\log \frac{\frac{1}{|\mathcal{C}|}}{P_A^n(\mathbf{c})}.$$

$$P_{\mathcal{C}}(a)=\frac{{\sum }_{\mathbf{c}\in \mathcal{C}}{n}_a(\mathbf{c})}{n|\mathcal{C}|},$$

$$\frac{1}{n}\mathbb{D}(P_{\tilde{\mathbf{A}}}\Vert P_A^n)=\mathbb{H}(P_{\mathcal{C}})-\frac{\log |\mathcal{C}|}{n}+\mathbb{D}(P_{\mathcal{C}}||P_A),$$

$$\mathbf{\gamma }(\mathbf{c}):=[n_{a_1}(\mathbf{c}),\dots ,n_{a_m}(\mathbf{c})].$$

$${\mathcal{C}}_{\text{CC}}=\{\mathbf{c}\in {\mathcal{A}}^n:\mathbf{\gamma }(\mathbf{c})=\mathbf{\gamma }\},$$

$$|{\mathcal{C}}_{\text{CC}}|=\left(\genfrac{}{}{0ex}{}{n}{{\gamma }_1,\dots ,{\gamma }_m}\right)=\frac{n!}{{\prod }_{i=1}^m{\gamma }_i!}.$$

$$k=\lfloor {\log }_2|{\mathcal{C}}_{\text{CC}}|\rfloor ,$$

$$\text{NBC}(\mathbf{u})=\sum _{j=1}^k{u}_j{2}^{k-j}.$$

$$d(\mathbf{u})=\frac{\text{NBC}(\mathbf{u})}{2^k}.$$

$$x(\varnothing )=0,y(\varnothing )=1,$$

$$x(\mathbf{s}{c}_i)=\{\begin{array}{l}x(\mathbf{s}),\text{ if }{c}_i=0\\ x(\mathbf{s})+y(\mathbf{s})P(0|\mathbf{s}),\text{ if }{c}_i=1\end{array}$$

$$y(\mathbf{s}{c}_i)=y(\mathbf{s})P(c_i|\mathbf{s}),$$

$$x(\mathbf{c})=\sum _{{\mathbf{c}}^{\prime }\in \tilde{\mathcal{C}}:\text{NBC}({\mathbf{c}}^{\prime })<\text{NBC}(\mathbf{c})}{P}_{\mathbf{C}}({\mathbf{c}}^{\prime }),$$

$$y(\mathbf{c})=\prod _{i=1}^{n}P(c_i|{\mathbf{c}}_1^{i-1})=P_{\mathbf{C}}(\mathbf{c}).$$

$$\frac{1}{2^k}\ge \underset{\mathbf{c}\in \tilde{\mathcal{C}}}{\max }|I(\mathbf{c})|.$$

$$P(0|\mathbf{s}):=\frac{N(\mathbf{s}0)}{N(\mathbf{s})},P(1|\mathbf{s}):=\frac{N(\mathbf{s}1)}{N(\mathbf{s})}.$$

$$P(\mathbf{c})$$

$$=\frac{N(c_1)}{N(\varnothing )}\frac{N({\mathbf{c}}_1^2)}{N(c_1)}\dots \frac{N({\mathbf{c}}_1^n)}{N({\mathbf{c}}_1^{n-1})}=\frac{N({\mathbf{c}}_1^n)}{N(\varnothing )}=\frac{1}{|\tilde{\mathcal{C}}|},$$

$$P(a|\mathbf{s})=\frac{N(\mathbf{s}a)}{N(\mathbf{s})}=\frac{|\{\mathbf{c}\in \tilde{\mathcal{C}}:{\mathbf{c}}_1^{l(\mathbf{s})}=\mathbf{s}\wedge c_{l(\mathbf{s})+1}=a\}|}{|\{\mathbf{c}\in \tilde{\mathcal{C}}:{\mathbf{c}}_1^{l(\mathbf{s})}=\mathbf{s}\}|},$$

$$\mathcal{C}=\{\mathbf{c}\in \tilde{\mathcal{C}}:\exists \mathbf{u}\in \{0,1{\}}^k\text{ s.t. }\mathbf{c}=f_{\text{DM}}(\mathbf{u})\},$$

$$N(\mathbf{s})=\left(\genfrac{}{}{0ex}{}{n-l(\mathbf{s})}{m-n_1(\mathbf{s})}\right).$$

$$\Gamma =\{{\mathbf{\gamma }}_1,\dots ,{\mathbf{\gamma }}_I\},$$

$$N(\mathbf{s})=\sum _{i=1}^I\left(\genfrac{}{}{0ex}{}{n-l(\mathbf{s})}{m_i-n_1(\mathbf{s})}\right),$$

$$\tilde{\mathcal{C}}=\{\mathbf{c}\in \{0,1{\}}^n:m_L\le n_1(\mathbf{c})\le m_U\}.$$

$$P(1|\mathbf{s})=\frac{m-n_1(\mathbf{s})}{n-l(\mathbf{s})}.$$

$$P(1|\mathbf{s})=\frac{m-n_1(\mathbf{s})}{n-l(\mathbf{s})+1}.$$

$$P(1|\mathbf{s})=\frac{{\sum }_{i=0}^m\left(\genfrac{}{}{0ex}{}{n-1-l(\mathbf{s})}{i-1-n_1(\mathbf{s})}\right)}{{\sum }_{i=0}^m\left(\genfrac{}{}{0ex}{}{n-l(\mathbf{s})}{i-n_1(\mathbf{s})}\right)},$$

$$m^{\ast }=\underset{m}{\mathrm{argmin}}\sum _{\mathbf{c}\in \tilde{\mathcal{C}}(m)}\frac{1}{|\tilde{\mathcal{C}}(m)|}\log \frac{\frac{1}{|\tilde{\mathcal{C}}(m)|}}{P_A^n(\mathbf{c})},$$

Full text

I Introduction

A distribution matcher (DM) reversibly maps a sequence ${\bm{U}}$ of independent and uniformly distributed bits into a sequence ${\bm{A}}$ of symbols to emulate a target distribution $P_{A}$. The output of the DM approximates a sequence of independent and identically distributed (IID) symbols, each distributed according to $P_{A}$. The accuracy of the approximation is measured by the Kullback–Leibler (KL) divergence between the probability distribution of the DM’s output and the probability distribution of the IID sequence. An inverse distribution matcher (DM*-1*) performs the inverse operation recovering ${\bm{U}}$ from ${\bm{A}}$. DMs can be used in communication systems, such as probabilistic amplitude shaping (PAS) [1], to adjust the distribution of transmitted symbols to a distribution beneficial for a certain channel, e.g., a distribution achieving capacity, or reducing the peak-to-average-power ratio. PAS was recently proposed for the 5G mobile system [2].

We focus here on block-to-block (b2b) DMs, where the input and output sequences have fixed lengths denoted by $k$, and $n$, respectively. The ratio $k/n$ is called the matching rate. Variable length DMs, i.e., non b2b DMs, lead to synchronization problems, error propagation, and variable transmission rate [3]. A DM can be seen as an encoder which maps $k$ input bits to a non-binary codeword of length $n$. The set of possible codewords, i.e., the codebook (or code), is chosen such that a certain output distribution is emulated. To achieve a good performance, i.e., high matching rate and low normalized KL divergence, relatively long output sequences are needed [4]. Long sequences imply a large codebook which may not be stored in memory. DMs which need to store the codebooks, e.g., look-up-table mappings, have thus limited performance.

A Constant-Composition DM (CCDM) [4] is a DM which employs arithmetic coding to generate codewords on-the-fly. In this way, a large codebook can be used without the need of storage. The CCDM is asymptotically optimal for $n\to\infty$, i.e., it achieves the maximal matching rate and vanishing normalized KL divergence. For finite $n$ however, the CCDM suffers from rate loss and high divergence that increases with decreasing $n$. The CCDM uses a constant-composition (CC) codebook, which constraints each codeword to have a fixed number of occurrences of each of the symbols from the alphabet, i.e., each codeword has the same composition. The CC constraint contributes to increased KL divergence and decreased matching rate by limiting the number of codewords. It is therefore of interest to improve the DM performance for smaller values of $n$ for at least two reasons: 1) low performance DM, e.g., with high rate loss, can waive the benefits of PAS; 2) throughput and parallelization—it allows to replace one DM with output length $n$ by $\alpha$ parallel DMs with output lengths $n/\alpha$ without significant decrease in the performance. This leads to increased throughput of PAS systems, as DM is the throughput limiting component in the PAS transmission chain.

In this work, we introduce a Multi-Composition (MC) codes which can be used to build a MC DM (MCDM). A MC codebook is a codebook which contains codewords of multiple compositions. By a proper construction of the MC codebook, the MCDM is able to employ an arithmetic coding algorithm to generate codewords from the MC codebook on-the-fly. The MCDM generalizes the CCDM, and the aforementioned asymptotic optimality of the CCDM holds for the MCDM. By relaxing the CC constraint, we obtain a DM which achieves higher matching rate and lower KL divergence for any $n$. Just recently, two other solutions have also been proposed in the literature, i.e., partition based DM [5] and shell mapping DM (SMDM) [6]. Roughly speaking, our MCDM is able to index more codewords than [5], thus leading to better performance. The proposed MCDM can also be implemented with low-complexity arithmetic coding requiring $\mathcal{O}(n)$ computational complexity [7], and can operate with $k$ and $n$ of arbitrary length. For the binary case, we present a MCDM whose performance can approach that of the optimal b2b DM.

The SMDM is optimal DM, which is suitable for short output lengths due to its complexity [6]. The CCDM is asymptotically optimal for long output lengths [4]. The proposed MCDM is based on arithmetic coding just as CCDM. This way, one algorithm (with different parameters) can be used for short and long sequences. This reduces the complexity of the system.

The structure of this work is organized as follows. In Sec. II we introduce distribution matching and the CCDM. In Sec. III we briefly describe how arithmetic coding can be applied for distribution matching. In Sec. IV we present the MCDM which can be efficiently implemented with the arithmetic coding scheme. Simulation results are given in Sec. V. Finally, conclusions are drawn in Sec. VI.

We use the following notations. We denote random variables (RVs) by capital letters, such as $A$, and realizations by small letters, such as $a$. A row vector is denoted by a bold symbol, e.g., ${\bm{a}}$. The $i$-th entry in the vector ${\bm{a}}$ is denoted by $a_{i}$, and a subvector $[a_{i},a_{i+1},\cdots,a_{j}]$ of ${\bm{a}}$ is denoted by ${\bm{a}}_{i}^{j}$. The length of a vector is denoted by $l({\bm{a}})$. E.g., we have ${\bm{a}}={\bm{a}}_{1}^{l({\bm{a}})}$. A RV uniformly distributed on a set $S$ is denoted by $\mathbb{U}_{S}$, i.e., $S\sim\mathbb{U}_{S}$ means that $P_{\mathbb{U}_{S}}(s)=1/|S|$ for $s\in S$.

II Distribution Matching

A one-to-one b2b DM is an injective function $f_{\text{DM}}$ from binary input sequences ${\bm{u}}\in\{0,1\}^{k}$ to codewords ${\bm{c}}$ from the codebook $\mathcal{C}\subseteq\mathcal{A}^{n}$, i.e.,

[TABLE]

where $\mathcal{A}$ is the output alphabet. We assume that the input sequence ${\bm{U}}$ is a random vector consisting of $k$ IID Bernoulli$(1/2)$ distributed bits. The output sequence of the DM is thus a random vector ${\bm{\tilde{A}}}=f_{\text{DM}}({\bm{U}})\sim\mathbb{U}_{\mathcal{C}}$ uniformly distributed on $\mathcal{C}$. The goal of the DM is to make its output "look" as if it was a sequence of IID RVs, each distributed according to the target probability distribution $P_{A}$. This is usually performed by minimizing the normalized KL divergence between the DM’s output ${\bm{\tilde{A}}}$ and the IID sequence ${\bm{A}}\sim P_{A}^{n}=\prod_{i=1}^{n}P_{A}$

[TABLE]

The divergence can thus be minimized by choosing a proper codebook $\mathcal{C}$. The empirical probability of a single symbol outputted by a DM is defined as

[TABLE]

where we use $n_{a}({\bm{c}})\vcentcolon=\left|\{i\colon c_{i}=a\}\right|$ to denote the number of occurrences of $a$ in the sequence ${\bm{c}}$ and $|\mathcal{C}|$ denotes the number of codewords in the codebook $\mathcal{C}$ (size of the codebook). In the literature, it is often believed that we need $P_{\mathcal{C}}=P_{A}$ (or $P_{\mathcal{C}}\approx P_{A}$) to minimize the divergence for finite $n$. However, this may not necessarily be true as we shall see in Sec. V or as pointed out by authors in [6, Example 2]. The divergence (2) can be equivalently written as

[TABLE]

where $\mathbb{H}(P_{\mathcal{C}})$ in the entropy of a RV with distribution $P_{\mathcal{C}}$. We observe that the necessary condition for vanishing normalized divergence is that $P_{\mathcal{C}}\to P_{A}$111Since $\mathbb{H}(P_{\mathcal{C}})-\frac{\log|\mathcal{C}|}{n}=\frac{1}{n}\mathbb{D}(P_{{\bm{\tilde{A}}}}||P_{\mathcal{C}}^{n})\geq 0$., however $\frac{1}{n}\mathbb{D}(P_{{\bm{\tilde{A}}}}\|P_{A}^{n})\to 0$ only for $n\to\infty$ [8]. Thus, we can only expect $P_{\mathcal{C}}\approx P_{A}$ for large $n$. Equation (4) also suggests that for a given $P_{\mathcal{C}}$, larger codebooks, i.e., with greater $|\mathcal{C}|$, are preferred.

In [9] it was shown that non-binary distribution matching, i.e., with $|\mathcal{A}|>2$, can be well approximated by multiple, binary distribution matchings, i.e., with $|\mathcal{A}|=2$, for typical use cases. For simplicity, we focus on the binary distribution matching in the following sections. The MCDM can be also directly used for non-binary distribution matching, as explained is Sec. IV-B.

CCDM was introduced in [4]. It uses a modified coding scheme [7] based on arithmetic coding to efficiently encode data into the codewords from a CC codebook. In the CC codebook each codeword has the same composition.

Definition 1**.**

Assume $\mathcal{A}\!=\!\{a_{1},\dotsc,a_{m}\}$. A composition of a vector ${\bm{c}}\in\mathcal{A}^{n}$ is a vector containing the numbers of occurrences in ${\bm{c}}$ of each of the symbols from the alphabet $\mathcal{A}$. We denote a composition by

[TABLE]

Example 1**.**

$\mathcal{A}\!=\!\{0,1\},{\bm{\gamma}}(1011)=[1,3]$.

That is, the CC codebook with the composition ${\bm{\gamma}}$ is

[TABLE]

and the size of the codebook can be expressed by the multinomial coefficient

[TABLE]

To guarantee a one-to-one mapping between the binary input sequences and the codewords, the CCDM can use at most $2^{k}$ input sequences of length $k$, where $k$ reads as

[TABLE]

where $\lfloor\cdot\rfloor$ is the floor function.

III Arithmetic Coding in Distribution Matching

The coding scheme [7] based on arithmetic coding has been proposed to efficiently realize encoding and decoding for the so-called $m$-out-of-$n$ codebooks, i.e., the binary constant-weight codebooks, which are a spacial case of the CC codebooks for binary alphabets. In [4] it is shown that the arithmetic coding scheme from [7] can be utilized for the CC codebooks with non-binary alphabets, i.e., for CCDM implementation. In what follows, we demonstrate that the arithmetic coding scheme presented in [7] can be also employed to efficiently implement our MCDM.

For simplicity, we consider the binary output alphabet $\mathcal{A}=\{0,1\}$. Assume an arbitrary codebook $\tilde{\mathcal{C}}\subseteq\mathcal{A}^{n}$. Each input data sequence ${\bm{u}}_{i},i\!=\!1,\dots,2^{k}$ corresponds to a distinct point $d({\bm{u}}_{i}),i\!=\!1,\dots,2^{k}$ from the interval $[0,1)$. On the other hand, each codeword ${\bm{c}}\in\tilde{\mathcal{C}}$ corresponds to a distinct subinterval $I({\bm{c}})$ of the interval $[0,1)$. The subintervals $I({\bm{c}}),{\bm{c}}\in\tilde{\mathcal{C}}$ are chosen such that they partition the interval $[0,1)$, i.e., they are pairwise disjoint and $\bigcup_{{\bm{c}}\in\tilde{\mathcal{C}}}I({\bm{c}})=[0,1)$. At the encoder, an input data sequence ${\bm{u}}$ is mapped to a codeword ${\bm{c}}$ if the corresponding point $d({\bm{u}})$ lines inside the corresponding interval $I({\bm{c}})$. At the decoder, first an interval $I({\bm{c}})$ is determined based on the received codeword ${\bm{c}}$. Then, a point $d({\bm{u}})\in I({\bm{c}})$ is determined and decoded to the sequence ${\bm{u}}$.

Assume a binary input sequence ${\bm{u}}=[u_{1},\dotsc,u_{k}]$. Let $\text{NBC}(\cdot)$ denote a function which returns the natural binary code (NBC) number corresponding to the sequence ${\bm{u}}$, i.e.,

[TABLE]

The sequence ${\bm{u}}$ is mapped to a point $d({\bm{u}})\in[0,1)$ via

[TABLE]

An interval $I({\bm{c}})$ for a codeword ${\bm{c}}=[c_{1},\dotsc,c_{n}]$ can be computed recursively using a chosen probability model $P_{{\bm{C}}}$ on codeword’s bits. The model $P_{{\bm{C}}}$ is specified in terms of the conditional probabilities (also called branching probabilities) of the next bit given the previous bits, i.e., $P_{C_{i}|{\bm{C}}_{1}^{i-1}}(\cdot|{\bm{s}})$, where ${\bm{s}}$ is a sequence denoting a prefix of the codeword. The beginning $x({\bm{c}})$ and the width $y({\bm{c}})$ of the interval $I({\bm{c}})=[x({\bm{c}}),x({\bm{c}})+y({\bm{c}}))$ can be computed by applying iteratively equations (12) and (13) for $i=1,\dots,n,$

[TABLE]

where ${\bm{s}}c_{i}$ denotes a concatenation of ${\bm{s}}$ and $c_{i}$, $\emptyset$ denotes an empty sequence, and $P(0|{\bm{s}})$ stands for $P_{C_{i}|{\bm{C}}_{1}^{i-1}}(0|{\bm{s}})$. Equation (12) implies a lexicographical ordering of the codewords according to $0<1$ with the most-significant-bit $c_{1}$. That is, for two codewords ${\bm{c}}_{1}$ and ${\bm{c}}_{2}$, if $\text{NBC}({\bm{c}}_{1})<\text{NBC}({\bm{c}}_{2})$, the $I({\bm{c}}_{1})$ will be placed in the interval $[0,1)$ below the $I({\bm{c}}_{2})$. Applying the above equations result in partitioning such that $\forall{\bm{c}}\in\tilde{\mathcal{C}}$

[TABLE]

That is, the codewords’ intervals partition $[0,1)$ and are ordered according to lexicographical ordering $0<1$. E.g., see Fig. 1.

A one-to-one mapping between data sequences and codewords can be established if each interval $I({\bm{c}}),{\bm{c}}\in\tilde{\mathcal{C}}$ contains at most one point $d({\bm{u}})$. This can be guaranteed by letting the distance between two adjacent points to be grater than the largest interval, i.e.,

[TABLE]

Since we are interested in maximizing $k$, it is reasonable to choose equal length intervals, i.e., $|I({\bm{c}})|=\frac{1}{|\tilde{\mathcal{C}}|}\forall{\bm{c}}\in\tilde{\mathcal{C}}$. In this case the greatest $k$ fulfilling (16) equals $\lfloor{\log_{2}|\tilde{\mathcal{C}}|}\rfloor$.

From (15) we have that the length of the interval $I({\bm{c}})$ is equal to the probability of the codeword ${\bm{c}}$ (by using the probability model $P_{{\bm{C}}}$ on codeword’s bits). We are interested in finding the conditional probabilities $P(0|{\bm{s}})$ and $P(1|{\bm{s}})$, where ${\bm{s}}$ is a binary sequence constituting the prefix of the codeword, such that the probability of each codeword ${\bm{c}}\in\tilde{\mathcal{C}}$ is equal. Let $N({\bm{s}})$ denote the number of codewords in $\tilde{\mathcal{C}}$ that have prefix ${\bm{s}}$, i.e., $N({\bm{s}})\vcentcolon=\left|\{{\bm{c}}\in\tilde{\mathcal{C}}\colon{\bm{c}}_{1}^{l({\bm{s}})}={\bm{s}}\}\right|$. We define the following conditional probabilities

[TABLE]

For any ${\bm{c}}=[c_{1},\dots,c_{n}]\in\tilde{\mathcal{C}}$ we have

[TABLE]

which shows that by employing model as in (17) we can obtain intervals of equal length.

To encode data into the codewords from $\tilde{\mathcal{C}}$ we apply arithmetic decompression of the input sequence ${\bm{u}}$ using the model (17), see Algorithm 1 for details. To decode the codeword, we apply arithmetic compression on the codeword ${\bm{c}}$ using the same model (17), see Algorithm 2 for details. Retrieving ${\bm{u}}$ from $I({\bm{c}})$ is the final step of Algorithm 2, and is performed in the line $11$ which follows from (10) and the fact that $d({\bm{u}})\in I({\bm{c}})$. In practice, to avoid numerical underflow, the intervals have to be rescaled during each step. The coding scheme can be also implemented using only integer calculations. For further implementation considerations, see e.g. [7].

Following the above steps we can efficiently encode/decode data into/from the codewords ${\bm{c}}\in\tilde{\mathcal{C}}$. In general, not all codewords from $\tilde{\mathcal{C}}$ will be used since, by (16), we can use at most $2^{\log_{2}\lfloor|\tilde{\mathcal{C}}|\rfloor}$ codewords. The selection of the used codewords is done implicitly by the encoding/decoding algorithm. We introduce the notion of the base codebook, which contains all codewords, i.e., the selected and non-selected ones.

Definition 2**.**

A base codebook, denoted by $\tilde{\mathcal{C}}$, for the coding scheme from Sec. III is a codebook which is used to compute the branching probabilities (17), i.e.,

[TABLE]

for any $a\in\mathcal{A}$ and any prefix ${\bm{s}}$.

The actual codebook $\mathcal{C}$, is the codebook actually used by the encoder/decoder, i.e.,

[TABLE]

where $f_{\text{DM}}$ is the encoder function. The actual codebook is a subset of the base codebook implicitly chosen by the encoding/decoding algorithm.

IV Multi-Composition Codebooks

Assume an arbitrary base codebook $\tilde{\mathcal{C}}\!=\!\{{\bm{c}}_{i}\!\in\!\{0,1\}^{n},i=1,\dotsc,N\}$. Using the coding scheme from Sec. III, we are able to encode/decode data into/from codewords from $\tilde{\mathcal{C}}$. This involves computing the probabilities (17). In a general case, finding (17) entails evaluating $N({\bm{s}})$ by counting the codewords from $\tilde{\mathcal{C}}$ which have the prefix ${\bm{s}}$. This is not feasible for large codebooks which we target. Therefore, we introduce a structure into the base codebook $\tilde{\mathcal{C}}$.

We observe that $N({\bm{s}})$ is easy to compute for base codebooks containing all codewords of a single composition. Assume a binary composition ${\bm{\gamma}}_{0}=[n-m,m]$ and the codebook $\tilde{\mathcal{C}}\!=\!\{{\bm{c}}\!\in\!\{0,1\}^{n}\colon\gamma({\bm{c}})={\bm{\gamma}}_{0}\}$, i.e., the so-called $m$-out-of-$n$ codebook containing all codewords of Hamming weight $m$. For any prefix ${\bm{s}}$, we have

[TABLE]

Consequently, $N({\bm{s}})$ is simple to compute for base codebooks containing all codewords of multiple compositions. Assume a set of different binary compositions

[TABLE]

where ${\bm{\gamma}}_{i}=[n-m_{i},m_{i}]$ for $i=1,\dotsc,I$, and the base codebook $\tilde{\mathcal{C}}\!=\!\{{\bm{c}}\!\in\!\{0,1\}^{n}\colon{\bm{\gamma}}({\bm{c}})\in\Gamma\}$. Here, we will refer to such a codebook as the MC base codebook. For any prefix ${\bm{s}}$, we have

[TABLE]

which is easy to evaluate. Note that $N({\bm{s}})$ depends only on two parameters $l({\bm{s}})$, and $n_{1}({\bm{s}})$, therefore it is also possible to store the precomputed values in a look-up-table (LUT). For implementation we need only to store the values of $P(0|{\bm{s}})$ (or $P(1|{\bm{s}})$). As $P(0|{\bm{s}})$ (or $P(1|{\bm{s}})$) depends only on $l({\bm{s}})$ and $n_{1}({\bm{s}})$, we need to store at most $n^{2}$ values. For an arbitrary base codebook, $N({\bm{s}})$ depends on the whole sequence ${\bm{s}}$ and it is not feasible to store all values in a LUT. Here, we will refer to a DM using the base MC codebook and the coding scheme from Sec. III as the Multi-Composition Distribution Matcher (MCDM).

IV-A Some Special Cases

By selecting the set of compositions (23) we can obtain a specific base codebook.

Definition 3**.**

A $[m_{L},m_{U}]$-out-of-$n$ codebook is a codebook with codewords of Hamming weight at least $m_{L}$ and at most $m_{U}$, i.e.,

[TABLE]

Based on Definition 3, we have the following special cases.

IV-A1 $m$-out-of-$n$ codebook

By selecting $\Gamma=\left\{[n-m,m]\right\}$ we obtain a CC codebook as in the CCDM. For CC codebook the probability in (17) has a particularly simple form, i.e.,

[TABLE]

IV-A2 $[m\!-\!1,m]$-out-of-$n$ codebook

By selecting $\Gamma=\left\{[n-m+1,m-1],[n-m,m]\right\}$ we obtain a codebook which contains two adjacent compositions. We refer to such a MCDM as 2C-MCDM. The probability in (17) also admits a simple form

[TABLE]

As such, the CCDM can be changed into the 2C-MCDM by just changing the denominator in the applied model for arithmetic coding.

IV-A3 $[0,m]$-out-of-$n$ codebook

By selecting $\Gamma=\left\{[n,0],\right.$ $\left.[n-1,1],\dots,[n-m,m]\right\}$ we obtain a codebook which contains all sequences up to Hamming weight $m$. This is the optimal codebook for an ideal DM which can encode into a codebook of arbitrary size.

Lemma 1**.**

Assume the output alphabet $\mathcal{A}=\{0,1\}$, and the target probability such that $P_{A}(0)>P_{A}(1)$. Assume a DM with output $\mathbb{U}_{\mathcal{C}}$. The actual codebook $\mathcal{C}$ which minimizes the normalized KL divergence (2)

1. (a)

consists of $2^{k}$ most likely codewords according to $P_{A}$, if we require $|\mathcal{C}|=2^{k}$ for some $k$ **[3]**. This DM can be implemented by the SMDM **[6]**. 2. (b)

is a $[0,m]$-out-of-$n$* codebook for some $m$, if $|\mathcal{C}|$ is not constrained [8].*

Proof.

See [3, Sec. IV], [8, Lemma 5]. ∎

The MCDM with the $[0,m]$-out-of-$n$ codebook, which we will refer to as Opt-MCDM, can be seen as an approximation of the optimal DM from Lemma 1b. Note that when the $[0,m]$-out-of-$n$ codebook size is equal to $2^{k}$, the Opt-MCDM is the optimal DM (as in Lemma 1a). In practice, the Opt-MCDM offers close to optimal performance. The probability in (17) is

[TABLE]

which depends only on $l({\bm{s}})$ and $n_{1}({\bm{s}})$, and it can therefore be precomputed and stored in a LUT for efficient implementation.

IV-B Non-binary Case

The coding scheme described in Sec. III can be adapted for non-binary distribution matching. The probability model $P_{{\bm{C}}}$ on codewords’ symbols can be obtained from the base codebook via equations analogous to (17). An MC codebook contains all codewords from multiple compositions, and the expression for $N({\bm{s}})$ becomes a sum of multinomials, which is admissible to evaluate or store for shorter codewords or fewer compositions in the base codebook. However, for the base codebooks with large number of compositions or long codewords, the storage/computation requirements can become prohibitive for large alphabet. Unless some structure is added when choosing the compositions, the MCDM is better suited for binary distribution matching, e.g., it can be used for non-binary distribution matching in combination with the bit-level distribution matcher [9].

V Results

V-A Distribution Matching Performance

We compare the CCDM with the $m$-out-of-$n$ codebook, MCDM with the $[m-1,m]$-out-of-$n$ codebook (2C-MCDM), and MCDM with the $[0,m]$-out-of-$n$ codebook (Opt-MCDM). We use the binary output alphabet $\mathcal{A}$ and the target probability distribution $P_{A}$ with $P_{A}(1)=0.422$. We vary the output length $n$. For each of the DMs we find the base codebook $\tilde{\mathcal{C}}$ which minimizes the KL divergence $\frac{1}{n}\mathbb{D}(\mathbb{U}_{\tilde{\mathcal{C}}}||P_{A}^{n})$. This is equivalent to finding

[TABLE]

where $\tilde{\mathcal{C}}(m)$ is the $m$-out-of-$n$, $[m-1,m]$-out-of-$n$, and $[0,m]$-out-of-$n$ codebook for the CCDM, 2C-MCDM, and Opt-MCDM, respectively. Next, we apply the coding scheme as in Sec. III to build DMs using the aforementioned optimized base codebooks. The results are presented in Fig. 2. The divergence and the empirical output distribution $P_{\mathcal{C}}$ were computed by enumerating all codewords for $n\leq 30$ and for higher $n$ via Monte-Carlo sampling. The number of samples was chosen so that the relative error of estimates lies within $3\%$ with probability not smaller than $90\%$. The matching rate is evaluated exactly. We also compute the parameters of the optimal DM from Lemma 1a implemented by SMDM [6].

Fig. 2 shows the superior performance of the MC codebooks in terms of the matching rate and KL divergence. Opt-MCDM stays very close to the optimal DM from Lemma 1a. In Fig. 2(a) the optimal DM achieves higher rate than the entropy of the target distribution $\mathbb{H}(P_{A})$ for $n\leq 30$. This is because for $n\leq 30$, the optimal base codebook is the $[0,n]$-out-of-$n$ codebook which contains all codewords of length $n$. This demonstrates that the lowest KL divergence can be achieved by performing no distribution matching at all, and hence the optimal codebook has $P_{\mathcal{C}}\neq P_{A}$. For large $n$, $P_{\mathcal{C}}(1)\to P_{A}(1)$ for all DMs, as this is a necessary condition for vanishing divergence for $n\to\infty$, as observed in Sec. II. In Fig. 2(c) for $n=110$, $P_{\mathcal{C}}$ coincide for all DMs. However, in Fig. 2(b) the Opt-MCDM achieves the lowest KL divergence thanks to the largest codebook, confirming the observations from Sec. II.

V-B PAS Framework

In practice, applying the Opt-MCDM instead of the CCDM for $n=110$ in a PAS communication system would mean ca. $3\%$ increase in the transmission rate (see Fig. 2(a)). Alternatively, assume we target the KL divergence $10^{-2}$. Instead of using one CCDM with $n=500$, we can use $5$ parallel Opt-MCDMs with $n=100$ to increase the throughput, as shown in Fig. 2(b).

Motivated by this considerations we apply the Opt-MCDM to a PAS system in a bit-level setup as in [9]. We follow the steps exactly as in [9] but instead of using the CCDM as a building block, we employ the Opt-MCDM for each bit-level. For each bit-level’s target probability we find the optimal codebook as in Sec. V-A. We compare the results with [9] employing the CCDMs, and the bit-interleaved coded-modulation (BICM) scheme without shaping [10]. For fair comparison, all schemes use the same WiMAX LDPC B-code of rate $5/6$ and codeword length of $576$ bits. LDPC decoder performs $50$ iterations. BICM operates with $3$ constellations: $4$-QAM, $16$-QAM, and $64$-QAM, which corresponds to transmission rates of $1.67$, $3.33$, $5.0$ bits per channel use (b/CU), respectively. Shaping schemes use the $256$-QAM constellation and match the BICM transmission rates by applying appropriate transmit signal distributions. Frame error rate (FER) versus signal-to-noise ratio (SNR) curves are presented in Fig. 3. By employing the MCDM instead of the CCDM, we gain $0.02$, $0.11$, and $0.23$ dB, at FER=$10^{-3}$ for rates $1.67$, $3.33$, $5.0$ b/CU, respectively.

VI Conclusions

In this work, we presented arithmetic coding based distribution matcher which uses multi-composition codes. The multi-composition distribution matcher generalizes the state of the art constant-composition distribution matcher, and is able to achieve higher matching rate and lower KL divergence.

VII Acknowledgments

Part of this work has been performed in the framework of the Horizon 2020 project ONE5G (ICT-760809) receiving funds from the European Union. The authors would like to acknowledge the contributions of their colleagues in the project, although the views expressed in this contribution are those of the authors and do not necessarily represent the project.

The authors would like to thank Onurcan Iscan, Ronald Böhnke, Najeeb Ul Hassan from Huawei Technologies, for discussions and helpful comments for improving the manuscript.