Optimal Compression for Two-Field Entries in Fixed-Width Memories

TL;DR

This paper addresses the challenge of optimally compressing two-field entries in fixed-width memory systems, providing efficient algorithms and demonstrating superior success probabilities over existing methods.

Contribution

It introduces an optimal code design algorithm for two-field fixed-width memory compression, including a constrained single-code variant, with proven decompositions and empirical validation.

Findings

Optimal codes significantly improve success probabilities.

Efficient dynamic programming algorithms are developed.

Empirical results show superiority over existing algorithms.

Abstract

Data compression is a well-studied (and well-solved) problem in the setup of long coding blocks. But important emerging applications need to compress data to memory words of small fixed widths. This new setup is the subject of this paper. In the problem we consider we have two sources with known discrete distributions, and we wish to find codes that maximize the success probability that the two source outputs are represented in $L$ bits or less. A good practical use for this problem is a table with two-field entries that is stored in a memory of a fixed width $L$. Such tables of very large sizes drive the core functionalities of network switches and routers. After defining the problem formally, we solve it optimally with an efficient code-design algorithm. We also solve the problem in the more constrained case where a single code is used in both fields (to save space for storing code dictionaries). For both code-design problems we find decompositions that yield efficient dynamic-programming algorithms. With an empirical study we show the success probabilities of the optimal codes for different distributions and memory widths. In particular, the study demonstrates the superiority of the new codes over existing compression algorithms.

Tables

Table 1. (A) First field with code σ 1 subscript 𝜎 1 \sigma_{1}

Element	Prob.	codeword
$a$	0.4	00
$b$	0.3	01
$c$	0.16	10
$d$	0.08	110
$e$	0.06	111

Table 2. (A) First field with code σ 1 subscript 𝜎 1 \sigma_{1}

Element	Prob.	codeword
$a$	0.4	00
$b$	0.3	01
$c$	0.16	10
$d$	0.08	110
$e$	0.06	111

Table 3. (B) Second field with code σ 2 subscript 𝜎 2 \sigma_{2}

Element	Prob.	codeword
$x$	0.5	0
$y$	0.3	10
$z$	0.2	11

Table 4. (C) Possible entries encoded by the encoding function Σ = ( σ 1 , σ 2 ) Σ subscript 𝜎 1 subscript 𝜎 2 \Sigma=(\sigma_{1},\sigma_{2})

Entry	Prob.	Encoding	Width	Width $\le \left(L=4\right)$
($a,x$)	0.20	00 0	3	$\surd$
($a,y$)	0.12	00 10	4	$\surd$
($a,z$)	0.08	00 11	4	$\surd$
($b,x$)	0.15	01 0	3	$\surd$
($b,y$)	0.09	01 10	4	$\surd$
($b,z$)	0.06	01 11	4	$\surd$
($c,x$)	0.08	10 0	3	$\surd$
($c,y$)	0.048	10 10	4	$\surd$
($c,z$)	0.032	10 11	4	$\surd$
($d,x$)	0.04	110 0	4	$\surd$
($d,y$)	0.024	110 10	5	-
($d,z$)	0.016	110 11	5	-
($e,x$)	0.03	111 0	4	$\surd$
($e,y$)	0.018	111 10	5	-
($e,z$)	0.012	111 11	5	-

Table 5. TABLE II: The values of F ​ ( k , N ) 𝐹 𝑘 𝑁 F(k,N) (top of each pair in the table) and Q ​ ( k , N ) 𝑄 𝑘 𝑁 Q(k,N) (bottom) for k ∈ [ 1 , n 1 = 5 ] 𝑘 delimited-[] 1 subscript 𝑛 1 5 k\in[1,n_{1}=5] , N ∈ [ 0 , 2 L = 16 ] 𝑁 delimited-[] 0 superscript 2 𝐿 16 N\in[0,2^{L}=16] for Example  3 with L = 4 , n 1 = 5 , n 2 = 3 formulae-sequence 𝐿 4 formulae-sequence subscript 𝑛 1 5 subscript 𝑛 2 3 L=4,n_{1}=5,n_{2}=3 . The optimal value of P s ​ u ​ c ​ c ​ e ​ s ​ s subscript 𝑃 𝑠 𝑢 𝑐 𝑐 𝑒 𝑠 𝑠 P_{success} is given by F ​ ( 5 , 16 ) 𝐹 5 16 F(5,16) and the codeword lengths for σ 1 subscript 𝜎 1 \sigma_{1} by Q ​ ( 5 , 16 ) 𝑄 5 16 Q(5,16) .

	$k=1$	$k=2$	$k=3$	$k=4$	$k=5$
$N=0$	0	0	0	0	0
	(-)	(-,-)	(-,-,-)	(-,-,-,-)	(-,-,-,-,-)
$N=1$	0.2	0.2	0.2	0.2	0.2
	(4)	(4,-)	(4,-,-)	(4,-,-,-)	(4,-,-,-,-)
$N=2$	0.32	0.35	0.35	0.35	0.35
	(3)	(4,4)	(4,4,-)	(4,4,-,-)	(4,4,-,-,-)
$N=3$		0.47	0.47	0.47	0.47
		(3,4)	(3,4,-)	(3,4,-,-)	(3,4,-,-,-)
$N=4$	0.4	0.56	0.56	0.56	0.56
	(2)	(3,3)	(3,3,-)	(3,3,-,-)	(3,3,-,-,-)
$N=5$			0.64	0.64	0.64
			(3,3,4)	(3,3,4,-)	(3,3,4,-,-)
$N=6$		0.64	0.688	0.688	0.688
		(2,3)	(3,3,3)	(3,3,3,-)	(3,3,3,-,-)
$N=7$			0.72	0.728	0.728
			(2,3,4)	(3,3,3,4)	(3,3,3,4,-)
$N=8$		0.7	0.768	0.768	0.768
		(2,2)	(2,3,3)	(2,3,3,-)	(2,3,3,-,-)
$N=9$			0.78	0.808	0.808
			(2,2,4)	(2,3,3,4)	(2,3,3,4,-)
$N=10$			0.828	0.832	0.838
			(2,2,3)	(2,3,3,3)	(2,3,3,4,4)
$N=11$				0.868	0.868
				(2,2,3,4)	(2,2,3,4,-)
$N=12$			0.86	0.892	0.898
			(2,2,2)	(2,2,3,3)	(2,2,3,4,4)
$N=13$				0.9	0.922
				(2,2,2,4)	(2,2,3,3,4)
$N=14$				0.924	0.94
				(2,2,2,3)	(2,2,3,3,3)
$N=15$					0.954
					(2,2,2,3,4)
$N=16$				0.94	0.972
				(2,2,2,2)	(2,2,2,3,3)

Equations

$$P_{success}=0.20+0.12+0.08+0.15+0.09+0.06+0.08+0.048+0.032+0.04+0.03=\mathrm{0.93.}$$

$$P_{success}(D,\Sigma )=\mathrm{Pr}(\Sigma (a_1,a_2)\ne \perp ),$$

$$F(k,N)=$$

$$\underset{{\sigma }_1}{\max }{P}_{success}(D,\Sigma =({\sigma }_1,{\sigma }_2={\sigma }_2))=F(k=n_1,N=2^L).$$

$$F(k,N)=$$

$$P_{success}\le \sum _{i=1}^{n^{\prime }}{p}_{1,i}\sum _{j=1}^{2^{L-\ell ({\sigma }_1(s_{1,i}))}}{p}_{2,j},$$

$$P_{success}=\sum _{i=1}^{n^{\prime }}{p}_{1,i}\sum _{j=1}^{2^{L-\ell ({\sigma }_1(s_{1,i}))}}{p}_{2,j},$$

$$\displaystyle F(k,N)=\max\Bigg{(}\max_{\ell_{0}\in[1,L]}\,\bigg{(}F(k-1,N-N_{\ell_{0}})+p_{1,k}\cdot\sum_{i=1}^{min(n_{2},2^{L-\ell_{0}})}p_{2,i}\bigg{)},F(k-1,N)\Bigg{)}.$$

$$F(k,N)=$$

$$(L/2,L/2+1,L/2-1,L/2+2,L/2-2,\dots ,L-1,1)$$

$$G(l,[k_1,k_2],N)=$$

$$G(l^{\prime },[k_1,k_2],N)=\underset{j\in [0,k_2-k_1+1]}{\max }G(l,[k_1,k_2-j],N-j\cdot N_{l^{\prime }}),$$

$$G(l,[k_1,k_1-1],N)\triangleq 0,\mathrm{for}N\ge 0.$$

$$G(l^{\prime },[k_1,k_2],N)$$

$$\underset{j\in [0,k_2-k_1+1]}{\max }$$

$$A_{i,\ell }^x=\sum _{r=\ell }^{n-1}{A}_{i-1,r}^{x-2^{-r}}.$$

$$Z_n=A_{i=n,\ell =1}^{x=1}=\sum _{r=1}^{n-1}{A}_{n-1,r}^{1-2^{-r}}.$$

$$Z_n\ge 2\cdot 2^{\lfloor \frac{n-4}{3}\rfloor }\ge 2\cdot 2^{\frac{n-6}{3}}=0.5\cdot 2^{\frac{n}{3}}=0.5\cdot (\sqrt[3]{2}{)}^n.$$

Full text

††footnotetext: This paper solves a problem first formulated in the paper “Compression for Fixed-Width Memories” presented at the IEEE International Symposium on Information Theory (ISIT) 2013, Istanbul, Turkey [1]. O. Rottenstreich is with Princeton University, Princeton NJ, USA (e-mail: orir@princeton.edu). Y. Cassuto is with the Viterbi Department of Electrical Engineering, Technion – Israel Institute of Technology, Haifa Israel (e-mail: ycassuto@ee.technion.ac.il). This work was supported in part by the Israel Science Foundation (ISF), and by the Intel Center for Computing Intelligence (ICRI-CI).

I Introduction

In the best-known data-compression problem, a discrete distribution on source elements is given, and one wishes to find a representation for the source elements with minimum expected length. This problem was solved by the well-known Huffman coding [2]. Huffman coding reduces the problem to a very efficient recursive algorithm on a tree, which solves it optimally. Indeed, Huffman coding has found use in numerous communication and storage applications. Minimizing the expected length of the coded sequence emitted by the source translates to optimally low transmission or storage costs in those systems. However, there is an important setup in which minimizing the expected length does not translate to optimal improvement in system performance. This setup is fixed-width memory. Information is stored in a fixed-width memory in words of $L$ bits each. A word is designated to store an entry of $d$ fields, each emitted by a source. In this case the prime objective is to fit the representations of the $d$ sources into $L$ bits, and not to minimize the expected length for each source.

The fixed-width memory setup is extremely useful in real applications. The most immediate use of it is in networking applications, wherein switches and routers rely on fast access to entries of very large tables [3]. The fast access requires simple word addressing, and the large table size requires good compression. In addition to this principal application, data-centric computing applications can similarly benefit from efficient compression of large data sets in fixed-width memories. In this paper we consider the compression of data entries with $d=2$ fields into memory words of $L$ bits, where $L$ is a parameter. This special case of two fields is motivated by switch and router tables in which two-field tables are most common. Throughout the paper we assume that the element chosen for an entry field is drawn i.i.d. from a known distribution for that field, and that elements of distinct fields are independent of each other.

Our operational model for the compression assumes that success to fit an entry to $L$ bits translates to fast data access, while failure results in a much slower access because a secondary slower memory is accessed to fit the entry. Therefore, we want to maximize the number of entries we succeed to fit, but do allow some (small) fraction of failures. Correspondingly, the performance measure we consider throughout the paper is $P_{success}$: the probability that the encoding of the two-field entry fits in $L$ bits. That is, we seek a uniquely decodable encoding function that on some entries is allowed to declare encoding failure, but does so with the lowest possible probability for the source distributions. We emphasize that we do not allow decoding errors, and all entries that succeed encoding are recovered perfectly at read time.

Maximizing $P_{success}$ is trivial when we encode the two fields jointly by a single code: we simply take the $2^{L}$ highest probabilities in the product distribution of the two fields, and represent each of these pairs with a codeword of length $L$. However, this solution has prohibitive cost because it requires a dictionary with size quadratic in the support of the individual distributions. For example, if each field takes one of $1000$ values, we will need a dictionary of order $10^{6}$ in the trivial solution. To avoid the space blow-up, we restrict the code to use dictionaries of sizes linear in the distribution supports. The way to guarantee linear dictionary sizes is by composing the entry code from two codes for the individual fields. Our task is to find two codes for the two fields, such that when the two codewords are placed together in the width-$L$ word, the entry fields can be decoded unambiguously. The design objective for the codes is to maximize $P_{success}$: the probability that the above encoding of the entry has length of at most $L$ bits. We first show by example that Huffman coding is not optimal for this design objective. Consider a data entry with two fields, such that the value of the first field is one of $5$ possible source elements $\{a,b,c,d,e\}$, and the value of the second field is one of $3$ elements $\{x,y,z\}$. The distributions on these elements are given in the two left columns of Table I.(A)-(B); the values of the two fields are drawn independently. We encode the two fields using two codes $\sigma_{1},\sigma_{2}$ specified in the right columns of Table I.(A)-(B), respectively. The codewords of $\sigma_{1}$, $\sigma_{2}$ are concatenated, and this encoding needs to be stored in the memory word. Table I.(C) enumerates the $5\cdot 3=15$ possible entries (combinations of values for the two fields), and for each gives its probability, its encoding, and the encoding length. The rightmost column of Table I.(C) indicates whether the encoding succeeds to fit in $L=4$ bits ($\surd$), or not ($-$). The success probability of the pair of codes $(\sigma_{1},\sigma_{2})$ is the sum of all probabilities in rows marked with $\surd$. This amounts to

[TABLE]

It can be checked that the codes $(\sigma_{1},\sigma_{2})$ give better $P_{success}$ than the success probability of the respective Huffman codes, which equals 0.78.

In the sequel we design code pairs $(\sigma_{1},\sigma_{2})$, which we call entry coding schemes. We refer to an entry coding scheme as optimal if it maximizes $P_{success}$ among all entry coding schemes in its class. In the paper we find optimal coding schemes for two classes with different restrictions on their implementation cost: one is given in Section III and one in Section IV.

Finding codes that give optimal $P_{success}$ cannot be done by known compression algorithms. Also, brute-force search for an optimal code has complexity that grows exponentially with the number of source elements. Even if we consider only monotone codes – those which assign codewords of lengths non-increasing with the element probabilities – we can show that the number of such codes grows asymptotically as at least $\eta^{n}$, where $\eta=\sqrt[3]{2}$. An exact count of monotone codes is provided in the Appendix. The infeasibility of code search motivates the study in this paper toward devising efficient polynomial-time algorithms for finding optimal fixed-width codes.

For the case of entries with $d=2$ fields, this paper solves the problem of optimal fixed-width compression. We present an efficient algorithm that takes any pair of element distributions and an integer $L$, and outputs an entry encoder with optimal $P_{success}$. The encoder has the unique-decodability property, and requires a dictionary whose size equals the sum of the element counts for the two fields. We also solve the problem for the more restrictive setup in which the same code needs to be used for both fields. This setup is motivated by systems with a more restricted memory budget, which cannot afford storing two different dictionaries for the two fields. For both setups finding optimal codes is formulated as efficient dynamic-programming algorithms building on clever decompositions of the problems to smaller problems.

Fig. 1 shows an illustration of a plausible realization of fixed-width compression in a networking application. First, (a) illustrates the write operation where the entry is encoded and is written to the fast SRAM memory if the encoded entry has a length of at most $L$ bits. Otherwise, it is stored in the slow DRAM memory. (b) shows the read operation where encoded data is first searched in the fast SRAM memory. If it is not found, the slow DRAM memory is also accessed after the entry index is translated.

Paper organization

In Section II, we present the formal model for the problem of compression for fixed-width memories. Setting up the problem requires some definitions extending classical single-source compression to $2$ sources modeling the $d=2$ fields in the entry. Another important novelty of Section II is the notion of an encoder that is allowed to fail on some of the input pairs. In Section III we derive an algorithm for finding an optimal coding scheme for fixed-width compression with $d=2$ fields. There are two main ingredients to that algorithm: 1) an algorithm finding an optimal prefix code for the first field given the code for the second field, and 2) a code for the second field that is optimal for any code used in the first field. In the second ingredient it turns out that the code needs not be a prefix code, but it does need to satisfy a property which we call padding invariance. Next, in Section IV we study the special case where the two fields of an entry need to use the same code. The restriction to use one code renders the algorithm of Section III useless for this case. Hence we develop a completely different decomposition of the problem that can efficiently optimize a single code jointly for both fields. In Section V we present empiric results showing the success probabilities of the optimal codes for different values of the memory width $L$. The results show some significant advantage of the new codes over existing Huffman codes. In addition, they compare the performance of the schemes using two codes (Section III) to those that use a single code (Section IV). Finally, Section VI summarizes our results and discusses directions for future work.

Relevant prior work

There have been several extensions of the problem solved by Huffman coding that received attention in the literature [4, 5, 6, 7, 8]. These are not directly related to fixed-width memories, but they use clever algorithmic techniques that may be found useful for this problem too. The notion of fixed-width representations is captured by the work of Tunstall on variable-to-fixed coding [9], but without the feature of multi-field entries. The problem of fixed-width encoding is also related to the problem of compression with low probability of buffer overflow [10, 11]. However, to the best of our knowledge the prior work only covers a single distribution and an asymptotic number of encoding instances (in our problem the number of encoding instances is a fixed $d$). The most directly related to the results of this paper is our prior work [12] considering a combinatorial version of this problem, where instead of source distributions we are given an instance with element pairs. This combinatorial version is less realistic for applications, because a new code needs to be designed after every change in the memory occupancy.

II Model and Problem Formulation

II-A Definitions

We start by providing some definitions, upon which we next cast the formal problem formulation. Throughout the paper we assume that data placed in the memory come from known distributions, as now defined.

Definition 1** (Element Distribution).**

An element distribution $(S,P)$ = $((s_{1},\dots,s_{n}),(p_{1},\dots,p_{n}))$ is characterized by an (ordered) set of elements with their corresponding positive appearance probabilities. An element of $S$ is drawn randomly and independently according to the distribution $P$, i.e., $\Pr\left(a=s_{i}\right)=p_{i}$, with $p_{i}>0$ and $\Sigma_{i=1}^{n}p_{i}=1$.

Throughout the paper we assume that the elements $s_{i}$ are ordered in non-increasing order of their corresponding probabilities $p_{i}$. Given an element distribution, a fixed-to-variable code $\sigma$, which we call code for short, is a set of binary codewords $B$ (of different lengths in general), with a mapping from $S$ to $B$ called an encoder, and a mapping from $B$ to $S$ called a decoder.

In this paper we are mostly interested in coding data entries composed of element pairs, so we extend Definition 1 to distributions of two-element data entries. For short, we refer to a data entry simply as an entry.

Definition 2** (Entry Distribution).**

An entry distribution $D=[(S_{1},P_{1}),(S_{2},P_{2})]$ = \Big{[}((s_{1,1},\dots,s_{1,n_{1}}), $(p_{1,1},\dots,p_{1,n_{1}})),$ $((s_{2,1},\dots,s_{2,n_{2}}),$ (p_{2,1},\dots,p_{2,n_{2}}))\Big{]} is a juxtaposition of two element distributions. Each field in an entry is drawn randomly and independently according to its corresponding distribution in $D$, i.e., $\Pr\left(a_{1}=s_{1,i}\right)=p_{1,i}$ and $\Pr\left(a_{2}=s_{2,i}\right)=p_{2,i}$, with $p_{1,i},p_{2,i}>0$. The numbers of possible elements in the first and second field of an entry are $n_{1}=|S_{1}|$ and $n_{2}=|S_{2}|$, respectively.

Example 1**.**

The entry distribution of the two fields illustrated in Table I can be specified as D=[(S_{1},P_{1}),(S_{2},P_{2})]=\Big{[}((a,b,c,d,e),(0.4,0.3,0.16,0.08,0.06)),((x,y,z),(0.5,0.3,0.2))\Big{]}.

Once we defined the distributions governing the creation of entries, we can proceed to definitions addressing the coding of such entries. In all our coding schemes, entries are encoded to bit strings of fixed length $L$ (possibly with some padding), where $L$ is a parameter equal to the width of the memory in use. An entry coding scheme is defined through its encoding function.

Definition 3** (Entry Encoding Function).**

An entry encoding function $\Sigma$ is a mapping $\Sigma:(S_{1},S_{2})\to\{0,1\}^{L}\cup\{\perp\}$, where $\{0,1\}^{L}$ is the set of binary vectors of length $L$, and $\perp$ is a special symbol denoting encoding failure.

The input to the encoding function are two elements, one for field $1$ taken from $S_{1}$, and one for field $2$ taken from $S_{2}$. In successful encoding, the encoder outputs a binary vector of length $L$ to be stored in memory, and this vector is obtained uniquely for this input entry. The encoder is allowed to fail, in which case the output is the special symbol $\perp$. When the encoder fails, we assume that an alternative representation is found for the entry, and stored in a different memory not bound to the width-$L$ constraint. This alternative representation is outside the scope of this work, but because we know it results in higher cost (in space and/or in time), our objective here is to minimize the failure probability of the encoding function. Before we formally define the failure probability, we give a definition of the decoding function matching the encoding function of Definition 3.

Definition 4** (Entry Decoding Function).**

An entry decoding function $\Pi$ is a mapping $\Pi:\{0,1\}^{L}\to(S_{1},S_{2})$, such that if $\Sigma(a_{1},a_{2})=b\neq\perp$, then $\Pi(b)=(a_{1},a_{2})$. The decoding function is limited to use $O(n_{1}+n_{2})$ space to store the mapping.

The definition of the decoding function is straightforward: it is the inverse mapping of the encoding function when encoding succeeds. The constraint of using $O(n_{1}+n_{2})$ space for dictionaries is crucial: combining the two fields of the entry to one product distribution on $n_{1}\cdot n_{2}$ elements is both practically infeasible and theoretically uninteresting. With encoding and decoding in place, we turn to define the important measure of encoding success probability.

Definition 5** (Encoding Success Probability).**

Given an entry distribution $D=[(S_{1},P_{1}),(S_{2},P_{2})]$ and an entry encoding function $\Sigma$, define the encoding success probability as

[TABLE]

where the probability is calculated over all pairs of $(a_{1},a_{2})$ from $(S_{1},S_{2})$ according to their corresponding probabilities $(P_{1},P_{2})$.

Concluding this section, we define an entry coding scheme as a pair of entry encoding function $\Sigma$ and matching entry decoding function $\Pi$, meeting the respective definitions above. Our principal objective in this paper is to find an entry coding scheme that maximizes the encoding success probability given the entry distribution and the output length $L$.

III Optimal Entry Coding

In this section we develop a general entry coding scheme that yields optimal encoding success probability under some reasonable assumptions on the entry encoding function.

III-A Uniquely decodable entry encoding

The requirement from the decoding function to invert the encoding function for all successful encodings implies that the encoding function must be uniquely decodable, that is, an encoder-output bit vector $b$ represents a unique entry $(a_{1},a_{2})$. Constraining the decoding function to using only $O(n_{1}+n_{2})$ space implies using two codes: $\sigma_{1}$ for $S_{1}$ and $\sigma_{2}$ for $S_{2}$, because joint encoding of $S_{1},S_{2}$ in general requires $O(n_{1}\cdot n_{2})$ space for decoding. We then associate an entry encoding function $\Sigma$ with the pair of codes $(\sigma_{1},\sigma_{2})$. The entry decoding function must first “parse” the vector $b$ to two parts: one used to decode $a_{1}\in S_{1}$ and one to decode $a_{2}\in S_{2}$. An effective way to allow this parsing is by selecting the code $\sigma_{1}$ for $S_{1}$ to be a prefix code, defined next.

Definition 6** (Prefix Code).**

For a set of elements $S$, a code $\sigma$ is called a prefix code if in its codeword set $B$ no codeword is a prefix (start) of any other codeword.

Thanks to using a prefix code for $S_{1}$, the decoder can read the length-$L$ vector $b$ from left to right and unequivocally find the last bit representing $a_{1}$, and in turn the remaining bits representing $a_{2}$. Hence in the remainder of the section we restrict the entry encoding functions to have a constituent prefix encoder for their first field. This restriction of the encoder to be prefix is reminiscent of the classical result in information theory by Kraft-McMillan [13] that prefix codes are sufficient for optimal fixed-to-variable uniquely decodable compression. In our case we cannot prove that encoding functions with a prefix code for $S_{1}$ are always sufficient for optimality, but we also could not find better than prefix encoding functions that attain the $O(n_{1}+n_{2})$ space constraint at the decoder. To be formally correct, we call an entry coding scheme optimal in this section if it has the maximum success probability among coding schemes whose first code is prefix.

For unique decodability in the second field also, we use a special code property we call padding invariance.

Definition 7** (Padding-Invariant Code).**

A code is called a padding-invariant code if after eliminating the trailing (last) zero bits from its codewords $B$ (possibly including the empty codeword $\emptyset$ of length 0), $|B|$ distinct binary vectors are obtained.

Example 2**.**

Consider the following codes: $\sigma$ with a set of codewords $\{0,11,100,101\}$, $\sigma^{\prime}$ with $\{\emptyset,1,110\}$ and $\sigma^{\prime\prime}$ with $\{1,10,00\}$. The code $\sigma$ is a prefix code since none of its codewords is a prefix of another codeword. It is also a padding-invariant code since after eliminating the trailing zeros, the four distinct codewords $\emptyset$ (the empty codeword), $11,1$ and $101$ are obtained. While $\sigma^{\prime}$ is also a padding-invariant code (the codewords $\emptyset,1,11$ are distinct), it is not a prefix code since 1 is a prefix of 110 and $\emptyset$ is a prefix of 1 and 110. The code $\sigma^{\prime\prime}$ is neither a prefix code (1 is a prefix of 10) nor a padding-invariant code (the codeword 1 is obtained after eliminating the trailing zero bits of 1 and 10).

We now show how a prefix code $\sigma_{1}$ and a padding-invariant code $\sigma_{2}$ are combined to obtain a uniquely-decodable entry encoding function. The following is not yet an explicit construction, but rather a general construction method we later specialize to obtain an optimal construction.

Construction 0**.**

Let $\sigma_{1}$ be a prefix code and $\sigma_{2}$ be a padding-invariant code. The vector $b=\Sigma(a_{1},a_{2})$ output by the entry encoding function $\Sigma=(\sigma_{1},\sigma_{2})$ is defined as follows. The entry encoding function first maps $a_{1}$ to a word of $\sigma_{1}$ and $a_{2}$ to a word of $\sigma_{2}$. Then the vector $b$ is formed by concatenating the codeword of $\sigma_{2}$ to the right of $\sigma_{1}$’s codeword, and padding the resulting binary vector with zeros on the right to get $L$ bits total. If the two codewords exceed $L$ bits before padding, the entry encoding function returns $\perp$.

We show that because $\sigma_{1}$ is a prefix code while $\sigma_{2}$ is a padding-invariant code (not necessarily prefix), the entry encoding function of Construction  ‣ III-A is uniquely decodable. It is easy to see also that padding invariance of the second code is necessary to guarantee that.

Property 1**.**

Let $\Sigma=(\sigma_{1},\sigma_{2})$ be an entry encoding function specified by Construction  ‣ III-A. Let $x=(x_{1},x_{2}),y=(y_{1},y_{2})$ be two entries composed of elements $x_{1},y_{1}\in S_{1}$, $x_{2},y_{2}\in S_{2}$. For $\cdot$ representing concatenation, let $\Sigma(x)=\sigma_{1}(x_{1})\cdot\sigma_{2}(x_{2})\cdot 0\cdot\ldots\cdot 0$ and $\Sigma(y)=\sigma_{1}(y_{1})\cdot\sigma_{2}(y_{2})\cdot 0\cdot\ldots\cdot 0$ be the two length-$L$ vectors output by the entry encoding function $\Sigma$. If $\Sigma(x)=\Sigma(y)$, then necessarily $x=y$, i.e., $x_{1}=y_{1},x_{2}=y_{2}$.

To show the correctness of this property we explain why the decoding of a coded entry is unique.

Proof.

We first explain that there is a single element $u_{1}\in S_{1}$ whose encoding $\sigma_{1}(u_{1})$ is a prefix of $\Sigma(x)=\Sigma(y)$. Assume the contrary, and let $u,u^{\prime}\in S_{1}$ be two (distinct) elements such that $\sigma_{1}(u)$ and $\sigma_{1}(u^{\prime})$ are both prefixes of $\Sigma(x)=\Sigma(y)$. It then follows that either $\sigma_{1}(u)$ is a prefix of $\sigma_{1}(u^{\prime})$ or $\sigma_{1}(u^{\prime})$ is a prefix of $\sigma_{1}(u)$. Since $u\neq u^{\prime}$ and $\sigma_{1}$ is a prefix code, both options are impossible, and we must have that there is indeed a single element $u_{1}\in S_{1}$ whose encoding $\sigma_{1}(u_{1})$ is a prefix of $\Sigma(x)=\Sigma(y)$. Thus necessarily $\sigma_{1}(x_{1})=\sigma_{1}(y_{1})$ and $x_{1}=y_{1}$. By eliminating the first identical bits that stand for $\sigma_{1}(x_{1})=\sigma_{1}(y_{1})$ from $\Sigma(x)=\Sigma(y)$, we are left with the encoding of the second field, possibly including zero trailing bits. By the properties of the padding-invariant code $\sigma_{2}$, it has at most a single element that maps to the bits remaining after eliminating any number of the trailing zero bits. We identify this codeword as $\sigma_{2}(x_{2})=\sigma_{2}(y_{2})$, and deduce the only possible element $x_{2}=y_{2}$ of this field that has this codeword. ∎

To decode an encoded entry composed of the encodings $\sigma_{1}(x_{1})$, $\sigma_{2}(x_{2})$, possibly with some padded zero bits, a decoder can simply identify the single element $(u_{1}=x_{1})\in S_{1}$ whose encoding $\sigma_{1}(u_{1}=x_{1})$ is a prefix of the encoded entry. As $\sigma_{1}$ is a prefix code, there cannot be more than one such element. The decoder can then identify the beginning of the second part of the encoded entry, which encodes a unique codeword of $\sigma_{2}$ with some trailing zero bits. This is necessarily $\sigma_{2}(x_{2})$ from which $x_{2}$ can be derived.

Toward finding an entry coding scheme with maximal encoding-success probability, we would now like to find codes $\sigma_{1}$ and $\sigma_{2}$ for Construction  ‣ III-A that are optimal given an entry distribution $D$ and the output length $L$. Since the direct joint computation of optimal $\sigma_{1}$ and $\sigma_{2}$ codes is difficult, we take a more indirect approach to achieve optimality. 1) we first derive an efficient algorithm to find an optimal prefix code for $S_{1}$ given a code for $S_{2}$, and then 2) we find a (padding-invariant) code for $S_{2}$ that is universally optimal for any code used for $S_{1}$. This way we reduce the joint optimization of $\sigma_{1}$,$\sigma_{2}$ (hard) to a conditional optimization of $\sigma_{1}$ given $\sigma_{2}$ (easier). We prove that this conditionally optimal $\sigma_{1}$ is also unconditionally optimal. To do so, we show that the code $\sigma_{2}$ assumed in the conditioning is optimal for any choice of $\sigma_{1}$. These two steps are detailed in the next two sub-sections, and later used together to establish an optimal coding scheme.

III-B An optimal prefix code for $S_{1}$ conditioned on a code for $S_{2}$

Our objective in this sub-section is to find a prefix code $\sigma_{1}$ that maximizes the encoding success probability given a code $\sigma_{2}$ for the second field. We show constructively that this task can be achieved with an efficient algorithm.

We denote $W_{i}=\left\lceil\log_{2}(n_{i})\right\rceil$ for $i\in\{1,2\}$ and $W=W_{1}+W_{2}$. Finding an optimal coding scheme when $L\geq W$ is easy. In that case we could allocate $W_{1}$ bits to $\sigma_{1}$ and $W_{2}$ bits to $\sigma_{2}$ and apply two independent fixed-length codes with a total of $W$ bits and obtain a success probability $1$. Thus we focus on the interesting case where $L\leq W-1$.

Given a code $\sigma_{2}=\sigma$ for $S_{2}$, we show a polynomial-time algorithm that finds an optimal conditional prefix code $\sigma_{1}$ for $S_{1}$. This code $\sigma_{1}$ will have an encoding function maximizing the probability $P_{success}(D,\Sigma=(\sigma_{1},\sigma_{2}=\sigma))$ given $\sigma_{2}=\sigma$, when $\sigma_{1}$ is restricted to be prefix. Then, we also say that $\Sigma=(\sigma_{1},\sigma_{2})$ with its corresponding decoding function $\Pi$ is an optimal conditional coding scheme.

To build the code $\sigma_{1}$ we assign codewords to $n^{\prime}$ elements of $S_{1}$, where $n^{\prime}\leq n_{1}$. Clearly, all such codewords have lengths of at most $L$ bits. For a (single) binary string $x$, let $\ell(x)$ denote the length in bits of $x$. Since for every element $a\in S_{1}$ which is assigned a codeword the code $\sigma_{1}$ satisfies $\ell(\sigma_{1}(a))\leq L$, it holds that $2^{-\ell(\sigma_{1}(a))}$ must be an integer multiple of $2^{-L}$. We define the weight of a codeword of length $\ell_{0}$ as the number of units of $2^{-L}$ in $2^{-\ell_{0}}$, denoted by $N_{\ell_{0}}=2^{-\ell_{0}}/2^{-L}=2^{L-\ell_{0}}$. The $n_{1}-n^{\prime}$ elements of $S_{1}$ not represented by $\sigma_{1}$ are said to have length $\ell_{0}=\infty$ and codeword weight zero. A prefix code exists with prescribed codeword lengths if they satisfy Kraft’s inequality [14]. In our terminology, this means that the sum of weights of the codewords of $\sigma_{1}$ need be at most $2^{L}$.

Definition 8**.**

Consider entries composed of an element from the first (highest probability) $k$ elements of $S_{1}$ (for $k\in[0,n_{1}]$), and an arbitrary element of $S_{2}$. For $N\in[0,2^{L}]$ and $k\in[0,n_{1}]$, we denote by $F(k,N)$ the maximal sum of probabilities of such entries that can be encoded successfully by a prefix code $\sigma_{1}$ whose sum of weights for the first $k$ codewords is at most $N$. Formally,

[TABLE]

where $\emph{I}[\cdot]$ is the indicator function. Note that $F(k,N)$ depends on the conditioned $\sigma_{2}$, but we keep this dependence implicit to simplify notation.

The following theorem relates the maximal success probability of a conditional coding scheme and the function $F(k,N)$.

Theorem 1**.**

The maximal success probability of a conditional coding scheme is given by

[TABLE]

Proof.

To satisfy Kraft’s inequality we should limit the sum of weights $N$ to $2^{L}$. In addition, the success probability of the coding scheme with an encoding function $\Sigma$ is calculated based on entries with any number of the $n_{1}$ elements of $S_{1}$. ∎

We next show how to compute $F(k,N)$ efficiently for all $k,N$, in particular for $k=n_{1}$, $N=2^{L}$ that yield the optimal conditional $\sigma_{1}$. To do that, we use the following recursive formula for $F(k,N)$. First note the boundary cases $F(k=0,N)=0$ for $N\geq 0$ and $F(k=0,N)=-\infty$ for $N<0$ (this means an invalid code). We can now present the formula of $F(k,N)$ that calculates its values for $k$ based on the values of the function for $k-1$ and $N^{\prime}\leq N$.

Lemma 2**.**

The function $F(k,N)$ satisfies for $N\geq 0,k\geq 1$

[TABLE]

Proof.

The optimal code that attains $F(k,N)$ either assigns a codeword to $s_{1,k}$ or does not. The two arguments of the outer max function in (2) are the respective success probabilities for these two choices. In the former case we consider all possible lengths of the codeword of $s_{1,k}$. A codeword length of $\ell_{0}$ reduces the available sum of weights for the first $k-1$ elements by $N_{\ell_{0}}=2^{L-\ell_{0}}$. In addition, an entry $(s_{1,k},s_{2,i})$ contributes to the success probability the value $p_{1,k}\cdot p_{2,i}$ if its encoding width (given $\ell_{0}$) is at most $L$. In the latter case the $k^{\text{th}}$ element does not contribute to the success probability and has no weight, hence $F(k,N)=F(k-1,N)$ in this case. ∎

Finally, the pseudocode of the dynamic-programming algorithm that finds the optimal conditional code based on the above recursive formula is given in Algorithm 1. It iteratively calculates the values of $F(k,N)$. It also uses a vector $Q(k,N)$ to represent the codeword lengths for the first at most $k$ elements of $S_{1}$ in a solution achieving $F(k,N)$.

Time Complexity: By the above description, there are $n_{1}$ iterations and in each $O(2^{L})$ values are calculated, each by considering $O(L)$ sums of $n_{2}$ elements. It follows that the time complexity of the algorithm is $O(n_{1}\cdot 2^{L}\cdot L\cdot n_{2})=O(n_{1}^{2}\cdot n_{2}^{2}\cdot L)$, which is polynomial in the size of the input. The last equality follows from the fact that $L<\lceil\log_{2}(n_{1})\rceil+\lceil\log_{2}(n_{2})\rceil$ in a non-trivial instance where some entries fail encoding.

III-C A universally optimal code for $S_{2}$

We now develop the second component required to complete an unconditionally optimal entry coding scheme: a code for the $S_{2}$ elements that is optimal for any code $\sigma_{1}$ used for the $S_{1}$ elements.

To that end we now define the natural notion of a monotone coding scheme, in which higher-probability elements are assigned codeword lengths shorter or equal to lower-probability ones.

Definition 9** (Monotone Coding Scheme).**

A coding scheme with an encoding function $\Sigma$ of an entry distribution $D=[(S_{1},P_{1}),(S_{2},P_{2})]$ is called monotone if is satisfies

(i)* for $j\in\{1,2\}$, $i<i^{\prime}$ implies that $\ell(\sigma_{j}(s_{j,i}))\leq\ell(\sigma_{j}(s_{j,i^{\prime}}))$ for two elements $s_{j,i},s_{j,i^{\prime}}\in S_{j}$.*

(ii)* for $j\in\{1,2\}$, the elements that are not assigned a codeword in $\sigma_{j}$ are the last elements in $S_{j}$ (if any).*

It is an intuitive fact that without loss of generality an optimal entry coding scheme can be assumed monotone. We prove this in the following.

Lemma 3**.**

For any entry distribution $D=[(S_{1},P_{1}),(S_{2},P_{2})]$, and any memory width $L\geq 1$, there exists a monotone optimal coding scheme.

Proof.

We show how to build an optimal monotone coding scheme based on any optimal coding scheme with an encoding function $\Sigma$. For $j\in\{1,2\}$ consider two arbitrary indices $i,i^{\prime}$ that satisfy $i<i^{\prime}$. Then necessarily $p_{j,i}\geq p_{j,i^{\prime}}$. If codewords are assigned to the two elements and $\ell(\sigma_{j}(s_{j,i}))>\ell(\sigma_{j}(s_{j,i^{\prime}}))$, we can replace $\sigma_{j}$ by a new code obtained by permuting the two codewords of $s_{j,i},s_{j,i^{\prime}}$. With this change, an entry $(a_{1},a_{2})$ with $a_{j}=s_{j,i}$ is encoded successfully after the change if the corresponding entry with $a_{j}=s_{j,i^{\prime}}$ was encoded successfully before the change. Then, we deduce that such a change cannot decrease $P_{success}$, and the result follows. In addition, if some elements are not assigned codewords, the same argument shows that these elements should be the ones with the smallest probabilities. ∎

The optimality of the code to be specified for $S_{2}$ will be established by showing it attains the following upper bound on success probability.

Proposition 4**.**

Given any code $\sigma_{1}$, the encoding success probability of any entry encoding function is bounded from above as follows.

[TABLE]

where $n^{\prime}$ is the highest index of an element $s_{1,i}$ that is assigned a codeword by $\sigma_{1}$.

Proof.

First by the monotonicity property proved in Lemma 3, $\sigma_{1}$ assigns codewords to elements $s_{1,i}$ with indices $i\in\{1,\ldots,n^{\prime}\}$, for some $n^{\prime}\leq n_{1}$. Hence for indices greater than $n^{\prime}$ the success probability is identically zero. Given an element $s_{1,i}$ with a $\sigma_{1}$ codeword of length $\ell_{i}$, at most $2^{L-\ell_{i}}$ elements of $S_{2}$ can be successfully encoded with it in an entry. So the inner sum in (5) is the maximal success probability given $s_{1,i}$. Summing over all $i$ and multiplying by $p_{1,i}$ gives the upper bound. ∎

It turns out that there exists a padding-invariant code $\hat{\sigma}_{2}$ that attains the upper bound of Proposition 4 with equality for any code $\sigma_{1}$. For the ordered set $S_{2}=(s_{2,1},\ldots,s_{2,n_{2}})$, let the encoder of $\hat{\sigma}_{2}$ map $s_{2,j}$ to the shortest binary representation of $j-1$ for $j\geq 2$, and to $\emptyset$ for $j=1$. The binary representation is put in the codeword from left to right, least-significant bit (LSB) first. Then we have the following.

Proposition 5**.**

Given any code $\sigma_{1}$, the encoding success probability of $\hat{\sigma}_{2}$ is

[TABLE]

where $n^{\prime}$ is the highest index of an element $s_{1,i}$ that is assigned a codeword by $\sigma_{1}$.

Proof.

If the codeword of $\sigma_{1}$ uses $\ell$ bits, there are $L-\ell$ bits left vacant for $\sigma_{2}$. The mapping specified for $\hat{\sigma}_{2}$ allows encoding successfully the first $2^{L-\ell}$ elements of $S_{2}$, which gives the stated success probability. ∎

In particular, when the encoding of $\sigma_{1}$ has length $L$, the single element $s_{2,1}$ is encoded successfully by the empty codeword $\emptyset$. Other examples are the two codewords ($\emptyset$, 1) when $\ell=L-1$, and the four codewords ($\emptyset$, 1, 01, 11) when $\ell=L-2$. It is clear that the code $\hat{\sigma}_{2}$ is padding invariant, because its codewords are minimal-length binary representations of integers. Now we are ready to specify the optimal entry coding scheme in the following.

Construction 1**.**

Given $L$ and $D$, let $\hat{\sigma}_{1}$ be the prefix code obtained by applying Algorithm 1 on the code $\hat{\sigma}_{2}$. Then the entry encoding function $\hat{\Sigma}=(\hat{\sigma}_{1},\hat{\sigma}_{2})$ is defined by applying Construction  ‣ III-A on $\hat{\sigma}_{1}$,$\hat{\sigma}_{2}$.

Theorem 6**.**

For any entry distribution $D$ and a memory width $L$, Construction 1 gives an optimal entry coding scheme, that is, a coding scheme that maximizes the success probability among all uniquely-decodable coding schemes with a prefix code in the first field.

From Theorem 6 we can readily obtain an efficient algorithm finding an optimal two-field entry encoding, which is given in Algorithm 2. Optimality is proved up to the assumption that the first code is a prefix code. It is not clear how one can obtain better codes than those with a prefix $\sigma_{1}$, while keeping the dictionary size $O(n_{1}+n_{2})$.

For the special case when $\sigma_{2}$ is $\hat{\sigma}_{2}$, the recursive formula for the calculation of the function $F(k,N)$ can be simplified as follows.

Lemma 7**.**

When $\sigma_{2}$ is $\hat{\sigma}_{2}$, the function $F(k,N)$ satisfies for $N\geq 0,k\geq 1$

[TABLE]

It is easily seen that (7) is obtained from (3) by replacing the indicator function with the partial sum that accommodates all the $S_{2}$ elements that have a short enough representation to fit alongside the $S_{1}$ element.

The following example illustrates Construction 1 on the entry distribution from the Introduction.

Example 3**.**

Consider the entry distribution D=\Big{[}((a,b,c,d,e),(0.4,0.3,0.16,0.08,0.06)),((x,y,z),(0.5,0.3,0.2))\Big{]} from Table I with $n_{1}=5,n_{2}=3$. The width parameter is $L=4$. For the ordered set $S_{2}=(x,y,z)$, we select the code $\hat{\sigma}_{2}$ by mapping $s_{2,1}$ to $\emptyset$ and $s_{2,j}$ (for $j=2,3$) to the shortest binary representation of $j-1$. Then, $\hat{\sigma}_{2}(x)=\emptyset,\hat{\sigma}_{2}(y)=1,\hat{\sigma}_{2}(z)=01$ and $\ell(\hat{\sigma}_{2}(x))=0,\ell(\hat{\sigma}_{2}(y))=1,\ell(\hat{\sigma}_{2}(z))=2$. The code $\hat{\sigma}_{2}(x)$ is a padding-invariant code. To get the prefix code $\hat{\sigma}_{1}$ we apply Algorithm 1 on the code $\hat{\sigma}_{2}$. We recursively calculate the values of $F(k,N)$ and $Q(k,N)$ for $k\in[1,n_{1}=5]$, $N\in[0,2^{L}=16]$. In particular, for each value of $k$ the values are calculated based on the previous value of $k$. The values are listed in Table II. Each column describes a different value of $k$. (Whenever values of $F$ and $Q$ are not shown, a specific value of $N$ does not improve the probability achieved for a smaller value of $N$ in the same column.) The value of $N$ implies a restriction on the values of the codeword lengths. If the lengths of the codewords are described by a set $Q$, they must satisfy $\Sigma_{\ell\in Q}2^{-\ell}/2^{-L}\leq N$, i.e. $\Sigma_{\ell\in Q}2^{-\ell}\leq N/16$.

We first explain the values for $k=1$, considering the contribution to the success probability of data entries with $a\in S_{1}$ as the first element. This happens w.p. $p_{1,1}=0.4$. For $N=1$ we must have $Q(1,1)=(4)$, i.e. the element a is assigned a codeword of length 4. Then, there is a single pair $(a,x)$ that can be encoded successfully and $P_{success}$ is given by $F(1,1)$ is $0.4\cdot 0.5=0.2$. Likewise, for $N=2$, we can have a codeword of length 3 for $a$ and the two pairs $(a,x),(a,y)$ can be encoded within $L=4$ bits, such that $F(1,2)=0.4\cdot(0.5+0.3)=0.32$. If $N=3$ we cannot further decrease the codeword length and improve the success probability. For $N=4$ we can have a codeword length of 2 bits, as described by $Q(1,4)=(2)$. This enables encoding successfully the three pairs $(a,x),(a,y),(a,z)$ with a success probability of 0.4 as given by $F(1,4)$. The values for larger values of $k$ are calculated in a similar manner based on the recursive formulas. The optimal codeword lengths for $\hat{\sigma}_{1}$ are given by $Q(k=n_{1}=5,N=2^{L}=16)=(2,2,2,3,3)$. This enables to encode successfully all pairs besides $(d,z),(e,z)$, achieving $P_{success}=0.972$ as given by $F(k=5,N=16)$.

Finally, by applying Construction  ‣ III-A on $\hat{\sigma}_{1}$,$\hat{\sigma}_{2}$ we obtain the entry encoding function $\hat{\Sigma}=(\hat{\sigma}_{1},\hat{\sigma}_{2})$.

IV Optimal Entry Encoding with the Same Code for Both Fields

In this section we move to study the problem of entry coding schemes for the special case where we require that both fields use the same code. It is commonly the case that both fields have the same element distribution, and then using one code instead of two can cut the dictionary storage by half. In practice this can offer a significant cost saving. Throughout this section we thus assume that the fields have the same distribution, but the results can be readily extended to the case where the distributions are different and we still want a single code with optimal success probability. Formally, in this section our problem is to efficiently design a single code $\sigma$ that offers optimal encoding success probability in a width-$L$ memory.

In the special case of a single distribution we have an element distribution $(S,P)$ = $((s_{1},\dots,s_{n}),(p_{1},\dots,p_{n}))$, and the entry distribution is $D=[(S,P),(S,P)]$. Now the space constraint for the decoder (to hold the dictionary) is to be of size at most $n$. This means that we need to find one code $\sigma$ for $(S,P)$ that will be used in both fields. To be able to parse the two fields of the entry, $\sigma$ needs to be a prefix code.

IV-A Observations on the problem

Before moving to solve the problem, it will be instructive to first understand the root difficulty in restricting both fields to use the same code. If we try to extend the dynamic-programming solution of Section III to the single-distribution,single-code case, we get the following maximization problem

[TABLE]

where we adapted the expression from (2) to the case of a single distribution and a single code. But now trying to extend the recursive expression for $F(k,N)$ in (3) gives

[TABLE]

which cannot be used by the algorithm because the indicator function now depends on lengths of codewords that were not assigned yet, namely for the elements $i\in[k+1,n]$. So even though we now only have a single code to design, this task is considerably more challenging than the conditional optimization of Section III-B. At this point the only apparent route to solve (8) is by trying exhaustively all length assignments to $S$ satisfying Kraft’s inequality, and enumerating the arguments of the max function in (8) directly. But this would be intractable.

IV-B Efficient algorithm for optimal entry encoding

In the remainder of the section we show an algorithm that offers an efficient way around the above-mentioned difficulty to assign codeword lengths to elements. We present this efficient algorithm formally, but first note its main idea.

The main idea: we showed in Section IV-A that it is not possible to maximize the single-code success probability for $k$ elements given the optimal codeword lengths for $k-1$ elements. So it does not work to successively add elements to the solution while maintaining optimality as an invariant. But fortunately, it turns out that it does work to successively add codeword lengths to the solution while maintaining optimality as an invariant. The subtle part is that the lengths need to be added in a carefully thought-of sequence, which in particular, is not the linear sequence $(1,2,\ldots,L-1)$ or its reverse-ordered counterpart. We show that if the codeword lengths are added in the order of the sequence

[TABLE]

(for even111For convenience we assume that $L$ is even, but all the results extend to odd $L$. $L$), then for any sub-sequence we can maximize the success probability given the optimal codeword lengths taken from the sub-sequence that is one shorter. For example, when $L=8$ our algorithm will first find an optimal code only using codeword length $L/2=4$; based on this optimum it will find an optimal code with lengths $4$ and $5$, and then continue to add the codeword lengths $3,6,2,7,1$ in that order.

We now turn to a more formal treatment of the algorithm. We first define the function holding the optimal success probabilities for sub-problems of the problem instance. The following Definition 10 is the adaptation of Definition 8 to the sequence of codeword lengths applicable in the single-code case.

Definition 10**.**

Consider assignments of finite codeword lengths to the consecutive elements $\{s_{k_{1}},\ldots,s_{k_{2}}\}$ from $S$, where the lengths are assigned from the values $\{L/2,L/2+1,L/2-1,\ldots,l\}$ taken from the sub-sequence of (10) that ends with $l$. For $N\in[0,2^{L}]$ we denote by $G(l,[k_{1},k_{2}],N)$ the maximal success probability for such an assignment whose sum of weights for these $k_{2}-k_{1}+1$ codewords is at most $N$. Formally,

[TABLE]

The following two theorems are the key drivers of the efficient dynamic-programming algorithm finding the optimal code.

Theorem 8**.**

Let $l=L/2-d+1$ for some integer $1\leq d\leq L/2-1$. For the length subsequent to $l$ in the sequence (10) $l^{\prime}=L/2+d$, we have the following

[TABLE]

where we define

[TABLE]

Proof.

Given maximal values $G$ for all values of $N$ and with lengths up to $l$ in the sequence, the maximal value $G$ when $l^{\prime}$ is also allowed is obtained by assigning length $l^{\prime}$ to between [math] and $k_{2}-k_{1}+1$ elements in the range $[k_{1},k_{2}]$. By the monotonicity of $p_{i}$, $l^{\prime}$ which is higher than all previous lengths must be assigned to the highest $j$ indices in the range $[k_{1},k_{2}]$. Thus for each $j$ the success probability is the value of $G$ for the corresponding range of elements $[k_{1},k_{2}-j]$ with the residual weight $N-j\cdot N_{l^{\prime}}$. In particular, the elements assigned length $l^{\prime}$ do not add to the success probability, because $l^{\prime}$ plus any length in the sub-sequence up to $l$ exceeds $L$. In the extreme case when $j=k_{2}-k_{1}+1$ (all elements assigned length $l^{\prime}$), the definition (13) when appearing in the right-hand side of (12) gives a valid assignment with success probability [math] if $N$ in the left-hand side is sufficiently large. ∎

An important property provided by the length sequence (10) is that the success probability up to $l^{\prime}$ depends only on the success probability up to $l$, no matter what the assignments are among the lengths up to $l$. With this property Theorem 8 allows to efficiently extend the optimality from $l$ of type $l=L/2-d+1$ (with $d\geq 1$) to the next length in the sequence. To complete what is required for an efficient algorithm, we need the same extension of optimality from $l$ of type $l=L/2+d$ to the next length in the sequence. We do this in the next theorem.

Theorem 9**.**

Let $l=L/2+d$ for some integer $1\leq d\leq L/2-1$. For the length subsequent to $l$ in the sequence (10) $l^{\prime}=L/2-d$, we have the following

[TABLE]

Proof.

Given maximal values $G$ for all values of $N$ and with lengths up to $l$ in the sequence, the maximal value $G$ when $l^{\prime}$ is also allowed is obtained by assigning length $l^{\prime}$ to between [math] and $k_{2}-k_{1}+1$ elements in the range $[k_{1},k_{2}]$. By the monotonicity of $p_{i}$, $l^{\prime}$ which is lower than all previous lengths must be assigned to the lowest $j$ indices in the range $[k_{1},k_{2}]$. Now the success probability has two components: first is the success between pairs of elements assigned lengths up to $l$ in the sequence, and second is the success between element pairs that involve the new length $l^{\prime}$ (recall that in Theorem 8 the second component did not exist, but here it does because $l^{\prime}$ is lower than all lengths up to $l$.) The first of the three terms in the summation of (14) gives the first component, and the latter two terms give the second component. Maximization is done as before by considering all possible $j$ with the residual weight $N-j\cdot N_{l^{\prime}}$. ∎

As in Theorem 8, the length sequence provides the property that maximization in Theorem 9 only depends on the success probabilities of the previous length sub-sequence, without care to the assignment of lengths among the lengths in that sub-sequence. In the following Algorithm 3 we formally present the algorithm for finding an optimal single prefix code, building on Theorems 8 and 9. For terseness we only track the optimal success probabilities $G$, omitting the more technical task of tracking the optimal assigned lengths, which is required to find the optimal code in a real implementation. The noteworthy parts of Algorithm 3 are:

• •

The initialization of $G$ to $-\infty$ for negative $N$, and to [math] for non-negative $N$ and empty ranges of elements (according to (13)).

• •

Starting the length sequence at $l=L/2$ and calcualting the success probability when all elements in the range are assigned that length.

• •

The main iteration following the progressions in the length sequence using Theorems 8 and 9.

• •

Outputing the optimal success probability for the code parameters as a maximization over all $n$ possible numbers of elements assigned codewords, starting from the first element.

Time Complexity: There are $L-1$ iterations and in each $O(2^{L})$ values are calculated for each of $O(n^{2})$ ranges of elements, each by considering $O(n)$ possibilities for the number of elements with the new codeword length. It follows that the time complexity of the algorithm is $O(n^{3}\cdot 2^{L}\cdot L)$, which is polynomial in the size of the input, or $O(n^{5}\cdot L)$ (recall that $2^{L}$ is at most quadratic in $n$). It is interesting to compare this complexity as one order of $n$ higher than that of Algorithm 2 ($O(n^{4}\cdot L)$) for the two-code case. We note that the $n^{5}$ complexity term is a loose bound, because many of the counted iterations are not exercised in a given run.

As mentioned in the beginning of the section, Algorithm 3 can be extended to the case where the two fields have different distributions on the same elements, and a single code with optimal success probability is found. We do not explore this generalization in the paper, but it is as simple as replacing some instances of $p_{i}$ in the algorithm by $q_{i}$ corresponding to the distribution of the second field.

Example 4**.**

Consider the entry distribution $D=[(S,P),(S,P)]$ with $(S,P)$ = $((s_{1},\dots,s_{n}),(p_{1},\dots,p_{n}))$ satisfying $(p_{1},\dots,p_{n})=(0.4,0.4,0.08,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01,0.01)$ with $n=15$. The width parameter is $L=6$ and assume that a single code is allowed. A simple possible coding scheme can assign codewords of length $L/2=3$ to the $2^{3}=8$ most probable elements $s_{1},\dots,s_{8}$ where codewords are not assigned for the remaining elements $s_{9},\dots,s_{15}$. This scheme successfully encodes all pairs of elements composed of two elements from $s_{1},\dots,s_{8}$ achieving success probability of P^{\prime}_{success}=\Big{(}\sum_{i=1}^{8}p_{i}\Big{)}^{2}=0.93^{2}=0.8649. Alternatively, the above optimal algorithm finds a code that achieves higher success probability. In this code elements $s_{1},s_{2}$ are assigned codewords of length 2 while elements $s_{3},\dots,s_{10}$ are assigned codewords of length 4. Later elements are not assigned codewords. This optimal scheme achieves an improved success probability of P_{success}=\Big{(}\sum_{i=1}^{2}p_{i}\Big{)}^{2}+\sum_{i=1}^{2}p_{i}\cdot\sum_{i=3}^{10}p_{i}+\sum_{i=3}^{10}p_{i}\cdot\sum_{i=1}^{2}p_{i}=0.8^{2}+0.8\cdot 0.15+0.15\cdot 0.8=0.88.

V Experimental Results

We examine the performance of the suggested coding schemes. In the experiments, the probabilities in the element distributions follow the Zipf distribution with different parameters. A low positive Zipf parameter $\mu$ results in a distribution that is close to the uniform distribution, while for a larger parameter the distribution is more biased.

As a first step, we compare the suggested optimal coding schemes to Huffman coding [2]. The results are shown in Fig. 2. First, in Fig. 2(a), we pick two different distributions on 128 elements for the two fields, and set the assumption that the codes of the two fields may be different. The distributions of the first and second field are Zipf with parameters $\mu_{1}=0.8$ and $\mu_{2}=2$, respectively. We compute the optimal codes $\sigma_{1}$ and $\sigma_{2}$ using Algorithm 2 from Section III, and plot their success probability. In comparison, we also plot the success probability of the Huffman code computed for each field according to its distribution. It can be seen that while Huffman codes minimize the expected codeword length, in a fixed-width memory their encoding success probability is inferior to the optimal code for all values of $L$. For $L=2$ for instance, Huffman coding does not encode successfully any pair of elements, while the optimal coding scheme achieves a probability of 0.162. A maximal difference in the success probability of 0.289 is obtained for $L=6$.

In Fig. 2(b) we examine the scenario of the two fields having the same element distribution, with the requirement that a single code is shared by the two fields. An optimal coding scheme for this case was given in Section IV. The two fields have the same 128 elements with a Zipf $\mu=1.6$ distribution. We compute the optimal code $\sigma$ using Algorithm 3, and plot its success probability. In comparison, we also plot the success probability of the Huffman code computed from the common element distribution. Here too Huffman coding gives inferior success to the optimal code, with a maximal probability difference of 0.194 at $L=4$. We also plot for comparison the success probability of the optimal coding scheme when allowing two different codes for the two fields. The results underscore the fact that allowing two different codes can improve success probability even when the two fields follow the same distribution.

Next, we examine the impact of the number of possible elements in the fields for different Zipf distributions. We assume a single shared code and that the two fields follow the same distribution. In Fig 3, we consider two such distributions and present the optimal success probability. In Fig 3(a), the distribution has a parameter $\mu=0.5$. We assume that there are $n$ elements in each field. The minimal width required to obtain a success probability of 1 for $n$ elements is given by $L=2\cdot\log_{2}n$, i.e. $L\geq 6$ for $n=8$ and $L\geq 8$ for $n=16$. For a given $L$, the optimal success probability decreases when the number of elements in each field increases. For $L=8$ it equals 0.449, 0.208 and 0.099 for $n=32,64$ and $128$, respectively. Likewise, in Fig 3(b), the distribution has a parameter $\mu=2$, which results in a more biased element distribution. While again for $n=8$, a success probability of 1 is observed only for $L\geq 6=2\cdot\log_{2}n$, the more biased distribution allows achieving high success probabilities even for larger values of $n$. For instance, while for $n=128$ a memory width of $L=2\cdot\log_{2}n=14$ is required to guarantee a success probability of 1, already for $L=8$ we obtain probability of 0.939 when $\mu=2$.

VI Conclusion

In this paper we developed compression algorithms for fixed-width memories. We presented a new optimization problem of maximizing the probability to encode entries within the memory width, and described constraints on the codes for the different input fields that guarantee uniquely-decodable data entries. We derived algorithms that find an optimal coding scheme with maximum success probability for the scenario of two codes for two fields, as well as for a single shared code. We also presented experimental results to evaluate the success-probability performance. The central open problem left by this paper is extending optimal coding to $d>2$ fields.

VII Acknowledgment

We would like to thank Neri Merhav for fruitful discussions and pointers.