Combinatorial Entropy Encoding

TL;DR

This paper introduces a novel lossless data compression method that encodes messages by their lexicographic index among permutations, matching Shannon entropy without needing prior source models.

Contribution

The paper presents a new entropy encoding technique based on lexicographic permutation indices, eliminating the need for prior entropy models unlike Huffman or arithmetic coding.

Findings

Achieves compression matching Shannon entropy

Does not require prior source entropy model

Uses a simple algorithm based on multinomial coefficients

Abstract

This paper proposes a novel entropy encoding technique for lossless data compression. Representing a message string by its lexicographic index in the permutations of its symbols results in a compressed version matching Shannon entropy of the message. Commercial data compression standards make use of Huffman or arithmetic coding at some stage of the compression process. In the proposed method, like arithmetic coding entire string is mapped to an integer but is not based on fractional numbers. Unlike both arithmetic and Huffman coding no prior entropy model of the source is required. Simple intuitive algorithm based on multinomial coefficients is developed for entropy encoding that adoptively uses low number of bits for more frequent symbols. Correctness of the algorithm is demonstrated by an example.

Figures

Tables

Table 1. TABLE I : All the permutations of word B A N A N A : ⌈ log 2 ⁡ ( 6 3 , 1 , 2 ) ⌉ = 6 subscript 2 binomial 6 3 1 2 6 \lceil\log_{2}\binom{6}{3,1,2}\rceil=6 bit long binary code along with knowledge of character frequencies ( f A = 3 , f B = 1 , f N = 2 ) formulae-sequence subscript 𝑓 𝐴 3 formulae-sequence subscript 𝑓 𝐵 1 subscript 𝑓 𝑁 2 (f_{A}=3,f_{B}=1,f_{N}=2) is sufficient to uniquely represent any permutation.

Lexicogra-	Permutation	Code	Lexicogra-	Permutation	Code	Lexicogra-	Permutation	Code
phic Index			phic Index			phic Index
0	nnbaaa	1	20	nabana	10101	40	nbaaan	101001
1	nbnaaa	10	21	anbana	10110	41	bnaaan	101010
2	bnnaaa	11	22	banana	10111	42	nabaan	101011
3	nnabaa	100	23	abnana	11000	43	anbaan	101100
4	nanbaa	101	24	naabna	11001	44	banaan	101101
5	annbaa	110	25	anabna	11010	45	abnaan	101110
6	nbanaa	111	26	aanbna	11011	46	naaban	101111
7	bnanaa	1000	27	baanna	11100	47	anaban	110000
8	nabnaa	1001	28	abanna	11101	48	aanban	110001
9	anbnaa	1010	29	aabnna	11110	49	baanan	110010
10	bannaa	1011	30	nnaaab	11111	50	abanan	110011
11	abnnaa	1100	31	nanaab	100000	51	aabnan	110100
12	nnaaba	1101	32	annaab	100001	52	naaabn	110101
13	nanaba	1110	33	naanab	100010	53	anaabn	110110
14	annaba	1111	34	ananab	100011	54	aanabn	110111
15	naanba	10000	35	aannab	100100	55	aaanbn	111000
16	ananba	10001	36	naaanb	100101	56	baaann	111001
17	aannba	10010	37	anaanb	100110	57	abaann	111010
18	nbaana	10011	38	aananb	100111	58	aabann	111011
19	bnaana	10100	39	aaannb	101000	59	aaabnn	111100

Equations

$$H$$

$$\begin{array}[]{lcl}H&=&-\left(3/6\log_{2}(3/6)+1/6\log_{2}(1/6)+2/6\log_{2}(2/6)\right)\\ &=&1.4591\text{Bits/Symbol}\\ \end{array}$$

$$\text{Compression Ratio}=\frac{\text{Uncompressed Size}}{\text{Compressed Size}}$$

$$\text{Space Saving (\%age)}=(1-\frac{\text{Compressed Size}}{\text{Uncompressed Size}})\times 100$$

$$P(n,f_1,f_2,\cdots ,f_t)$$

$$n=f_1+f_2+\cdots +f_t$$

$$\left(\genfrac{}{}{0ex}{}{6}{3,1,2}\right)$$

$$H$$

$$={\log }_2(\frac{1}{p_1^{p_1}\cdot p_2^{p_2}\cdots p_t^{p_t}})$$

$$n\cdot H$$

$$2^{n\cdot H}$$

$$\frac{n^n}{f_1^{f_1}\cdot f_2^{f_2}\cdots f_t^{f_t}}>\frac{n!}{f_1!\cdot f_2!\cdots f_t!}$$

$$n^n>\frac{n!}{f_1!\cdot f_2!\cdots f_t!}{f}_1^{f_1}\cdot f_2^{f_2}\cdots f_t^{f_t}$$

$$(f_1+\cdots +f_t{)}^n>(\genfrac{}{}{0ex}{}{n}{f_1,\cdots ,f_t!})f_1^{f_1}\cdots f_t^{f_t}$$

$$W_{\text{Left Turn}}\left\{\left(\genfrac{}{}{0ex}{}{n-1}{f_0-1,f_1}\right)\to \left(\genfrac{}{}{0ex}{}{n}{f_0,f_1}\right)\right\}$$

$$W_{\text{Right Turn}}\left\{\left(\genfrac{}{}{0ex}{}{n-1}{f_0,f_1-1}\right)\to \left(\genfrac{}{}{0ex}{}{n}{f_0,f_1}\right)\right\}$$

$$L_i$$

$$251$$

$$131$$

$$47$$

$$26$$

$$11$$

$$1$$

$$0$$

$$\displaystyle\sum_{f_{1}+f_{2}+\cdots+f_{t}=n}\binom{n}{f_{1},f_{2},\cdots,f_{t}}\alpha_{1}^{f_{1}}\alpha_{2}^{f_{2}}\cdots\alpha_{t}^{f_{t}}\$$

$$\text{medmath}{W}_{\left(\genfrac{}{}{0ex}{}{n-1}{f_1,\cdots ,f_k-1\cdots ,f_t}\right)\to \left(\genfrac{}{}{0ex}{}{n}{f_1,\cdots ,f_k,\cdots ,f_t}\right)}=\sum _{j=1}^{k-1}\left(\genfrac{}{}{0ex}{}{n-1}{f_1,\cdots ,f_j-1,\cdots ,f_t}\right)$$

$$P(n,f_1,f_2,f_3)=\left(\genfrac{}{}{0ex}{}{6}{3,1,2}\right)=\frac{6!}{3!2!1!}=\frac{6\cdot 5\cdot 4\cdot 3\cdot 2\cdot 1}{3\cdot 2\cdot 1\cdot 1\cdot 2\cdot 1}=60$$

Full text

I Introduction

Efficient storage and communication of ever growing digital content has kept researches busy designing new techniques for compressing this information efficiently. The ubiquitous presence of multimedia enabled communication devices produce enormous amount of data to be stored and communicated efficiently. Research in bio-informatics has introduced whole new dimension where large amount of DNA information has to be processed. The handling, storage and transmission of these data is already beyond current available resources.

The science of data compression originated with Shannon’s famous paper on information theory [1] where he introduced entropy as measure of the information. Considering the source generating discrete set of symbols each of which is independent and identically distributed random variable, he showed that it is not possible to compress the source coding rate (average bits per symbol) below the Shannon entropy of the source and be able to reconstruct the original message without the loss of information. To define entropy suppose that a message is made up of $t$ alphabets in set $A={\alpha_{1},\alpha_{2},\cdots,\alpha_{t}}$. If the alphabet frequencies are ${f_{1},f_{2},\cdots,f_{t}}$ respectively with message length $n=f_{1}+f_{2}+\cdots+f_{t}$ then Shannon entropy of the message is

[TABLE]

where $p_{i}=f_{i}/n$

Shannon, along with Fano, invented a variable length prefix coding scheme to compress data by assigning shorter codes to more frequent symbols and longer codes to rarely occurring symbols. A slightly improved version was presented by Huffman [2] that constructed the code assignment tree from leaves to the root, in contrast to the Shannon-Fano coding tree that built from root to the leaves. Huffman encoding is sometimes used directly to compress the data while more often it is used as final step to compress symbols after other compression algorithm has taken place. Tools like PKZIP, JPEG, MP3, BZIP2 make use of Huffman coding. For efficient compression, in the original Huffman encoding, it is required that the character probabilities be known at the start of the compression process. This can come either from an accurate estimate of the information source or from statistical pre-processing of the data before putting it to compression algorithm. This encoding approaches the Shannon limit when the data size, $n$, is large and the character probabilities are powers of $1/2$. It is least efficient when the data consists of binary symbols since each Huffman symbol is integer number of bits in size. Such is the problem with other prefix codes such as Golomb, Elias and unary coding as only rarely symbol probabilities obey this condition in practice.

To solve the problem of inefficient Huffman encoding due to integer bits another statistical based data compression scheme known as arithmetic encoding [3]was introduced. Though theoretically promising, its use remained initially limited because of the patent issues. It encodes all the data into one fractional number, represented by bits no more than the Shannon limit. It’s beauty is that it works even for binary alphabet, assigning fractional bits to symbol. One drawback is that the complete message is processed with no inter-alphabet boundaries. Since digital computers are more comfortable working with integers, the fractional ranges have to be frequently converted to integer ranges in practice. Some of these limitations were solved by range coding [4]. However these are still costly because of multiplication operations.

The author was unable to find a statistical encoding scheme that mentions or uses combinatorics to compress the data. Previously the source statistics are defined in terms of the symbol probabilities, the fractional numbers. Our point of view is to consider a message as one permutation of its constituent symbols. Given the symbols and their frequencies in the message, we can represent the input data with a small integer denoting the lexicographic position of the message in list of permutations of its symbols. This index takes far fewer bits than the original message and achieves perfect compression. Contribution of this paper is also to develop an logarithm that calculates the dictionary index of a message as permutation of its constituent symbols with given frequencies for message encoding and vice versa for decoding. The index calculated this way remains below the Shannon entropy of the message.

The paper presents a data compression technique that to the best knowledge of the author has not been presented before. The message compression is achieved by combinadics, a mixed radix numbering system. The algorithm treats the input message as one permutation of its constituent symbols. Like both Huffman encoding and Arithmetic encoding an alphabet frequency list is needed for decompression of the message. The compression however does not need pre-processing or statistical modeling of the information source. Rather the frequency of each letter encountered at the time of compression is incremented and the compression process continues at the mean time. Compression has linear time complexity $\mathbf{O}(n)$ while decompression is $\mathbf{O}(t\cdot n)$.

In this paper $n$ represents the message length, $t$ represents the alphabet size. The set of alphabets $A=(\alpha_{1},\alpha_{2},\cdots,\alpha_{t})$ is replaced with set $(0,1,2,\cdots,t-1)$ to use with message encoding examples. Each occurrence of an alphabet is termed as a message symbol. There are $n$ symbols in $n$ length message.

II Compression

The data compression is treated as combinatorics problem in this paper. When alphabet frequences in the message are not known then for a message length $n$ and alphabet size $t$ the number of all possible messages is $n^{t}$. For example if message length is six $(n=6)$ and three alphabets $(t=3)$ are used from set $A=(0,1,2)$, the number of possible messages would be $3^{5}=243$. The binary bits required to represent the message can be calulated as $\log_{2}(3^{5})=9.5098$. With 6 symbols long message, the average information per symbol is $9.5098/6=1.5850$ bits/symbol. The message can not be decodeably represented with fewer than 10 bits if no other statistics information is available.

If the probabilites of the alphabets are known in advance then the number of bits requiered to span all possible messages would be much lower as given by Shannon source coding limit. If the alphabets $(0,1,2)$ appear in the message with probabilities ${3/6,1/6,2/6}$ respectively, the optimum symbol size by Shannon source coding theorem in Equation-I

[TABLE]

Reducing the average information per symbol to the Shannon limit is the goal of every compression method.

Compression ratio is a measure of efficiency for a compression method. It is the ratio

[TABLE]

A closely related measure quanifying the percentage amount of space saved by the compression method is

[TABLE]

The compression ratio in our case is $1.5850/1.4591=1.0863$ percent or 7.94% reduction in the amount of space needed to store the mesage.

This paper treats the compression as a combinatorial problem. We evaluate the total permutations of the message given the alphabets used and their respective occurance frequencies in the messsage. The same statistical information is used by Shannon, Huffman and arithmetic encoding as alphabet probability of occurance in the message. Instead of working with alphabet probabilities we work with the alphabet frequencies. Taking the previous example further, calculate the number of permutations of 6-symbol message formed by three alphabets, appearing in the message with frequencies three, one and two say. The permutations calculation is the same as combinatorial problem of placing 6 balls: say three red, two green and one blue, in 6 urns. The enumerations are calculated using the multinomial formula

[TABLE]

where

[TABLE]

The number of permutations for the example using Equation-II would be

[TABLE]

A unique ID can be attributed to each of these 60 permutations using $(\log_{2}(60))=5.9069$. Because a binary number can not have fractional number of bits therefore 6-bits would suffice to uniquely identify each permutation. As a comparasion, this is much smaller than the Shannon length $6\times 1.4591=8.7546=$ 9-bits. It achieves the space saving of ratio $1/1.5850=63.09\%$.

It essentially means that we need a mechanism to establish bijection between the message permutation and its order in the list of all the permutations of given alphabet with given frequency counts. That number represented in biary format would correspond to the origional message. The best selection would be the entry number in the dictionary of all permutations of the origional message. Algorithms exist that can find the next permutation in lexicographic order given the present permutation. However, to the best of the author’s knowledge, no algorithm has been presented in literature that can find the permutation lexicographic position given the message or vice versa. This paper presents origional algorithm to map a message to and from its lexicographic order with simple method. A mixed radix number system consisting of multinomial coefficients is developed to perform this mapping.

The reader might get skeptic of getting a bit representation smaller than the Shannon limit in our toy example. The combinatorial compression would approaches the Shannon limit as $n$ approaches $\infty$ for uniform alphabet frequencies but remaining always below the Shannon limit.

For a comparasion the Shannon formula can be re-arranged to enumerate the corresponding patterns for given alphabet probabilities $p_{1},p_{2},\cdots,p_{t}$.

[TABLE]

As $H$ is the average length per symbol, $n\cdot H$ would give the total message length and $2^{nH}$ will give the total number of possible messages with $nH$ bits.

[TABLE]

Using the relation between alphabet frequency and its probability $p_{i}=f_{i}/n$

[TABLE]

It is trivial to prove that Equation-14 is always greater than Equation-II for all $n$ and $f_{i}$.

Lemma 1**.**

[TABLE]

for all $n$ and $f_{i}$. Where $\sum_{i=1}^{t}f_{i}=n$

• Proof

The inequality can be rearranged to the form

[TABLE]

Clearly (17) is true because right hand side is a term in the expansion of left hand side. Q.E.D.

Theorem 2**.**

Let a source stream of independent identically distributed random variables of length $n$, of which each symbol comes from the set $A=\left\{\alpha_{1},\alpha_{2},\cdots,\alpha_{t}\right\}$ with frequencies $(f_{1},f_{2},\cdots,f_{t})$ respectivly. It is possible to compress the origional message to $\log_{2}\binom{n}{f_{1},f_{2},\cdots,f_{t}}<nH$ bits with lossless reconstruction for all $n$ and $f_{i}$.

• Proof

Given the alphabet frequencies $(f_{1},f_{2},\cdots,f_{t})$, due to basic combinatorics there can not be more that $\binom{n}{f_{1},f_{2},\cdots,f_{t}}$ permutations of the to-be-compressed-message. If a bijection is established between each permutation of the message and a number representing the lexicographic position of the permutation, no more than $\log_{2}\binom{n}{f_{1},f_{2},\cdots,f_{t}}$ would be needed to represent any of the permutations. This lexicographic position along with alphabet frequencies would accurately regenerate the origional message. Due to Lemma-1 $1/n\log_{2}\binom{n}{f_{1},f_{2},\cdots,f_{t}}$ is always smaller than $H$. The difference is more prominent for small $n$. It is also substantial when one or two alphabets occur more frequently than others. For large $n$ and uniform $f_{i}$, the lemma approximates equality.

III Data Compression for Binary Source

The data compression for combianatoric encoding is first explained by a binary source with alphabet $\mathbf{A}=\left\{a=0,b=1\right\}$. Figure-4 shows a few examples of combinatorial encoding. The figure proves the bijection of message permutations and its lexicographic order using the Pascal’s triangle. Each node in the directed acyclic graph can be reached from root in $\binom{n}{f_{0},f_{1}}a^{f_{0}}b^{f_{1}}$ ways, where $n$ is the level of the node in the graph and $f_{0}$ and $f_{1}$ are the number of left and right turn needed to reach the node, respectively. Here the binomial coefficient is written in more generic form using $\binom{n}{k}=\binom{n}{k,n-k}$. It will help extend the discussion to multinomial coefficients. The edge wigths between two nodes are given by the eqation

[TABLE]

Each path traversed with above edge weights will give a unique path cost, refered to here as lexicographic number of the path.

Compression is achieved by calculating the lexicographic order of the input message. For example if the input message is [11011100101], its lexicographic order or the compressed representation can be computed using Algorithm-1:

[TABLE]

The number 251 can be represented with 8-bits, a saving of three bits. For decompression we will need the information $(L_{i}=251,~{}f_{1}=7,~{}f_{0}=4)$.

For easy hand calculations the above algorithm can be summarized as follows:

• •

Index message bits from zero starting at Least Significant Bit (LSB) of the message. Let $i$ be the bit index.

• •

Start a counter $j$ from LSB side and increment wheneveve a ONE is encountered. If LSB start with ONE then first value of $j$ would be one.

• •

Compute lexicographic order of the message as $\sum_{i=0}^{n-1}b_{i}\binom{i}{j}$, where $b_{i}$ is the $i^{th}$ bit and $\binom{i}{j}=0$ for $i<j$

It should be noted lexicographic order remains unchanged for any number of zeros on left of message MSB. It helps in representing the bits in more compact form. However zeros to the right of LSB are significant and they effect the lexicographic order.

For lossless decompression of the message we need the lexicographic order of the message with given frequencies of the alphabets.

Using Algorithm-2 the compressed data $(L_{i}=251,~{}f_{1}=7,~{}f_{0}=4)$ can be decompressed as follows.

[TABLE]

IV Data Compression For t-Alphabet Source

The data compression for multi alphabet source is achieved on the same principes as used by the binary source. An acyclic directed graph can be constructed in this case also. The major difference is that there are now $t$ outwards paths from each node, connecting to next $t$ nodes. Edge weights are weighted multinomial coeffiecients instead of binomial coefficients. If we have the alphabet as $(0,1,2,\cdots,t-1)$, then each node in the directed graph has the data string $\binom{n}{f_{1},\cdots,f_{t}}\alpha_{1}^{f_{1}}\cdots\alpha_{t}^{f_{t}}$. Figure-3 shows the graph for three alphabet case.

The nodes on each level have labels

[TABLE]

The weight of the directed edge connecting two nodes ${}_{\binom{n-1}{f_{1},\cdots,f_{k}-1\cdots,f_{t}}\alpha_{1}^{f_{1}}\cdots\alpha_{k}^{f_{k}-1}\cdots\alpha_{t}^{f_{t}}\rightarrow\binom{n}{f_{1},\cdots,f_{k},\cdots,f_{t}}\alpha_{1}^{f_{1}}\cdots\alpha_{k}^{f_{k}}\cdots\alpha_{t}^{f_{t}}}$ is given by

[TABLE]

IV-A *Encoding Example for Tertiary Alphabet: *B A N A N A

All possible permutations of the letters in BANANA are

[TABLE]

So there are merely 60 different permutations. If we consider dictionary entry number of each of these permutations, any permutation can be represented by a binary number of no more than 6-bits length. That corresponds to one bit per symbol which is considerable compression as compared to 2-bits/symbol required for ordinary binary encoding.

V Conclusion

An entropy encoding technique based on permutation index representation of the message string is presented. The encoding and decoding algorithms work by assigning integer weights to symbol place value based on multinational coefficients. Encoding process use only addition and subtraction and unlike arithmetic or range encoding require no multiplication or division operations. With pre-calculated binomial or multinomial coefficients the method achieves the compression accuracy of arithmetic coding at the operation speed of Huffman coding. The binary messages need binomial coefficients as digit place values while multinomial coefficients are used for higher cardinality alphabet sets.