Compression with the tudocomp Framework

TL;DR

The paper introduces tudocomp, a framework for implementing and comparing text compression algorithms, demonstrated through case studies on new Lempel-Ziv-based algorithms optimized for highly repetitive texts.

Contribution

The paper presents tudocomp, a flexible framework that simplifies development and benchmarking of compression algorithms, along with two novel algorithms tailored for repetitive data.

Findings

The framework enables efficient implementation and comparison of compression algorithms.

The new algorithms perform well on highly repetitive texts.

Tudocomp facilitates rapid development of compression methods.

Abstract

We present a framework facilitating the implementation and comparison of text compression algorithms. We evaluate its features by a case study on two novel compression algorithms based on the Lempel-Ziv compression schemes that perform well on highly repetitive texts.

Equations

$$\mathsf{BWT}\left[j\right]=\{\begin{array}{ll}T[n]& \text{ifleavevmodenobreak }\mathsf{SA}\left[j\right]=1,\\ T[\mathsf{SA}\left[j\right]-1]& \text{otherwise, }\end{array}$$

$$\ell :=\{\begin{array}{l}1\text{leavevmodenobreak ifleavevmodenobreak }T[i..j+1]\text{leavevmodenobreak is a unique substring ofleavevmodenobreak }T,\text{otherwise:}\\ 1+\max \left\{\ell \in {\mathbb{N}}_0\mid \forall k=1,\dots ,\ell \nexists c\in \Sigma \setminus \left\{T[j+k+1]\right\}:T[i..j+k]c\text{leavevmodenobreak occurs inleavevmodenobreak }T\right\},\end{array}$$

Full text

1 Introduction

Engineering novel compression algorithms is a relevant topic, shown by recent approaches like bc-zip [7], Brotli [1], or Zstandard111https://github.com/facebook/zstd. Engineers of data compression algorithms face the fact that it is cumbersome (a) to build a new compression program from scratch, and (b) to evaluate and benchmark a compression algorithm against other algorithms objectively. We present the highly modular compression framework tudocomp that addresses both problems. To tackle problem (a), tudocomp contains standard techniques like VByte [28], Elias-$\gamma$/$\delta$, or Huffman coding. To tackle problem (b), it provides automatic testing and benchmarking against external programs and implemented standard compressors like Lempel-Ziv compressors. As a case study, we present the two novel compression algorithms lcpcomp and LZ78U, their implementations in tudocomp, and their evaluations with tudocomp. lcpcomp is based on Lempel-Ziv 77, substituting greedily the longest remaining repeated substring. LZ78U is based on Lempel-Ziv 78, with the main difference that it allows a factor to introduce multiple new characters.

1.1 Related Work

There are many222e.g., http://www.squeezechart.com or http://www.maximumcompression.com compression benchmark websites measuring compression programs on a given test corpus. Although the compression ratio of a novel compression program can be compared with the ratios of the programs listed on these websites, we cannot infer which program runs faster or more memory efficiently if these programs have not been compiled and run on the same machine. Efforts in facilitating this kind of comparison have been made by wrapping the source code of different compression algorithms in a single executable that benchmarks the algorithms on the same machine with the same compile flags. Examples include lzbench333https://github.com/inikep/lzbench and Squash444https://quixdb.github.io/squash-benchmark.

Considering frameworks aiming at easing the comparison and implementation of new compression algorithms, we are only aware of the C++98 library ExCom [14]. The library contains a collection of compression algorithms. These algorithms can be used as components for a compression pipeline. However, ExCom does not provide the same flexibility as we had in mind; it provides only character-wise pipelines, i.e., it does no bitwise transmission of data. Its design does not use meta-programming features; a header-only library has more potential for optimization since the compiler can inline header-implemented (possibly performance critical) functions easily.

A broader focus is set in Giuseppe Ottaviano’s succinct library [13] and Simon Gog’s Succinct Data Structure Library 2.0 (SDSL) [12]. These two libraries provide integer coders and helper functions for working on the bit level.

1.2 Our Results/Approach

Our lossless compression framework tudocomp aims at supporting and facilitating the implementation of novel compression algorithms. The philosophy behind tudocomp is to support building a pipeline of modules that transforms an input to a compressed binary output. This pipeline has to be flexible: appending, exchanging and removing a module in the pipeline in a plug-and-play manner is in the main focus of the design of tudocomp. Even a module itself can be refined into submodules.

To this end, tudocomp is written in modern C++14. On the one hand, the language allows us to write compile time optimized code due to its meta programming paradigm. On the other hand, its fine-grained memory management mechanisms support controlling and monitoring the memory footprint in detail. We provide a tutorial, an exhaustive documentation of the API, and the source code at http://tudocomp.org with the permissive Apache License 2.0 to encourage developers to use and foster the framework.

In order to demonstrate its usefulness, we added reference implementations of common compression and encoding schemes (see Section 2). On top of that, we present two novel algorithms (see Section 3) which we have implemented in our framework. We give a detailed evaluation of these algorithms in Section 4, thereby exposing the benchmarking and the visualization tools of tudocomp.

2 Description of the tudocomp Framework

On the topmost abstraction level, tudocomp defines the abstract types Compressor and Coder. A compressor transforms an input into an output so that the input can be losslessly restored from the output by the corresponding decompressor. A coder takes an elementary data type like a character and writes it to a compressed bit sequence. As with compressors, each coder is accompanied by a decoder taking care of restoring the original data from its compressed bit sequence. By design, a coder can take the role of a compressor, but a compressor may not be suitable as a coder (e.g., a compressor that needs random access on the whole input).

tudocomp provides implementations of the compressors and the coders shown in the tables below. Each compressor and coder gets an identifier (right column of each table).

[TABLE]

[TABLE]

[TABLE]

The behavior of a compressor or coder can be modified by passing different parameters. A parameter can be an elementary data type like an integer, but it can also be an instance of a class that specifies certain subtasks like integer coding. For instance, the compressor lzss_lcp(threshold, coder) takes an integer threshold and a coder (to code an LZ77 factor) as parameters. The coder is supplied as a parameter such that the compressor can call the coder directly (instead of alternatively piping the output of lzss_lcp to a coder).

The support of class parameters eases the deployment of the design pattern strategy [11]. A strategy determines what algorithm or data structure is used to achieve a compressor-specific task.

Library and Command Line Tool

tudocomp consists of two major components: a standalone compression library and a command line tool tdc. The library contains the core interfaces and implementations of the aforementioned compressors and coders. The tool tdc exposes the library’s functionality in form of an executable that can run compressors directly on the command line. It allows the user to select a compressor by its identifier and to pass parameters to it, i.e., the user can specify the exact compression strategy at runtime.

Example

For instance, the LZ78U compressor (Section 3.3) expects a compression strategy, an integer coder, and an integer variable specifying a threshold. Its strategy can define parameters by itself, like which string coder to use. A valid call is ./tdc -a ’lz78u(coder = bit, comp = buffering(string_coder = huff), threshold = 3)’ input.txt -o output.tdc, where tdc compresses the file input.txt and stores the compressed bit sequence in the file output.tdc. To this end, it uses the compressor lz78u parametrized by the coder bit for integer values, by the compression strategy buffering with huff to code strings, and by a threshold value of $3$. Note that coder and string_coder are parameters for two independently selectable coders. When selecting a coder we have to pay attention that a static entropy coder like huff needs to parse its input in advance (to generate a codeword for each occurring character). To this end, we can only apply the coder huff with a compression strategy that buffers the output (for lz78u this strategy is called buffering). To stream the output (i.e., the opposite of buffering the complete output in RAM), we can use the alternative strategy streaming. This strategy also requires a coder, but contrary to the buffering strategy, that coder does not need to look at the complete output (e.g., universal codes like gamma).

In this fashion, we can build more sophisticated compression pipelines like lzma applying different coders for literals, pointers, and lengths. Each coder is unaware of the other coders, as if every coder was processing an independent stream.

Decompression

After compressing an input using a certain compression strategy, the tool adds a header to the compressed file so that it can decompress it without the need for specifying the compression strategy again. However, this behavior can be overruled by explicitly specifying a decompression strategy, e.g., in order to test different decompression strategies.

Helper classes

tudocomp provides several classes for easing common tasks when engineering a new compression algorithm, like the computation of $\mathop{}\mathopen{}\mathsf{SA}\mathopen{}$, $\mathop{}\mathopen{}\mathsf{ISA}\mathopen{}$ or $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}$. tudocomp generates $\mathop{}\mathopen{}\mathsf{SA}\mathopen{}$ with divsufsort555https://github.com/y-256/libdivsufsort, and $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}$ with the $\Phi$-algorithm [17]. The arrays $\mathop{}\mathopen{}\mathsf{SA}\mathopen{}$, $\mathop{}\mathopen{}\mathsf{ISA}\mathopen{}$, and $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}$ can be stored in plain arrays or in packed arrays with a bit width of $\left\lceil\lg n\right\rceil$ (where $n$ is the length of the input text), i.e., in a bit-compact representation. We provide the modes plain, compressed, and delayed to describe when/whether a data structure should be stored in a bit-compact representation: In plain mode, all data structures are stored in plain arrays; in compressed mode, all data structures are built in a bit-compact representation. In delayed mode, tudocomp first builds a data structure $A$ in a plain array; when all other data structures are built whose constructions depended on $A$, $A$ gets transformed into a bit-compact representation. While direct and compressed are the fastest or the memory-friendliest modes, respectively, the data structures produced by delayed are the same as compressed, though delayed is faster than compressed.

If more elaborated algorithms are desired (e.g., for producing compressed data structures like the compressed suffix array), it is easy to use tudocomp in conjunction with SDSL for which we provide an easy binding.

Combining streaming and offline approaches

A compressor can stream its input (online approach) or request the input to be loaded into memory (offline approach). Compressors can be chained to build a pipeline of multiple compression modules, like as in Figure 1.

2.1 Example Implementation of a Compressor

Finally, the compressor bwt can be used as part of a pipeline to achieve good compression quality: Given a move-to-front compressor mtf and a Huffman coder huff, we can build a chain bwt:rle:mtf:encode(huff). The compressor encode is a wrapper that turns a coder into a compressor. The last code fragment (c) on the left shows the calls of this pipeline and a call of bwt only. Using stat, we measure the file sizes (in bytes) of the input pc-english (see Section 4) and both outputs.

2.2 Specific Features

tudocomp excels with the following additional properties:

Few Build Requirements

To deploy tudocomp, the build management software cmake, the version control system git, Python 3, and a C++14 compiler are required. cmake automatically downloads and builds other third-party software components like the SDSL. We tested the build process on Unix-like build environments, namely Debian Jessie, Ubuntu Xenial, Arch Linux 2016, and the Ubuntu shell on Windows 10.

Unit Tests

tudocomp offers semi-automatic unit tests. For a registered compressor, tudocomp can automatically generate test cases that check whether the compressor can compress and decompress a set of selected inputs successfully. These inputs include border cases like the empty string, a run of the same character, samples on various subranges in UTF-8, Fibonacci strings, Thue-Morse strings, and strings with a high number of runs [22]. These strings can be generated on-the-fly by tdc as an alternative input.

Type Inferences

The C++ standard does neither provide a syntax for constraining type parameters (like generic type bounding in Java) nor for querying properties of a class at runtime (i.e., reflection). To address this syntactic lack, we augment each class exposed to tdc and to the unit tests with a so-called type. A type is a string identifier. We expect that classes with the same type provide the same public methods. Types resemble interfaces of Java, but contrary to those, they are not subject to polymorphism. Common types in our framework are Compressor and Coder. The idea is that, given a compressor that accepts a Coder as a parameter, it should accept all classes of type Coder. To this end, each typed class is augmented with an identifier and a description of all parameters that the class accepts. All typed classes are exposed by the tool tdc that calls a typed class by its identifier with the described parameters. Types provide a uniform, but simple declaration of all parameters (e.g., integer values, or strategy classes). The aforementioned exemplaric call of lz78u at the beginning of Section 2 illustrates the uniform declaration of the parameters of a compressor.

Evaluation tools

To evaluate a compressor pipeline, tudocomp provides several tools to facilitate measuring the compression ratio, the running time, and the memory consumption. By adding --stats to the parameters of tdc, the tool monitors these measurement parameters: It additionally tracks the running time and the memory consumption of the data structures in all phases. A phase is a self-defined code division like a pre-processing phase, or an encoding phase. Each phase can collect self-defined statistics like the number of generated factors. All measured data is collected in a JSON file that can be visualized by the web application found at http://tudocomp.org/charter. An example is given in Figure 6.

In addition, we have a command line comparison tool called compare.py that runs a predefined set of compression programs (that can be tudocomp compressors or external compression programs). Its primary usage is to compare tudocomp compression algorithms with external compression programs. It monitors the memory usage with the tool valgrind --tool=massif --pages-as-heap=yes. This tool is significantly slower than running tdc with --stats.

3 New Compression Algorithms

With the aid of tudocomp, it is easy to implement new compression algorithms. We demonstrate this by introducing two novel compression algorithms: lcpcomp and LZ78U. To this end, we first recall some definitions.

3.1 Theoretical Background

Let $\Sigma$ denote an integer alphabet of size $\sigma=\left|\Sigma\right|\leq n^{\mathop{}\mathopen{}\mathcal{O}\mathopen{}(1)}$ for a natural number $n$. We call an element $T\in\Sigma^{*}$ a string. The empty string is $\epsilon$ with $\left|\epsilon\right|=0$. Given $x,y,z\in\Sigma^{*}$ with $T=xyz$, then $x$, $y$ and $z$ are called a prefix, substring and suffix of $T$, respectively. We call $T[i..]$ the $i$-th suffix of $T$, and denote a substring $T[i]\cdots T[j]$ with $T[i..j]$.

For the rest of the article, we take a string $T$ of length $n$. We assume that $T[n]$ is a special character \texttt{\}\notin\Sigma$smaller than all characters of$\Sigma$so that no suffix of$T$is a prefix of another suffix of$T$.

$\mathop{}\mathopen{}\mathsf{SA}\mathopen{}$ and $\mathop{}\mathopen{}\mathsf{ISA}\mathopen{}$ denote the suffix array [21] and the inverse suffix array of $T$, respectively. $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[2..n\right]$ is an array such that $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right]$ is the length of the longest common prefix of the lexicographically $i$-th smallest suffix with its lexicographic predecessor for $i=2,\ldots,n$. The BWT [3] of $T$ is the string $\mathop{}\mathopen{}\mathsf{BWT}\mathopen{}$ with

[TABLE]

for $1\leq j\leq n$. The arrays $\mathop{}\mathopen{}\mathsf{SA}\mathopen{}$, $\mathop{}\mathopen{}\mathsf{ISA}\mathopen{}$, $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}$ and $\mathop{}\mathopen{}\mathsf{BWT}\mathopen{}$ can be constructed in time linear to the number of characters of $T$ [18].

As a running example, we take the text T:={\tt aaababaaabaababa\}$. The arrays$\mathop{}\mathopen{}\mathsf{SA}\mathopen{}$,$\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}$and$\mathop{}\mathopen{}\mathsf{BWT}\mathopen{}$ of this example text are shown in Figure 2.

Given a bit vector $B$ with length $\left|B\right|$, the operation $B.\operatorname{rank}_{1}(i)$ counts the number of ‘1’-bits in $B[1..i]$, and the operation $B.\operatorname{select}_{1}(i)$ yields the position of the $i$-th ‘1’ in $B$.

There are data structures [15, 4] that can answer $\operatorname{rank}{}$ and $\operatorname{select}{}$ queries on $B$ in constant time, respectively. Each of them uses $\mathop{}\mathopen{}o\mathopen{}(\left|B\right|)$ additional bits of space, and both can be built in $\mathop{}\mathopen{}\mathcal{O}\mathopen{}(\left|B\right|)$ time.

The suffix trie of $T$ is the trie of all suffixes of $T$. The suffix tree [26] of $T$, denoted by $\mathsf{ST}$, is the tree obtained by compacting the suffix trie of $T$. It has $n$ leaves and at most $n$ internal nodes. The string stored in an edge $e$ is called the edge label of $e$, and denoted by $\lambda(e)$. The string depth of a node $v$ is the length of the concatenation of all edge labels on the path from the root to $v$. The leaf corresponding to the $i$-th suffix is labeled with $i$.

Each node of the suffix tree is uniquely identified by its pre-order number. We can store the suffix tree topology in a bit vector (e.g., DFUDS [2] or BP [15, 23]) such that $\operatorname{rank}$ and $\operatorname{select}$ queries enable us to address a node by its pre-order number in constant time. If the context is clear, we implicitly convert an $\mathsf{ST}$ node to its pre-order number, and vice versa. We will use the following constant time operations on the suffix tree:

• •

$\mathop{}\mathopen{}\operatorname{parent}\mathopen{}(v)$ selects the parent of the node $v$,

• •

$\mathop{}\mathopen{}\operatorname{level-anc}\mathopen{}(\ell,d)$ selects the ancestor of the leaf $\ell$ at depth $d$ (level ancestor query), and

• •

$\mathop{}\mathopen{}\operatorname{leaf-select}\mathopen{}(i)$ selects the $i$-th leaf (in lexicographic order).

A factorization of $T$ of size $z$ partitions $T$ into $z$ substrings $T=F_{1}\cdots F_{z}$. These substrings are called factors. In particular, we have:

Definition 3.1**.**

A factorization $F_{1}\cdots F_{z}=T$ is called the Lempel-Ziv-77 (LZ77) factorization [30] of $T$ with a threshold $t\geq 1$ iff $F_{x}$ is either the longest substring of length at least $t$ occurring at least twice in $F_{1}\cdots F_{x}$, or, if such a substring does not exist, a single character. We merge successive occurrences of the latter type of factors to a single factor and call it a remaining substring.

The usual definition of the LZ77 factorization fixes $t=1$. We introduced the version with a threshold to make the comparison with lcpcomp (Section 3.2) fairer.

Definition 3.2**.**

A factorization $F_{1}\cdots F_{z}=T$ is called the Lempel-Ziv-78 (LZ78) factorization [31] of $T$ iff $F_{x}=F_{y}\cdot c$ with $F_{y}=\operatorname{argmax}_{S\in\left\{F_{y}:y<x\right\}\cup\left\{\epsilon\right\}}\left|S\right|$ and $c\in\Sigma$ for all $1\leq x\leq z$.

3.2 lcpcomp

The idea of lcpcomp is to search for long repeated substrings and substitute one of their occurrences with a reference to the other. Large values in the LCP-array indicate such long repeated substrings. There are two major differences to the LZ77 compression scheme: (1) while LZ77 only allows back-references, lcpcomp allows both back and forward references; and (2) LZ77 factorizes $T$ greedily from left to right, whereas lcpcomp makes substitutions at arbitrary positions in the text, greedily chosen such that the number of substituted characters is maximized. This process is repeated until all remaining repeated substrings are shorter than a threshold $t$. On termination, lcpcomp has generated a factorization $T=F_{1}\cdots F_{z}$, where each $F_{j}$ is either a remaining substring, or a reference $(i,\ell)$ with the intended meaning “copy $\ell$ characters from position $i$” (see Figure 3 for an example).

Algorithm

The LCP array stores the longest common prefix of two lexicographically neighboring suffixes. The largest entries in the LCP array correspond to the longest substrings of the text that have at least two occurrences. Given a suffix $T[\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]..]$ whose entry $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right]$ is maximal among all other values in $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}$, we know that $T[\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]..\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]+\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right]-1]=T[\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i-1\right]..\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i-1\right]+\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right]-1]$, i.e., we can substitute $T[\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]..\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]+\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right]-1]$ with the reference $(\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i-1\right],\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right])$. In order to find a suffix whose $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}$ entry is maximal, we need a data structure that maintains suffixes ordered by their corresponding LCP values. We use a maximum heap for this task. To this end, the heap stores suffix array indices whose keys are their LCP values (i.e., insert $i$ with key $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right]$, $2\leq i\leq n$). The heap stores only those indices whose keys are at least $t$.

While the heap is not empty, we do the following:

Remove the maximum from the heap; let $i$ be its value. 2. 2.

Report the reference $(\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i-1\right],\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right])$ and the position $\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]$ as a triplet $(\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i-1\right],$ $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right],$ $\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right])$. 3. 3.

For every $1\leq k\leq\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right]-1$, remove the entry $\mathop{}\mathopen{}\mathsf{ISA}\mathopen{}\left[{\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]+k}\right]$ from the heap (as these positions are covered by the reported reference). 4. 4.

Decrease the keys of all entries $j$ with $\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]-\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right]\leq\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[j\right]<\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]$ to $\min(\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[j\right],\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]-\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[j\right])$. (If a key becomes smaller than $t$, remove the element from the heap.) By doing so, we prevent the substitution of a substring of $T[\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]..\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]+\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right]-1]$ at a later time.

As an invariant, the key $\ell$ of a suffix array index $i$ stored in the heap will always be the maximal number of characters such that $T[i..i+\ell-1]$ occurs at least twice in the remaining text.

The reported triplets are collected in a list. To compute the final output, we sort the triplets by their third component (storing the starting position of the substring substituted by the reference stored in the first two components). We then scan simultaneously over the list and the text to generate the output. Figure 4 demonstrates how the lcpcomp factorization of the running example is done step-by-step.

The code on the left implements the compression strategy of lcpcomp that uses a maximum heap. We transfered the code from the compressor class to a strategy class since the lcpcomp compression scheme can be implemented in different ways. Each strategy receives a text. Its goal is to compute all factors (created by the create_factor method). In the depicted strategy, we use a maximum heap to find all factors. The heap is implemented in the class ArrayMaxHeap. An instance of that class stores an array $A$ of keys and an array heap maintaining (key-value)-pairs of the form $(A[i],i)$ with the order $(A[i],i)<(A[j],j):\Leftrightarrow A[i]<A[j]$. To access a specific element in the heap by its value, the class has an additional array storing the position of each value in the heap.

Although a reference $r$ can refer to a substring that has been substituted by another reference after the creation of $r$, in Lem. A.1 (Appendix), we show that it is always possible to restore the text.

Time Analysis

We insert at most $n$ values into the heap. No value is inserted again. Finally, we use the following lemma to get a running time of $\mathop{}\mathopen{}\mathcal{O}\mathopen{}(n\lg n)$ :

Lemma 3.3**.**

The key of a suffix array entry is decreased at most once.

Proof.

Let us denote the key of a value $i$ stored in the heap by $K[i]$. Assume that we have decreased the key $K[j]$ of some value $j$ stored in the heap after we have substituted a substring $T[i..i+\ell-1]$ with a reference. It holds that $K[j]=\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]-\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[j\right]-1>\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]-\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[j\right]-1-m\geq K[{\mathop{}\mathopen{}\mathsf{ISA}\mathopen{}\left[{\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[j\right]+m}\right]}]$ for all $m$ with $1\leq m\leq K[j]$, i.e., there is no suffix array entry that can decrease the key of $j$ again. ∎

3.2.1 Decompression

Decompressing lcpcomp-compressed data is harder than decompressing LZ77, since references in lcpcomp can refer to positions that have not yet been decoded. Figure 3 depicts the references built on our running example by arrows.

In order to cope with this problem, we add, for each position $i$ of the original text, a list $L_{i}$ storing the text positions waiting for this text position getting decompressed.

First, we determine the original text size (the compressor stores it as a VByte before the output of the factorization). Subsequently, while there is some compressed input, we do the following, using a counting variable $i$ as a cursor in the text that we are going to rebuild:

• •

If the input is a character $c$, we write $T[i]\leftarrow c$, and increment $i$ by one.

• •

If the input is a reference consisting of a position $s$ and a length $\ell$, we check whether $T[s+j]$ is already decoded, for each $j$ with $0\leq j\leq\ell-1$:

• –

If it is, then we can restore $T[i+j]\leftarrow T[s+j]$.

• –

Otherwise, we add $i+j$ to the list $L_{s+j}$.

In either case, we increment $i$ by $\ell$.

An additional procedure is needed to restore the text completely by processing the lists: On writing $T[i]\leftarrow c$ for some text position $i$ and some character $c$, we further write $T[j]\leftarrow T[i]$ for each $j$ stored in $L_{i}$ (if $L_{j}$ is not empty, we proceed recursively). Afterwards, we can delete $L_{i}$ since it will be no longer needed. The decompression runs in $\mathop{}\mathopen{}\mathcal{O}\mathopen{}(n)$ time, since we perform a linear scan over the decompressed text, and each text position is visited at most twice.

3.2.2 Implementation Improvements

In this section, we present an $\mathop{}\mathopen{}\mathcal{O}\mathopen{}(n)$ time compression algorithm alternative to the heap strategy and a practical improvement of the decompression strategy.

Compression

This strategy computes an array $A_{\ell}$ storing all suffix array entries $j$ with $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[j\right]=\ell$, for each $\ell$ with $t\leq\ell\leq\max_{k}\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[k\right]$. To compute the references, we sequentially scan the arrays in decreasing order, starting with the array that stores the suffixes with the maximum $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}$ value. On substituting a substring $T[\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]..\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]+\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right]-1]$ with the reference $(\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i-1\right],\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right])$, we update the $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}$ array (instead of updating the keys in the heap). We set $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[{\mathop{}\mathopen{}\mathsf{ISA}\mathopen{}\left[{\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]+k}\right]}\right]\leftarrow 0$ for every $1\leq k\leq\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right]-1$ (deletion), and $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[j\right]\leftarrow\min(\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[j\right],\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]-\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[j\right])$ for every $j$ with $\mathop{}\mathopen{}\mathsf{ISA}\mathopen{}\left[{\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]-\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right]}\right]\leq j<i$ (decrease key). Unlike the heap implementation, we do not delete an entry from the arrays. Instead, we look up the current $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}$ value of an element when we process it: Assume that we want to process $A_{\ell}[i]$. If $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[A_{\ell}[i\right]]=\ell$, then we proceed as above. Otherwise, we have updated the $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}$ value of the suffix starting at position $A_{\ell}[i]$ to the value $\ell^{\prime}:=\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[A_{\ell}[i\right]]<\ell$. In this case, we append $A_{\ell}[i]$ to $A_{\ell^{\prime}}$ (if $\ell^{\prime}<t$, we do nothing), and skip computing the reference for $A_{\ell}[i]$. By doing so, we either omit the substring $A_{\ell}[i]$ if $\ell^{\prime}<t$, or delay the processing of the value $A_{\ell}[i]$. A suffix array entry gets delayed at most once, analogously to Lemma 3.3. In total, the algorithm runs in $\mathop{}\mathopen{}\mathcal{O}\mathopen{}(n)$ time, since it performs basic arithmetic operations on each text position at most twice.

Decompression

We use a heuristic to improve the memory usage. The heuristic defers the creation of the lists $L_{i}$ storing the text positions that are waiting for the position $i$ to get decompressed. If a reference needs a substring that has not yet been decompressed, we store the reference in a list $L$. By doing so, we have reconstructed at least all substrings that have not been substituted by a reference during the compression. Subsequently, we try to decompress each reference stored in $L$, removing successfully decompressed references from $L$. If we repeat this step, more and more text positions can become restored. Clearly, after at most $n$ iterations, we would have restored the original text completely, but this would cost us $\mathop{}\mathopen{}\mathcal{O}\mathopen{}(n^{2})$ time. Instead, we run this algorithm only for a fixed number of times $b$. Afterwards, we mark all not yet decompressed positions in a bit vector $B$, and build a rank data structure on top of $B$. Next, we create a list $L_{i}$ for each marked text position $B.\operatorname{rank}(i)$ as in the original algorithm. The difference to the original algorithm is that $L_{i}$ now corresponds to $B.\operatorname{rank}(i)$. Finally, we run the original algorithm using the lists $L_{i}$ to restore the remaining characters.

3.3 LZ78U

A factorization $F_{1}\cdots F_{z}=T$ is called the LZ78U factorization of $T$ iff $F_{x}:=T[i..j+\ell]$ with $T[i..j]=\operatorname{argmax}_{S\in\left\{F_{y}:y<x\right\}\cup\left\{\epsilon\right\}}\left|S\right|$ and

[TABLE]

for all $1\leq x\leq z$. Informally, we enlarge an LZ78 factor representing a repeated substring $T[i..i+\ell-1]$ to $T[i..i+\ell]$ as long as the number of occurrences of $T[i..i+\ell-1]$ and $T[i..i+\ell]$ are the same.

Having the LZ78U factorization $F_{1},\ldots,F_{z}$ of $T$, we can output each factor $F_{x}$ as a tuple $(y,S_{x})$ such that $F_{x}=F_{y}S_{x}$, where $F_{y}$ ($0\leq y<x$) is the longest previous factor (set $F_{0}:=\epsilon$) that is a prefix of $F_{x}$, and $S_{x}$ is the suffix determined by the factorization. We call $y$ the referred index and $S_{x}$ the factor label of the $x$-th factor. Transforming the factors to this output induces a dictionary tree, called the LZ78U-tree, in which

• •

every node corresponds to a factor,

• •

the parent of a node $v$ corresponds to the referred index of $v$, and

• •

the edge between the node of the $x$-th factor and its parent is labeled with the factor label of the $x$-th factor.

Figure 5 shows a comparison to the LZ78-trie. By the definition of the factorizations, the LZ78-trie is a subtree of the suffix trie, whereas the LZ78U-tree is a subtree of the suffix tree. The latter can be seen by the fact that the suffix tree compacts the unary paths of the suffix trie. This fact is the foundation of the algorithm we present in the following. It builds the LZ78U-tree on top of the suffix tree. The algorithm is an easier computable variant of the LZ78 algorithms in [10, 19].

The Algorithm

The internal suffix tree nodes can be mapped to the pre-order numbers $[1..n]$ injectively by using rank/select data structures on the suffix tree topology. This allows us to use $n\lg n$ bits for storing a factor id in each internal suffix tree node. To this end, we create an array $R$ of $n\lg n$ bits. All elements of the array are initially set to zero. In order to compute the factorization, we scan the text from left to right. Given that we are at text position $i$, we locate the suffix tree leaf $\ell\leftarrow\mathop{}\mathopen{}\operatorname{leaf-select}\mathopen{}(i)$ corresponding to the $i$-th suffix. Let $p\leftarrow\mathop{}\mathopen{}\operatorname{parent}\mathopen{}(\ell)$ be $\ell$’s parent.

• •

If $R[p]\not=0$, then $p$ corresponds to a factor $F_{x}$. Let $c$ be the first character of the edge label $\lambda(p,\ell)$. The substring $F_{x}c$ occurs exactly once in $T$, otherwise $\ell$ would not be a leaf. Consequently, we output a factor consisting of the referred index $R[p]$ and the string label $c$. We further increment $i$ by the string depth of $p$ plus one.

• •

Otherwise, using level ancestor queries, we search for the highest node $v\leftarrow\mathop{}\mathopen{}\operatorname{level-anc}\mathopen{}(\ell,d)$ with $R[v]=0$ on the path between the root (exclusively) and $p$ (iterate over the depth $d$ starting with zero). We set $R[v]\leftarrow z+1$, where $z$ is the current number of computed factors. We output the referred index $R[\mathop{}\mathopen{}\operatorname{parent}\mathopen{}(v)]$ and the string $\lambda(\mathop{}\mathopen{}\operatorname{parent}\mathopen{}(v),v)$. Finally, we increment $i$ by the string depth of $v$.

Since level ancestor queries can be answered in constant time, we can compute a factor in time linear to its length. Summing over all factors we get linear time overall. We use $n\lg n+\left|\mathsf{ST}{}\right|$ bits of working space.

Improved Compression Ratio

To achieve an improved compression ratio, we factorize the factor labels: If $S_{x}$ is the label of the $x$-th factor $f_{x}$, then we factorize $S_{x}=G_{1}\cdots G_{m}$ with $G_{j}:=\operatorname{argmax}_{S\in\left\{F_{y}:y<x,\left|F_{y}\right|\geq t\right\}\cup\Sigma}\left|S\right|$ greedily chosen for ascending values of $j$ with $1\leq j\leq m$, with a threshold $t\geq 1$. By doing so, the string $S_{x}$ gets partitioned into characters and former factors longer than $t$. The factorization of $S_{x}$ is done in $\mathop{}\mathopen{}\mathcal{O}\mathopen{}(\left|S_{x}\right|)$ time by traversing the suffix tree with level ancestor queries, as above (the only difference is that we do not introduce a new factor to the LZ78U factorization).

4 Practical Evaluation

Table 1 shows the text collections used for the evaluation in the tudocomp benchmarks. We provide a tool that automatically downloads and prepares a superset of the collections used in this evaluation. The collections with the prefixes pc or pcr belong to the Pizza&Chili Corpus666http://pizzachili.dcc.uchile.cl. The Pizza&Chili Corpus is divided in a real text corpus (pc), and in a repetitive corpus (pcr). The collection hashtag is a tab-separated values file with five columns (integer values, a hashtag and a title) [9]. The collection tagme is a list of Wikipedia fragments777http://acube.di.unipi.it/tagme-dataset. Finally, we present two new text collections. The first collection, called wiki-all-vital, consists of the approx. $10000$ most vital Wikipedia articles888https://en.wikipedia.org/wiki/Wikipedia:Vital_articles/Expanded. We gathered all articles and processed them with the Wikipedia extractor of TANL [24] to convert each article into plain text. The second collection, named commoncrawl, is composed of a random subset of a web crawl999http://commoncrawl.org; this subset contains only the plain texts (i.e., without header and HTML tags) of web sites with ASCII characters.

Setup

The experiments were conducted on a machine with 32 GB of RAM, an Intel Xeon CPU E3-1271 v3 and a Samsung SSD 850 EVO 250GB. The operating system was a 64-bit version of Ubuntu Linux $14.04$ with the kernel version 3.13. We used a single execution thread for the experiments. The source code was compiled using the GNU compiler g++ 6.2.0 with the compile flags -O3 -march=native -DNDEBUG.

lcpcomp Strategies

For lcpcomp we use the heap strategy and the list decompression strategy described in Section 3.2. We call them heap and compact, respectively. The strategies described in Section 3.2.2 are called arrays (compression) and scan (decompression). The decompression strategy scan takes the number of scans $b$ as an argument. We encode the remaining substrings of lcpcomp with a static low entropy encoder sle. The coder is similar to a Huffman coder, but it additionally treats all $3$-grams of the remaining substrings as symbols of the input. We evaluated lcpcomp only with the coder sle, since it provided the best compression ratio. We produced $\mathop{}\mathopen{}\mathsf{SA}\mathopen{}$, $\mathop{}\mathopen{}\mathsf{ISA}\mathopen{}$ and $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}$ in the delayed mode.

LZ78U Implementation

We used the suffix tree implementation cst_sada of SDSL, since it provides all required operations like level ancestor queries.

Figure 6 visualizes the execution of lcpcomp with the strategy arrays in different phases for the collection pc-english. The figure is generated with the JSON output of tdc by the chart visualization application on our website http://tudocomp.org/charter. We loaded the text (200MiB), constructed $\mathop{}\mathopen{}\mathsf{SA}\mathopen{}$ (800MiB, 32 bits per entry), computed $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}$ (500MiB, 20-bits per entry), computed $\mathop{}\mathopen{}\mathsf{ISA}\mathopen{}$ (700MiB, 28 bits per entry), and shrunk $\mathop{}\mathopen{}\mathsf{SA}\mathopen{}$ to 700MiB. Summing these memory sizes gives a memory offset of 1.9GiB when lcpcomp started its actual factorization. The factorization is divided in LCP value ranges. After the factorization, the factors were sorted and finally transformed to a binary bit sequence by sle. Most of the running time was spent on building $\mathop{}\mathopen{}\mathsf{SA}\mathopen{}$, roughly 1GiB was spent for creating the lists $L_{i}$ containing the suffix array entries with an LCP value of $i$.

Finally, we compare the implemented algorithms of tudocomp with some classic compression programs like gzip by our comparison tool compare.py. The output of the tool is shown in Table 2. The compressor lzss_lcp computes the LZ77 factorization (Def. 3.1) by a variant of [16]. The compressor bwtzip is an alias for the compression pipeline bwt:rle:mtf:encode(huff) devised in Section 2.1. The programs bzip2 and gzip do not compress the highly repetitive collection pcr-cere as well as any of the tudocomp compressors (excluding the plain usage of a coder). Still, our algorithms are inferior to lzma -9 in the compression ratio and the decompression speed. The high memory consumption of LZ78U is mainly due to the usage of the compressed suffix tree.

5 Conclusions

The framework tudocomp consists of a compression library, the command line executable tdc, a comparison tool, and a visualization tool. The library provides classic compressors and standard coders to facilitate building a compressor, or constructing a complex compression pipeline. Since the library was built with a focus on high modularity, a compression pipeline does not have to get statically compiled. Instead, the tool tdc can assemble a compression pipeline at runtime. Such a pipeline, given as a parameter to tdc, can be adjusted in detail at runtime.

We demonstrated tudocomp’s capabilities with the implementation of two new compressors: lcpcomp, a variant of LZ77, and LZ78U, a variant of LZ78. Both new variants show better compression ratios than their respective originals, but have a higher memory consumption and also slower decompression times. Further research is needed to address these issues.

Future Research

The memory footprint of lcpcomp could be dropped by exchanging the array implementations of $\mathop{}\mathopen{}\mathsf{SA}\mathopen{}$, $\mathop{}\mathopen{}\mathsf{ISA}\mathopen{}$ and $\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}$ with compressed data structures like a compressed suffix array, an inverse suffix array sampling, and a permuted LCP (PLCP) array, respectively. We are currently investigating a variant that only observes the peaks in the PLCP array to compute the same output as lcpcomp. If the number of peaks is $\pi$, then this algorithm needs at most $\pi\lg n$ bits on top of $\mathop{}\mathopen{}\mathsf{SA}\mathopen{}$, $\mathop{}\mathopen{}\mathsf{ISA}\mathopen{}$ and the PLCP array.

We are optimistic that we can improve the compression ratio of our algorithms by adapting sophisticated approaches in how the factors are chosen [1, 7, 8] and how the factors are finally coded [5].

Appendix A Cycle-Free Lemma of lcpcomp

Lemma A.1**.**

The output of lcpcomp contains enough information to restore the original text.

Proof.

We want to show that the output is free of cycles, i.e., there is no text position $i$ for that $i\underbrace{\rightarrow\cdots\rightarrow}_{\text{cycle length}}i$ holds, where $\rightarrow$ is a relation on text positions such that $i\rightarrow j$ holds iff there is a substring $T[i^{\prime}..i^{\prime}+\ell-1]$ with $i\in[i^{\prime},i^{\prime}+\ell-1]$ that has been substituted by a reference $(j-i+i^{\prime},\ell)$. If the text is free of cycles, then each substituted text position can be restored by following a finite chain of references.

First, we show that is not possible to create cycles of length two. Assume that we substituted $T[\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]..\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]+\ell_{i}-1]$ with $(\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i-1\right],\ell_{i})$ for $t\leq\ell_{i}\leq\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right]$. The algorithm will not choose $T[\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i-1\right]+k..\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i-1\right]+k+\ell_{k}-1]$ for $0\leq k\leq\ell_{i}$ and $t\leq\ell_{k}\leq\mathop{}\mathopen{}\mathsf{LCP}\mathopen{}\left[i\right]-k$ to be substituted with $(\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]+k,\ell_{k})$, since $T[\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]+k..]>T[\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i-1\right]+k..]$ and therefore $\mathop{}\mathopen{}\mathsf{ISA}\mathopen{}\left[{\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i\right]+k}\right]>\mathop{}\mathopen{}\mathsf{ISA}\mathopen{}\left[{\mathop{}\mathopen{}\mathsf{SA}\mathopen{}\left[i-1\right]+k}\right]$. Finally, by the transitivity of the lexicographic order (i.e., the order induced by the suffix array), it is neither possible to produce larger cycles. ∎

Appendix B LZ78U Offline Algorithm

Instead of directly constructing the array $R$ that is necessary to determine the referred indices, we create a list $F$ storing the marked LZ-trie nodes, and a bit vector $B$ marking the internal nodes belonging to the LZ-tree. Initially, only the root node is marked in $B$. Let $i$, $p$ and $\ell$ be defined as in the above tree traversal. If $B[p]$ is set, then we append $\ell$ to $F$ and increment $i$ by one. Otherwise, by using level ancestor queries, we search for the highest node $v$ with $B[v]=0$ on the path between the root and $p$. We set $B[v]\leftarrow 1$, and append $v$ to $F$. Additionally, we increment $i$ by $\left|\lambda(\mathop{}\mathopen{}\operatorname{parent}\mathopen{}(v),v)\right|$. By doing so, we have computed the factorization.

In order to generate the final output, we augment $B$ with a rank data structure, and create a permutation $N$ that maps a marked suffix tree node to the factor it belongs. The permutation $N$ is represented as an array of $z\lg z$ bits, where $N[B.\operatorname{rank}_{1}(F[x])]\leftarrow x$, for $1\leq x\leq z$. At this point, we no longer need $F$. The rest of the algorithm sorts the factors in the factor index order. To this end, we create an array $R$ with $z\lg z$ bits to store the referred indices, and an array $S$ with $z\lg n$ bits to store the factor labels. To compute $S$ and $R$, we scan all marked nodes in $B$: Since the $x$-th marked node $v$ corresponds to the $N[x]$-th factor, we can fill up $S$ easily: If $v$ is a leaf, we store the first character of $\lambda(\mathop{}\mathopen{}\operatorname{parent}\mathopen{}(v),v)$ in $S[N[x]]$; otherwise ($v$ is an internal node), we store the whole string. Filling $R$ is also easy if $v$ is a child of the root: we simply store the referred index [math]. Otherwise, the parent $p$ of $v$ is not the root; $p$ corresponds to the $y$-th factor, where $y:=N[B.\operatorname{rank}_{1}(p)]$.

The algorithm using $\left|\mathsf{ST}\right|+n+z(\lg(2n)+\lg z)+2z\lg n+\mathop{}\mathopen{}o\mathopen{}(n)$ bits of working space, and runs in linear time.

Appendix C LZ78U Code Snippet

Appendix D More Evaluation

In this section, the execution time is measured in second, and all data sizes are measured in mebibytes (MiB). In Table 3, we selected the $t$ with the best compression ratio and the $b$ with the shortest decompression time. Although $t$ and $b$ tend to correlate with the compression speed and decompression memory, respectively, selecting values for $t$ and $b$ that yield a good compression ratio or a fast decompression speed seems difficult.

In Table 4, we fixed two values of $t$ and three values of $b$. The compression ratio of the strategies heap and arrays differ slightly, since the lcpcomp compression scheme does not specify a tie breaking rule for choosing a longest repeated substring.

Figure 8 compares the number of factors of lzss_lcp with lcpcomp’s arrays strategy on all aforementioned datasets. We varied the threshold $t$ from 4 up to 22 and measured for each $t$ the number of created factors. In all cases, lcpcomp produces less factors than lzss_lcp with the same threshold.

Appendix E LZ78U Pseudo Codes