Practical and Effective Re-Pair Compression

TL;DR

This paper presents a practical implementation of Re-Pair compression that improves space efficiency and introduces a linear-time heuristic for better grammar encoding, achieving near-optimal compression on real datasets.

Contribution

The authors develop a practical, space-efficient Re-Pair implementation and a linear-time heuristic for grammar encoding that approaches the information-theoretic lower bound.

Findings

Improved working space to (1.5+ε)n words for Re-Pair.

Grammar encoding uses only 2.8% more bits than the theoretical minimum.

In tests, the compressor outperforms 7-Zip in half of the cases.

Abstract

Re-Pair is an efficient grammar compressor that operates by recursively replacing high-frequency character pairs with new grammar symbols. The most space-efficient linear-time algorithm computing Re-Pair uses $(1+\epsilon)n+\sqrt n$ words on top of the re-writable text (of length $n$ and stored in $n$ words), for any constant $\epsilon>0$; in practice however, this solution uses complex sub-procedures preventing it from being practical. In this paper, we present an implementation of the above-mentioned result making use of more practical solutions; our tool further improves the working space to $(1.5+\epsilon)n$ words (text included), for some small constant $\epsilon$. As a second contribution, we focus on compact representations of the output grammar. The lower bound for storing a grammar with $d$ rules is $\log(d!)+2d\approx d\log d+0.557 d$ bits, and the most efficient encoding algorithm in the literature uses at most $d\log d + 2d$ bits and runs in $\mathcal O(d^{1.5})$ time. We describe a linear-time heuristic maximizing the compressibility of the output Re-Pair grammar. On real datasets, our grammar encoding uses---on average---only $2.8\%$ more bits than the information-theoretic minimum. In half of the tested cases, our compressor improves the output size of 7-Zip with maximum compression rate turned on.