On optimally partitioning a text to improve its compression

TL;DR

This paper presents an efficient algorithm for partitioning strings to improve compression, guaranteeing near-optimal results within a factor of (1+ε) of the best possible, addressing a longstanding open problem.

Contribution

It introduces the first algorithm with provable approximation guarantees for optimal string partitioning in compression, running in near-linear time.

Findings

Algorithm guarantees (1+ε)-approximate optimal partitioning.

Runs in O(n log_{1+ε} n) time, near-linear complexity.

Provides a practical solution to a previously open problem.

Abstract

In this paper we investigate the problem of partitioning an input string T in such a way that compressing individually its parts via a base-compressor C gets a compressed output that is shorter than applying C over the entire T at once. This problem was introduced in the context of table compression, and then further elaborated and extended to strings and trees. Unfortunately, the literature offers poor solutions: namely, we know either a cubic-time algorithm for computing the optimal partition based on dynamic programming, or few heuristics that do not guarantee any bounds on the efficacy of their computed partition, or algorithms that are efficient but work in some specific scenarios (such as the Burrows-Wheeler Transform) and achieve compression performance that might be worse than the optimal-partitioning by a $\Omega(\sqrt{\log n})$ factor. Therefore, computing efficiently the optimal solution is still open. In this paper we provide the first algorithm which is guaranteed to compute in $O(n \log_{1+\eps}n)$ time a partition of T whose compressed output is guaranteed to be no more than $(1+\epsilon)$-worse the optimal one, where $\epsilon$ may be any positive constant.