Optimal Construction of Compressed Indexes for Highly Repetitive Texts

TL;DR

This paper introduces optimal algorithms for constructing key compressed text indexes like BWT, PLCP, and LZ77 parsing, specifically optimized for highly repetitive texts with low run counts, improving efficiency over previous methods.

Contribution

It presents new algorithms that are both time and space optimal for highly repetitive texts, leveraging the measure of repetitiveness to outperform existing general algorithms.

Findings

Algorithms run in $O(n/ ext{log}_{\sigma} n + r ext{ polylog } n)$ time

Significant improvements over previous $O(n)$ time algorithms for certain inputs

Applicable to various string processing problems like Lyndon factorization and run-length compressed suffix arrays

Abstract

We propose algorithms that, given the input string of length $n$ over integer alphabet of size $\sigma$, construct the Burrows-Wheeler transform (BWT), the permuted longest-common-prefix (PLCP) array, and the LZ77 parsing in $O(n/\log_{\sigma}n+r\,{\rm polylog}\,n)$ time and working space, where $r$ is the number of runs in the BWT of the input. These are the essential components of many compressed indexes such as compressed suffix tree, FM-index, and grammar and LZ77-based indexes, but also find numerous applications in sequence analysis and data compression. The value of $r$ is a common measure of repetitiveness that is significantly smaller than $n$ if the string is highly repetitive. Since just accessing every symbol of the string requires $\Omega(n/\log_{\sigma}n)$ time, the presented algorithms are time and space optimal for inputs satisfying the assumption $n/r\in\Omega({\rm polylog}\,n)$ on the repetitiveness. For such inputs our result improves upon the currently fastest general algorithms of Belazzougui (STOC 2014) and Munro et al. (SODA 2017) which run in $O(n)$ time and use $O(n/\log_{\sigma} n)$ working space. We also show how to use our techniques to obtain optimal solutions on highly repetitive data for other fundamental string processing problems such as: Lyndon factorization, construction of run-length compressed suffix arrays, and some classical "textbook" problems such as computing the longest substring occurring at least some fixed number of times.