Optimal-Time Text Indexing in BWT-runs Bounded Space

TL;DR

This paper introduces an advanced text index that efficiently counts and locates pattern occurrences in highly repetitive texts within space bounds related to the BWT run count, achieving optimal query times.

Contribution

It extends the Run-Length FM-index to support efficient locating of pattern occurrences and achieves optimal time bounds in space proportional to the number of BWT runs.

Findings

Supports locating pattern occurrences in O(r) space and loglogarithmic time per occurrence.

Achieves optimal pattern counting in O(m) time within O(r log(n/r)) space.

Provides a text extraction structure with near-optimal time complexity.

Abstract

Indexing highly repetitive texts --- such as genomic databases, software repositories and versioned text collections --- has become an important problem since the turn of the millennium. A relevant compressibility measure for repetitive texts is $r$, the number of runs in their Burrows-Wheeler Transform (BWT). One of the earliest indexes for repetitive collections, the Run-Length FM-index, used $O(r)$ space and was able to efficiently count the number of occurrences of a pattern of length $m$ in the text (in loglogarithmic time per pattern symbol, with current techniques). However, it was unable to locate the positions of those occurrences efficiently within a space bounded in terms of $r$. Since then, a number of other indexes with space bounded by other measures of repetitiveness --- the number of phrases in the Lempel-Ziv parse, the size of the smallest grammar generating the text, the size of the smallest automaton recognizing the text factors --- have been proposed for efficiently locating, but not directly counting, the occurrences of a pattern. In this paper we close this long-standing problem, showing how to extend the Run-Length FM-index so that it can locate the $occ$ occurrences efficiently within $O(r)$ space (in loglogarithmic time each), and reaching optimal time $O(m+occ)$ within $O(r\log(n/r))$ space, on a RAM machine of $w=\Omega(\log n)$ bits. Within $O(r\log (n/r))$ space, our index can also count in optimal time $O(m)$. Raising the space to $O(r w\log_\sigma(n/r))$, we support count and locate in $O(m\log(\sigma)/w)$ and $O(m\log(\sigma)/w+occ)$ time, which is optimal in the packed setting and had not been obtained before in compressed space. We also describe a structure using $O(r\log(n/r))$ space that replaces the text and extracts any text substring of length $\ell$ in almost-optimal time $O(\log(n/r)+\ell\log(\sigma)/w)$. (...continues...)

Figures

Tables

Table 1. Table 1: Previous and our new results on counting, locating, extracting, and supporting suffix tree functionality. We simplified some formulas with tight upper bounds. The main variables are the text size n 𝑛 n , pattern length m 𝑚 m , number of occurrences o ​ c ​ c 𝑜 𝑐 𝑐 occ of the pattern, alphabet size σ 𝜎 \sigma , Lempel-Ziv parsing size z 𝑧 z , smallest grammar size g 𝑔 g , 𝐵𝑊𝑇 𝐵𝑊𝑇 \mathit{BWT} runs r 𝑟 r , CDAWG size e 𝑒 e , and machine word length in bits w 𝑤 w . Variable h ≤ n ℎ 𝑛 h\leq n is the depth of the dependency chain in the Lempel-Ziv parse, and ϵ > 0 italic-ϵ 0 \epsilon>0 is an arbitrarily small constant. Symbols r ¯ ¯ 𝑟 \overline{r} or e ¯ ¯ 𝑒 \overline{e} mean r 𝑟 r or e 𝑒 e of T 𝑇 T plus r 𝑟 r or e 𝑒 e of its reverse. The z 𝑧 z in Nishimoto et al.  [ 79 ] refers to the Lempel-Ziv variant that does not allow overlaps between sources and targets (Kreft and Navarro [ 58 ] claim the same but their index actually works in either variant). Rytter [ 88 ] and Charikar et al.  [ 18 ] enable the given extraction time because they produce balanced grammars of the given size (as several others that came later). Takagi et al.  [ 94 ] claim time O ​ ( m ​ log ⁡ σ + o ​ c ​ c ) 𝑂 𝑚 𝜎 𝑜 𝑐 𝑐 O(m\log\sigma+occ) but they can reach O ​ ( m + o ​ c ​ c ) 𝑂 𝑚 𝑜 𝑐 𝑐 O(m+occ) by using perfect hashing.

Index	Space	Count time
Mäkinen et al. [65, Thm. 17]	$O\left(r\right)$	$O\left(m\left(\frac{\log \sigma }{\log \log r}+{\left(\log \log n\right)}^2\right)\right)$
This paper (Lem. 1)	$O\left(r\right)$	$O\left(m\log {\log }_w\left(\sigma +n/r\right)\right)$
This paper (Thm. 8.1)	$O\left(r\log \left(n/r\right)\right)$	$O\left(m\right)$
This paper (Thm. 8.2)	$O\left(rw{\log }_{\sigma }\left(n/r\right)\right)$	$O\left(m\log \left(\sigma \right)/w\right)$

Table 2. Table 2: Our contributions.

Functionality	Space (words)	Time
Count (Lem. 1)	$O\left(r\right)$	$O\left(m\log {\log }_w\left(\sigma +n/r\right)\right)$
Count (Thm. 8.1)	$O\left(r\log \left(n/r\right)\right)$	$O\left(m\right)$
Count (Thm. 8.2)	$O\left(rw{\log }_{\sigma }\left(n/r\right)\right)$	$O\left(m\log \left(\sigma \right)/w\right)$
Count $+$ Locate (Thm. 3.1)	$O\left(r\right)$	$O\left(m\log {\log }_w\left(\sigma +n/r\right)+occ\log {\log }_w\left(n/r\right)\right)$
Count $+$ Locate (Thm. 3.1)	$O\left(r\log {\log }_w\left(n/r\right)\right)$	$O\left(m\log {\log }_w\left(\sigma +n/r\right)+occ\right)$
Count $+$ Locate (Thm. 5.1)	$O\left(r\log \left(n/r\right)\right)$	$O\left(m+occ\right)$
Count $+$ Locate (Thm. 5.2)	$O\left(rw{\log }_{\sigma }\left(n/r\right)\right)$	$O\left(m\log \left(\sigma \right)/w+occ\right)$
Extract (Thm. 4.1)	$O\left(r\log \left(n/r\right)\right)$	$O\left(\log \left(n/r\right)+\ell \log \left(\sigma \right)/w\right)$
Access $\mathrm{𝑆𝐴}$, $\mathrm{𝐼𝑆𝐴}$, $\mathrm{𝐿𝐶𝑃}$ (Thm. 6.1–6.3)	$O\left(r\log \left(n/r\right)\right)$	$O\left(\log \left(n/r\right)+\ell \right)$
Suffix tree (Thm. 7.1)	$O\left(r\log \left(n/r\right)\right)$	$O\left(\log \left(n/r\right)\right)$ for most operations

Table 3. Table 3: Suffix tree operations.

Operation	Description
Root()	Suffix tree root.
Locate($v$)	Text position $i$ of leaf $v$.
Ancestor($v,w$)	Whether $v$ is an ancestor of $w$.
SDepth($v$)	String depth for internal nodes, i.e., length of string represented by $v$.
TDepth($v$)	Tree depth, i.e., depth of tree node $v$.
Count($v$)	Number of leaves in the subtree of $v$.
Parent($v$)	Parent of $v$.
FChild($v$)	First child of $v$.
NSibling($v$)	Next sibling of $v$.
SLink($v$)	Suffix-link, i.e., if $v$ represents $a\cdot \alpha$ then the node that represents $\alpha$, for $a\in [1..\sigma ]$.
WLink($v,a$)	Weiner-link, i.e., if $v$ represents $\alpha$ then the node that represents $a\cdot \alpha$.
SLinki($v$)	Iterated suffix-link.
LCA($v,w$)	Lowest common ancestor of $v$ and $w$.
Child($v,a$)	Child of $v$ by letter $a$.
Letter($v,i$)	The $ith$ letter of the string represented by $v$.
LAQS($v,d$)	String level ancestor, i.e., the highest ancestor of $v$ with string-depth $\ge d$.
LAQT($v,d$)	Tree level ancestor, i.e., the ancestor of $v$ with tree-depth $d$.

Table 4. Table 4: Complexities of suffix tree operations. Letter ( v , i 𝑣 𝑖 v,i ) can also be solved in time O ​ ( i ⋅ t 𝐿𝐹 ) = O ​ ( i ​ log ⁡ log w ⁡ ( n / r ) ) 𝑂 ⋅ 𝑖 subscript 𝑡 𝐿𝐹 𝑂 𝑖 subscript 𝑤 𝑛 𝑟 O(i\cdot t_{\mathit{LF}})=O(i\log\log_{w}(n/r)) .

Operation	Generic	Our
	Complexity	Complexity
Root()	$1$	$1$
Locate($v$)	$t_{\mathrm{𝑆𝐴}}$	$\log \left(n/r\right)$
Ancestor($v,w$)	$1$	$1$
SDepth($v$)	$t_{\mathrm{RMQ}}+t_{\mathrm{𝐿𝐶𝑃}}$	$\log \left(n/r\right)$
TDepth($v$)	$t_{\mathrm{𝑆𝐴}}$	$\log \left(n/r\right)$
Count($v$)	$1$	$1$
Parent($v$)	$t_{\mathrm{𝐿𝐶𝑃}}+t_{\mathrm{SV}}$	$\log \left(n/r\right)$
FChild($v$)	$t_{\mathrm{RMQ}}$	$\log \left(n/r\right)$
NSibling($v$)	$t_{\mathrm{𝐿𝐶𝑃}}+t_{\mathrm{SV}}$	$\log \left(n/r\right)$
SLink($v$)	$t_{\mathrm{𝐿𝐹}}+t_{\mathrm{RMQ}}+t_{\mathrm{SV}}$	$\log \left(n/r\right)$
WLink($v$)	$t_{\mathrm{𝐿𝐹}}$	$\log {\log }_w\left(n/r\right)$
SLinki($v$)	$t_{\mathrm{𝑆𝐴}}+t_{\mathrm{RMQ}}+t_{\mathrm{SV}}$	$\log \left(n/r\right)$
LCA($v,w$)	$t_{\mathrm{RMQ}}+t_{\mathrm{SV}}$	$\log \left(n/r\right)$
Child($v,a$)	$t_{\mathrm{𝐿𝐶𝑃}}+\left(t_{\mathrm{RMQ}}+t_{\mathrm{𝑆𝐴}}+t_{\mathrm{𝐿𝐹}}\right)\log \sigma$	$\log \left(n/r\right)\log \sigma$
Letter($v,i$)	$t_{\mathrm{𝑆𝐴}}+t_{\mathrm{𝐿𝐹}}$	$\log \left(n/r\right)$
LAQS($v,d$)	$t_{\mathrm{SV}}$	$\log \left(n/r\right)$
LAQT($v,d$)	$\left(t_{\mathrm{RMQ}}+t_{\mathrm{𝐿𝐶𝑃}}\right)\log n$	$\log \left(n/r\right)\log n$

Equations

$$LF(i)=C[i]+{\mathrm{rank}}_{L[i]}(L,i).$$

$$s{p}_i$$

$$e{p}_i$$

$$SA[LF(i)]=SA[i]-1.$$

$${\mathrm{rank}}_c(L,i)=D[C^{\prime }[c]+{\mathrm{rank}}_c(L^{\prime },pred(i).p-1)]+[\text{if}{L}^{\prime }[pred(i).p]=c\text{then}i-pred(i)+1\text{else}0]$$

$$W^{\prime }[{\pi }^{\prime }(\mu (p_i^{+}+1)+1)..{\pi }^{\prime }(\mu (p_{i+1}^{-}-1)-1)]$$

Full text

1 Introduction

The data deluge has become a routine problem in most organizations that aim to collect and process data, even in relatively modest and focused scenarios. We are concerned about string (or text, or sequence) data, formed by collections of symbol sequences. This includes natural language text collections, DNA and protein sequences, source code repositories, digitalized music, and many others. The rate at which those sequence collections are growing is daunting, and outpaces Moore’s Law by a significant margin [92]. One of the key technologies to handle those growing datasets is compact data structures, which aim to handle the data directly in compressed form, without ever decompressing it [78]. In general, however, compact data structures do not compress the data by orders of magnitude, but rather offer complex functionality within the space required by the raw data, or a moderate fraction of it. As such, they do not seem to offer the significant space reductions that are required to curb the sharply growing sizes of today’s datasets.

What makes a fundamental difference, however, is that the fastest-growing string collections are in many cases highly repetitive, that is, most of the strings can be obtained from others with a few modifications. For example, most genome sequence collections store many genomes from the same species, which in the case of, say, humans differ by 0.1% [85] (there is some discussion about the exact percentage). The 1000-genomes project111http://www.internationalgenome.org uses a Lempel-Ziv-like compression mechanism that reports compression ratios around 1% [34]. Versioned document collections and software repositories are another natural source of repetitiveness. For example, Wikipedia reports that, by June 2015, there were over 20 revisions (i.e., versions) per article in its 10 TB content, and that p7zip compressed it to about 1%. They also report that what grows the fastest today are the revisions rather than the new articles, which increases repetitiveness.222https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia A study of GitHub (which surpassed 20 TB in 2016)333https://blog.sourced.tech/post/tab_vs_spaces reports a ratio of commit (new versions) over create (brand new projects) around 20.444http://blog.coderstats.net/github/2013/event-types, see the ratios of push per create and commit per push. Repetitiveness also arises in other less obvious scenarios: it is estimated that about 50% of (non-versioned) software sources [51], 40% of the Web pages [46], 50% of emails [25], and 80% of tweets [95], are near-duplicates.

When the versioning structure is explicit, version management systems are able to factor out repetitiveness efficiently while providing access to any version. The idea is simply to store the first version of a document in plain form and then the edits of each version of it, so as to reconstruct any version efficiently. This becomes much harder when there is not a clear versioning structure (as in genomic databases) or when we want to provide more advanced functionalities, such as counting or locating the positions where a string pattern occurs across the collection. In this case, the problem is how to reduce the size of classical data structures for indexed pattern matching, like suffix trees [98] or suffix arrays [66], so that they become proportional to the amount of distinct material in the collection. It should be noted that all the work on statistical compression of suffix trees and arrays [74] is not useful for this purpose, as it does not capture this kind of repetitiveness [58, Lem. 2.6].

Mäkinen et al. [63, 91, 64, 65] pioneered the research on structures for searching repetitive collections. They regard the collection as a single concatenated text $T[1..n]$ with separator symbols, and note that the number $r$ of runs (i.e., maximal substrings formed by a single symbol) in the Burrows-Wheeler Transform [17] of the text is very low on repetitive texts. Their index, Run-Length FM-Index (RLFM-index), uses $O(r)$ words and can count the number of occurrences of a pattern $P[1..m]$ in time $O(m\log n)$ and even less. However, they are unable to locate where those positions are in $T$ unless they add a set of samples that require $O(n/s)$ words in order to offer $O(s\log n)$ time to locate each occurrence. On repetitive texts, either this sampled structure is orders of magnitude larger than the $O(r)$-size basic index, or the locating time is unacceptably high.

Many proposals since then aimed at reducing the locating time by building on other measures related to repetitiveness: indexes based on the Lempel-Ziv parse [61] of $T$, with size bounded in terms of the number $z$ of phrases [58, 36, 79, 6]; indexes based on the smallest context-free grammar [18] that generates $T$, with size bounded in terms of the size $g$ of the grammar [22, 23, 35]; and indexes based on the size $e$ of the smallest automaton (CDAWG) [16] recognizing the substrings of $T$ [6, 94, 4]. The achievements are summarized in Table 1; note that none of those later approaches is able to count the occurrences without enumerating them all. We are not considering in this paper indexes based on other measures of repetitiveness that only apply in restricted scenarios, such as based on Relative Lempel-Ziv [59, 24, 8, 27] or on alignments [72, 73].

There are a few known asymptotic bounds between the repetitiveness measures $r$, $z$, $g$, and $e$: $z\leq g=O(z\log(n/z))$ [88, 18, 50] and $e=\Omega(\max(r,z,g))$ [6, 5]. Several examples of string families are known that show that $r$ is not comparable with $z$ and $g$ [6, 84]. Experimental results [65, 58, 6, 21], on the other hand, suggest that in typical repetitive texts it holds $z<r\approx g\ll e$.

In highly repetitive texts, one expects not only to have a compressed index able to count and locate pattern occurrences, but also to replace the text with a compressed version that nonetheless can efficiently extract any substring $T[i..i+\ell]$. Indexes that, implicitly or not, contain a replacement of $T$, are called self-indexes. As can be seen in Table 1, self-indexes with $O(z)$ space require up to $O(n)$ time per extracted character, and none exists within $O(r)$ space. Good extraction times are instead obtained with $O(g)$, $O(z\log(n/z))$, or $O(e)$ space. A lower bound [97] shows that $\Omega((\log n)^{1-\epsilon}/\log g)$ time, for every constant $\epsilon>0$, is needed to access one random position within $O(\mathrm{poly}(g))$ space. This bound shows that various current techniques using structures bounded in terms of $g$ or $z$ [15, 11, 37, 7] are nearly optimal (note that $g=\Omega(\log n)$, so the space of all these structures is $O(\mathrm{poly}(g))$). In an extended article [19], they give a lower bound in terms of $r$, but only for binary texts and $\log r=o(w)$: $\Omega\left(\frac{\log n}{w^{\epsilon/(1-\epsilon)}\log r}\right)$ for any constant $\epsilon>0$, where $w=\Omega(\log n)$ is the number of bits in the RAM word. In fact, since there are string families where $z=\Omega(r\log n)$ [84], no extraction mechanism in space $O(\mathrm{poly}(r))$ can escape in general from the lower bound [97].

In more sophisticated applications, especially in bioinformatics, it is desirable to support a more complex set of operations, which constitute a full suffix tree functionality [45, 81, 62]. While Mäkinen et al. [65] offered suffix tree functionality, they had the same problem of needing $O(n/s)$ space to achieve $O(s\log n)$ time for most suffix tree operations. Only recently a suffix tree of size $O(\overline{e})$ supports most operations in time $O(\log n)$ [6, 5], where $\overline{e}$ refers to the $e$ measure of $T$ plus that of $T$ reversed.

Summarizing Table 1 and our discussion, the situation on repetitive text indexing is as follows.

The RLFM-index is the only structure able to efficiently count the occurrences of $P$ in $T$ without having to enumerate them all. However, it does not offer efficient locating within $O(r)$ space. 2. 2.

The only structure clearly smaller than the RLFM-index, using $O(z)$ space [58], has unbounded locate time. Structures using $O(g)$ space, which is about the same space of the RLFM-index, have an additive penalty quadratic in $m$ in their locate time. 3. 3.

Structures offering lower locate time require $O(z\log(n/z))$ space or more [36, 79, 13], $O(\overline{r}+z)$ space [6] (where $\overline{r}$ is the sum of $r$ for $T$ and its reverse), or $O(e)$ space or more [6, 94, 4]. 4. 4.

Self-indexes with efficient extraction require $O(z\log(n/z))$ space or more [37, 7], $O(g)$ space [15, 11], or $O(e)$ space or more [94, 4]. 5. 5.

The only efficient compressed suffix tree requires $O(\overline{e})$ space [5].

Efficiently locating the occurrences of $P$ in $T$ within $O(r)$ space has been a bottleneck and an open problem for almost a decade. In this paper we give the first solution to this problem. Our precise contributions, largely detailed in Tables 1 and 2, are the following.

We improve the counting time of the RLFM-index to $O(m\log\log_{w}(\sigma+n/r))$, where $\sigma\leq r$ is the alphabet size of $T$. 2. 2.

We show how to locate each occurrence in time $O(\log\log_{w}(n/r))$, within $O(r)$ space. We reduce the locate time to $O(1)$ per occurrence by using slightly more space, $O(r\log\log_{w}(n/r))$. 3. 3.

We give the first structure building on $\mathit{BWT}$ runs that replaces $T$ while retaining direct access. It extracts any substring of length $\ell$ in time $O(\log(n/r)+\ell\log(\sigma)/w)$, using $O(r\log(n/r))$ space. As discussed, even the additive penalty is near-optimal [97]. 4. 4.

By using $O(r\log(n/r))$ space, we obtain optimal locate time, $O(m+occ)$. This had been obtained only using $O(e)$ [4] or $O(\overline{e})$ [94] space. By increasing the space to $O(rw\log_{\sigma}(n/r))$, we obtain optimal locate time $O(m\log(\sigma)/w+occ)$ in the packed setting (i.e., the pattern symbols come packed in blocks of $w/\log\sigma$ symbols per word). This had not been achieved so far by any compressed index, but only on uncompressed ones [75]. 5. 5.

By using $O(r\log(n/r))$ space, we obtain optimal count time, $O(m)$. We also achieve optimal count time in the packed setting, $O(m\log(\sigma)/w)$, by increasing the space to $O(rw\log_{\sigma}(n/r))$. As for locate, this is the first compressed index achieving optimal-time count. 6. 6.

We give the first compressed suffix tree whose space is bounded in terms of $r$, $O(r\log(n/r))$ words. It implements most navigation operations in time $O(\log(n/r))$. There exist only comparable suffix trees within $O(\overline{e})$ space [5], taking $O(\log n)$ time for most operations. 7. 7.

We uncover new relations between $r$, $g$, $g_{rl}\leq g$ (the smallest *run-length * context-free grammar [80], which allows size-1 rules of the form $X\rightarrow Y^{t}$), $z$, $z_{no}$ (the Lempel-Ziv parse that does not allow overlaps), and $b$ (the smallest bidirectional parsing of $T$ [93]). Namely, we show that $b\leq r$, $z\leq 2g_{rl}=O(b\log(n/b))$, $\max(g,z_{no})=O(b\log^{2}(n/b))$, and that $z$ can be $\Omega(b\log n)$.

Contribution 1 is a simple update of the RLFM-index [65] with newer data structures for rank and predecessor queries [10]. Contribution 2 is one of the central parts of the paper, and is obtained in two steps. The first uses the fact that we can carry out the classical RLFM counting process for $P$ in a way that we always know the position of one occurrence in $T$ [84, 82]; we give a simpler proof of this fact. The second shows that, if we know the position in $T$ of one occurrence of $\mathit{BWT}$, then we can obtain the preceding and following ones with an $O(r)$-size sampling. This is achieved by using the $\mathit{BWT}$ runs to induce phrases in $T$ (which are somewhat analogous to the Lempel-Ziv phrases [61]) and showing that the positions of occurrences within phrases can be obtained from the positions of their preceding phrase beginning. The time $O(1)$ is obtained by using an extended sampling. Contribution 3 creates an analogous of the Block Tree [7] built on the $\mathit{BWT}$-induced phrases, which satisfy the same property that any distinct string has an occurrence overlapping a boundary between phrases. For Contribution 4, we discard the RLFM-index and use a mechanism similar to the one used in Lempel-Ziv or grammar indexes [22, 23, 58] to find one primary occurrence, that is, one that overlaps phrase boundaries; then the others are found with the mechanism to obtain neighboring occurrences already described. Here we use a stronger property of primary occurrences that does not hold on those of Lempel-Ziv or grammars, and that might have independent interest. Further, to avoid time quadratic in $m$ to explore all the suffixes of $P$, we use a (deterministic) mechanism based on Karp-Rabin signatures [3, 36], which we show how to compute from a variant of the structure we create for extracting text substrings. The optimal packed time is obtained by enlarging samplings. In Contribution 5, we use the components used in Contribution 4 to find a pattern occurrence, and then find the $\mathit{BWT}$ range of the pattern with range searches on the longest common prefix array $\mathit{LCP}$, supported by compressed suffix tree primitives (see next). Contribution 6 needs basically direct access to the suffix array $\mathit{SA}$, inverse suffix array $\mathit{ISA}$, and array $\mathit{LCP}$ of $T$, for which we show that a recent grammar compression method achieving locally consistent parsing [49, 48] interacts with the $\mathit{BWT}$ runs/phrases to produce run-length context-free grammars of size $O(r\log(n/r))$ and height $O(\log(n/r))$. The suffix tree operations also need primitives on the $\mathit{LCP}$ array that compute range minimum queries and previous/next smaller values [33]. Finally, for Contribution 7 we show that a locally-consistent-parsing-based run-length grammar of size $O(b\log(n/b))$ can compress a bidirectional scheme of size $b$, and this yields upper bounds on $z$ as well. We also show that the $\mathit{BWT}$ runs induce a valid bidirectional scheme on $T$, so $b\leq r$, and use known examples [84] where $z=\Omega(r\log n)$. The other relations are derived from known bounds.

2 Basic Concepts

A string is a sequence $S[1..\ell]=S[1]S[2]\ldots S[\ell]$, of length $\ell=|S|$, of symbols (or characters, or letters) chosen from an alphabet $[1..\sigma]=\{1,2,\ldots,\sigma\}$, that is, $S[i]\in[1..\sigma]$ for all $1\leq i\leq\ell$. We use $S[i..j]=S[i]\ldots S[j]$, with $1\leq i,j\leq\ell$, to denote a substring of $S$, which is the empty string $\varepsilon$ if $i>j$. A prefix of $S$ is a substring of the form $S[1..i]$ and a suffix is a substring of the form $S[i..\ell]$. The yuxtaposition of strings and/or symbols represents their concatenation.

We will consider indexing a text $T[1..n]$, which is a string over alphabet $[1..\sigma]$ terminated by the special symbol \=1$, that is, the lexicographically smallest one, which appears only at$T[n]=$$. This makes any lexicographic comparison between suffixes well defined.

Our computation model is the transdichotomous RAM, with a word of $w=\Omega(\log n)$ bits, where all the standard arithmetic and logic operations can be carried out in constant time. In this article we generally measure space in words.

2.1 Suffix Trees and Arrays

The suffix tree [98] of $T[1..n]$ is a compacted trie where all the $n$ suffixes of $T$ have been inserted. By compacted we mean that chains of degree-1 nodes are collapsed into a single edge that is labeled with the concatenation of the individual symbols labeling the collapsed edges. The suffix tree has $n$ leaves and less than $n$ internal nodes. By representing edge labels with pointers to $T$, the suffix tree uses $O(n)$ space, and can be built in $O(n)$ time [98, 69, 96, 26].

The suffix array [66] of $T[1..n]$ is an array $\mathit{SA}[1..n]$ storing a permutation of $[1..n]$ so that, for all $1\leq i<n$, the suffix $T[\mathit{SA}[i]..]$ is lexicographically smaller than the suffix $T[\mathit{SA}[i+1]..]$. Thus $\mathit{SA}[i]$ is the starting position in $T$ of the $i$th smallest suffix of $T$ in lexicographic order. This can be regarded as an array collecting the leaves of the suffix tree. The suffix array uses $n$ words and can be built in $O(n)$ time [56, 57, 52].

All the occurrences of a pattern string $P[1..m]$ in $T$ can be easily spotted in the suffix tree or array. In the suffix tree, we descend from the root matching the successive symbols of $P$ with the strings labeling the edges. If $P$ is in $T$, the symbols of $P$ will be exhausted at a node $v$ or inside an edge leading to a node $v$; this node is called the locus of $P$, and all the $occ$ leaves descending from $v$ are the suffixes starting with $P$, that is, the starting positions of the occurrences of $P$ in $T$. By using perfect hashing to store the first characters of the edge labels descending from each node of $v$, we reach the locus in optimal time $O(m)$ and the space is still $O(n)$. If $P$ comes packed using $w/\log\sigma$ symbols per computer word, we can descend in time $O(m\log(\sigma)/w)$ [75], which is optimal in the packed model. In the suffix array, all the suffixes starting with $P$ form a range $\mathit{SA}[sp..ep]$, which can be binary searched in time $O(m\log n)$, or $O(m+\log n)$ with additional structures [66].

The inverse permutation of $\mathit{SA}$, $\mathit{ISA}[1..n]$, is called the inverse suffix array, so that $\mathit{ISA}[j]$ is the lexicographical position of the suffix $T[j..n]$ among the suffixes of $T$.

Another important concept related to suffix arrays and trees is the longest common prefix array. Let $lcp(S,S^{\prime})$ be the length of the longest common prefix between strings $S$ and $S^{\prime}$, that is, $S[1..lcp(S,S^{\prime})]=S^{\prime}[1..lcp(S,S^{\prime})]$ but $S[lcp(S,S^{\prime})+1]\not=S^{\prime}[lcp(S,S^{\prime})+1]$. Then we define the longest common prefix array $\mathit{LCP}[1..n]$ as $\mathit{LCP}[1]=0$ and $\mathit{LCP}[i]=lcp(T[\mathit{SA}[i-1]..],T[\mathit{SA}[i]..])$. The $\mathit{LCP}$ array uses $n$ words and can be built in $O(n)$ time [55].

2.2 Self-indexes

A self-index is a data structure built on $T[1..n]$ that provides at least the following functionality:

Count:

Given a pattern $P[1..m]$, count the number of occurrences of $P$ in $T$.

Locate:

Given a pattern $P[1..m]$, return the positions where $P$ occurs in $T$.

Extract:

Given a range $[i..i+\ell-1]$, return $T[i..i+\ell-1]$.

The last operation allows a self-index to act as a replacement of $T$, that is, it is not necessary to store $T$ since any desired substring can be extracted from the self-index. This can be trivially obtained by including a copy of $T$ as a part of the self-index, but it is challenging when the self-index uses less space than a plain representation of $T$.

In principle, suffix trees and arrays can be regarded as self-indexes that can count in time $O(m)$ or $O(m\log(\sigma)/w)$ (suffix tree, by storing $occ$ in each node $v$) and $O(m\log n)$ or $O(m+\log n)$ (suffix array, with $occ=ep-sp+1$), locate each occurrence in $O(1)$ time, and extract in time $O(1+\ell\log(\sigma)/w)$. However, they use $O(n\log n)$ bits, much more than the $n\log\sigma$ bits needed to represent $T$ in plain form. We are interested in compressed self-indexes [74, 78], which use the space required by a compressed representation of $T$ (under some entropy model) plus some redundancy (at worst $o(n\log\sigma)$ bits). We describe later the FM-index, a particular self-index of interest to us.

2.3 Burrows-Wheeler Transform

The Burrows-Wheeler Transform of $T[1..n]$, $\mathit{BWT}[1..n]$ [17], is a string defined as $\mathit{BWT}[i]=T[\mathit{SA}[i]-1]$ if $\mathit{SA}[i]>1$, and \mathit{BWT}[i]=T[n]=\$$ if \mathit{SA}[i]=1$. That is,$\mathit{BWT}$has the same symbols of$T$ in a different order, and is a reversible transform.

The array $\mathit{BWT}$ is obtained from $T$ by first building $\mathit{SA}$, although it can be built directly, in $O(n)$ time and within $O(n\log\sigma)$ bits of space [70]. To obtain $T$ from $\mathit{BWT}$ [17], one considers two arrays, $L[1..n]=\mathit{BWT}$ and $F[1..n]$, which contains all the symbols of $L$ (or $T$) in ascending order. Alternatively, $F[i]=T[\mathit{SA}[i]]$, so $F[i]$ follows $L[i]$ in $T$. We need a function that maps any $L[i]$ to the position $j$ of that same character in $F$. The formula is $\mathit{LF}(i)=C[c]+\mathrm{rank}[i]$, where $c=L[i]$, $C[c]$ is the number of occurrences of symbols less than $c$ in $L$, and $\mathrm{rank}[i]$ is the number of occurrences of symbol $L[i]$ in $L[1..i]$. A simple $O(n)$-time pass on $L$ suffices to compute arrays $C[i]$ and $\mathrm{rank}[i]$ using $O(n\log\sigma)$ bits of space. Once they are computed, we reconstruct T[n]=\$$ and T[n-k]\leftarrow L[\mathit{LF}^{k-1}(1)]$for$k=1,\ldots,n-1$, in$O(n)$ time as well.

2.4 Compressed Suffix Arrays and FM-indexes

Compressed suffix arrays [74] are a particular case of self-indexes that simulate $\mathit{SA}$ in compressed form. Therefore, they aim to obtain the suffix array range $[sp..ep]$ of $P$, which is sufficient to count since $P$ then appears $occ=ep-sp+1$ times in $T$. For locating, they need to access the content of cells $\mathit{SA}[sp],\ldots,\mathit{SA}[ep]$, without having $\mathit{SA}$ stored.

The FM-index [28, 29] is a compressed suffix array that exploits the relation between the string $L=\mathit{BWT}$ and the suffix array $\mathit{SA}$. It stores $L$ in compressed form (as it can be easily compressed to the high-order empirical entropy of $T$ [68]) and adds sublinear-size data structures to compute (i) any desired position $L[i]$, (ii) the generalized rank function $\mathrm{rank}_{c}(L,i)$, which is the number of times symbol $c$ appears in $L[1..i]$. Note that these two operations permit, in particular, computing $\mathrm{rank}[i]=\mathrm{rank}_{L[i]}(L,i)$, which is called partial rank. Therefore, they compute

[TABLE]

For counting, the FM-index resorts to backward search. This procedure reads $P$ backwards and at any step knows the range $[sp_{i},ep_{i}]$ of $P[i..m]$ in $T$. Initially, we have the range $[sp_{m+1}..ep_{m+1}]=[1..n]$ for $P[m+1..m]=\varepsilon$. Given the range $[sp_{i+1}..ep_{i+1}]$, one obtains the range $[sp_{i}..ep_{i}]$ from $c=P[i]$ with the operations

[TABLE]

Thus the range $[sp..ep]=[sp_{1}..ep_{1}]$ is obtained with $O(m)$ computations of $\mathrm{rank}$, which dominates the counting complexity.

For locating, the FM-index (and most compressed suffix arrays) stores sampled values of $\mathit{SA}$ at regularly spaced text positions, say multiples of $s$. Thus, to retrieve $\mathit{SA}$, we find the smallest $k$ for which $\mathit{SA}[\mathit{LF}^{k}(i)]$ is sampled, and then the answer is $\mathit{SA}[i]=\mathit{SA}[\mathit{LF}^{k}(i)]+k$. This is because function $\mathit{LF}$ virtually traverses the text backwards, that is, it drives us from $L[i]$, which points to some $\mathit{SA}[i]$, to its corresponding position $F[j]$, which is preceded by $L[j]$, that is, $\mathit{SA}[j]=\mathit{SA}[i]-1$:

[TABLE]

Since it is guaranteed that $k<s$, each occurrence is located with $s$ accesses to $L$ and computations of $\mathit{LF}$, and the extra space for the sampling is $O((n\log n)/s)$ bits, or $O(n/s)$ words.

For extracting, a similar sampling is used on $\mathit{ISA}$, that is, we sample the positions of $\mathit{ISA}$ that are multiples of $s$. To extract $T[i..i+\ell-1]$ we find the smallest multiple of $s$ in $[i+\ell..n]$, $j=s\cdot\lceil(i+\ell)/s\rceil$, and extract $T[i..j]$. Since $\mathit{ISA}[j]=p$ is sampled, we know that $T[j-1]=L[p]$, $T[j-2]=L[\mathit{LF}(p)]$, and so on. In total we require at most $\ell+s$ accesses to $L$ and computations of $\mathit{LF}$ to extract $T[i..i+\ell-1]$. The extra space is also $O(n/s)$ words.

For example, using a representation [10] that accesses $L$ and computes partial ranks in constant time (so $\mathit{LF}$ is computed in $O(1)$ time), and computes $\mathrm{rank}$ in the optimal $O(\log\log_{w}\sigma)$ time, an FM-index can count in time $O(m\log\log_{w}\sigma)$, locate each occurrence in $O(s)$ time, and extract $\ell$ symbols of $T$ in time $O(s+\ell)$, by using $O(n/s)$ space on top of the empirical entropy of $T$ [10]. There exist even faster variants [9], but they do not rely on backward search.

2.5 Run-Length FM-index

One of the sources of the compressibility of $\mathit{BWT}$ is that symbols are clustered into $r\leq n$ runs, which are maximal substrings formed by the same symbol. Mäkinen and Navarro [63] proved a (relatively weak) bound on $r$ in terms of the high-order empirical entropy of $T$ and, more importantly, designed an FM-index variant that uses $O(r)$ words of space, called Run-Length FM-index or RLFM-index. They later experimented with several variants of the RLFM-index, where the variant RLFM+ [65, Thm. 17] corresponds to the original one [63].

The structure stores the run heads, that is, the first positions of the runs in $\mathit{BWT}$, in a data structure $E=\{1\}\cup\{1<i\leq n,\mathit{BWT}[i]\not=\mathit{BWT}[i-1]\}$ that supports predecessor searches. Each element $e\in E$ has associated the value $e.p=|\{e^{\prime}\in E,e^{\prime}\leq e\}|$, which gives its position in a string $L^{\prime}[1..r]$ that stores the run symbols. Another array, $D[0..r]$, stores the cumulative lengths of the runs after sorting them lexicographically by their symbols (with $D[0]=0$). Let array $C^{\prime}[1..\sigma]$ count the number of runs of symbols smaller than $c$ in $L$. One can then simulate

[TABLE]

at the cost of a predecessor search ($pred$) in $E$ and a $\mathrm{rank}$ on $L^{\prime}$. By using up-to-date data structures, the counting performance of the RLFM-index can be stated as follows.

Lemma 1

The Run-Length FM-index of a text $T[1..n]$ whose $\mathit{BWT}$ has $r$ runs can occupy $O(r)$ words and count the number of occurrences of a pattern $P[1..m]$ in time $O(m\log\log_{w}(\sigma+n/r))$. It also computes $\mathit{LF}$ and access to any $\mathit{BWT}[p]$ in time $O(\log\log_{w}(n/r))$.

Proof

We use the RLFM+ [65, Thm. 17], using the structure of Belazzougui and Navarro [10, Thm. 10] for the sequence $L^{\prime}$ (with constant access time) and the predecessor data structure described by Belazzougui and Navarro [10, Thm. 14] to implement $E$ (instead of the bitvector they originally used). They also implement $D$ with a bitvector, but we use a plain array. The sum of both operation times is $O(\log\log_{w}\sigma+\log\log_{w}(n/r))$, which can be written as $O(\log\log_{w}(\sigma+n/r))$. To access $\mathit{BWT}[p]=L[p]$ we only need a predecessor search on $E$, which takes time $O(\log\log_{w}(n/r))$. Finally, we compute $\mathit{LF}$ faster than a general rank query, as we only need the partial rank query $rank_{L[i]}(L,i)$. This can be supported in constant time on $L^{\prime}$ using $O(r)$ space, by just recording all the answers, and therefore the time for $\mathit{LF}$ on $L$ is also dominated by the predecessor search on $E$, with $O(\log\log_{w}(n/r))$ time. ∎

We will generally assume that $\sigma$ is the effective alphabet of $T$, that is, the $\sigma$ symbols appear in $T$. This implies that $\sigma\leq r\leq n$. If this is not the case, we can map $T$ to an effective alphabet $[1..\sigma^{\prime}]$ before indexing it. A mapping of $\sigma^{\prime}\leq r$ words then stores the actual symbols when extracting a substring of $T$ is necessary. For searches, we have to map the $m$ positions of $P$ to the effective alphabet. By storing a predecessor structure of $O(\sigma^{\prime})=O(r)$ words, we map each symbol of $P$ in time $O(\log\log_{w}(\sigma/\sigma^{\prime}))$ [10, Thm. 14]. This is within the bounds given in Lemma 1, which therefore holds for any alphabet size.

To provide locating and extracting functionality, Mäkinen et al. [65] use the sampling mechanism we described for the FM-index. Therefore, although they can efficiently count within $O(r)$ space, they need a much larger $O(n/s)$ space to support these operations in time proportional to $O(s)$. Despite various efforts [65], this has been a bottleneck in theory and in practice since then.

2.6 Compressed Suffix Trees

Suffix trees provide a much more complete functionality than self-indexes, and are used to solve complex problems especially in bioinformatic applications [45, 81, 62]. A compressed suffix tree is regarded as an enhancement of a compressed suffix array (which, in a sense, represents only the leaves of the suffix tree). Such a compressed representation must be able to simulate the operations on the classical suffix tree (see Table 4 later in the article), while using little space on top of the compressed suffix array. The first such compressed suffix tree [89] used $O(n)$ extra bits, and there are variants using $o(n)$ extra bits [33, 31, 87, 41, 1].

Instead, there are no compressed suffix trees using $O(r)$ space. An extension of the RLFM-index [65] still needs $O(n/s)$ space to carry out most of the suffix tree operations in time $O(s\log n)$. Some variants that are designed for repetitive text collections [1, 76] are heuristic and do not offer worst-case guarantees. Only recently a compressed suffix tree was presented [5] that uses $O(\overline{e})$ space and carries out operations in $O(\log n)$ time.

3 Locating Occurrences

In this section we show that, if the $\mathit{BWT}$ of a text $T[1..n]$ has $r$ runs, we can have an index using $O(r)$ space that not only efficiently finds the interval $\mathit{SA}[sp..ep]$ of the occurrences of a pattern $P[1..m]$ (as was already known in the literature, see previous sections) but that can locate each such occurrence in time $O(\log\log_{w}(n/r))$ on a RAM machine of $w$ bits. Further, the time per occurrence may become constant if the space is raised to $O(r\log\log_{w}(n/r))$.

We start with Lemma 2, which shows that the typical backward search process can be enhanced so that we always know the position of one of the values in $\mathit{SA}[sp..ep]$. Our proof simplifies a previous one [84, 82]. Lemma 3 then shows how to efficiently obtain the two neighboring cells of $\mathit{SA}$ if we know the value of one. This allows us extending the first known cell in both directions, until obtaining the whole interval $\mathit{SA}[sp..ep]$. In Lemma 4 we show how this process can be sped up by using more space. Theorem 3.1 then summarizes the main result of this section.

In Section 3.1 we extend the idea in order to obtain $\mathit{LCP}$ values analogously to how we obtain $\mathit{SA}$ values. While not of immediate use for locating, this result is useful later in the article and also has independent interest.

Lemma 2

We can store $O(r)$ words such that, given $P[1..m]$, in time $O(m\log\log_{w}(\sigma+n/r))$ we can compute the interval $\mathit{SA}[sp,ep]$ of the occurrences of $P$ in $T$, and also return the position $j$ and contents $\mathit{SA}[j]$ of at least one cell in the interval $[sp,ep]$.

Proof

We store a RLFM-index and predecessor structures $R_{c}$ storing the position in $\mathit{BWT}$ of the right and left endpoints of each run of copies of $c$. Each element in $R_{c}$ is associated to its corresponding text position, that is, we store pairs $\langle i,\mathit{SA}[i]-1\rangle$ sorted by their first component (equivalently, we store in the predecessor structures their concatenated binary representation). These structures take a total of $O(r)$ words.

The interval of characters immediately preceding occurrences of the empty string is the entire $\mathit{BWT}[1..n]$, which clearly includes $P[m]$ as the last character in some run (unless $P$ does not occur in $T$). It follows that we find an occurrence of $P[m]$ in predecessor time by querying $pred(R_{P[m]},n)$.

Assume we have found the interval $\mathit{BWT}[sp,ep]$ containing the characters immediately preceding all the occurrences of some (possibly empty) suffix $P[i+1..m]$ of $P$, and we know the position and contents of some cell $\mathit{SA}[j]$ in the corresponding interval, $sp\leq j\leq ep$. Since $\mathit{SA}[\mathit{LF}(j)]=\mathit{SA}[j]-1$, if $\mathit{BWT}[j]=P[i]$ then, after the next application of $\mathit{LF}$-mapping, we still know the position and value of some cell $\mathit{SA}[j^{\prime}]$ corresponding to the interval $\mathit{BWT}[sp^{\prime},ep^{\prime}]$ for $P[i..m]$, namely $j^{\prime}=\mathit{LF}(j)$ and $\mathit{SA}[j^{\prime}]=\mathit{SA}[j]-1$.

On the other hand, if $\mathit{BWT}[j]\neq P[i]$ but $P$ still occurs somewhere in $T$ (i.e., $sp^{\prime}\leq ep^{\prime}$), then there is at least one $P[i]$ and one non-$P[i]$ in $\mathit{BWT}[sp,ep]$, and therefore the interval intersects an extreme of a run of copies of $P[i]$. Then, a predecessor query $pred(R_{P[i]},ep)$ gives us the desired pair $\langle j^{\prime},\mathit{SA}[j^{\prime}]-1\rangle$ with $sp\leq j^{\prime}\leq ep$ and $\mathit{BWT}[j^{\prime}]=P[i]$.

Therefore, by induction, when we have computed the $\mathit{BWT}$ interval for $P$, we know the position and contents of at least one cell in the corresponding interval in $\mathit{SA}$.

To obtain the desired time bounds, we concatenate all the universes of the $R_{c}$ structures into a single one of size $\sigma n$, and use a single structure $R$ on that universe: each $\langle x,\mathit{SA}[x-1]\rangle\in R_{c}$ becomes $\langle x+(c-1)n,\mathit{SA}[x]-1\rangle$ in $R$, and a search $pred(R_{c},y)$ becomes $pred(R,(c-1)n+y)-(c-1)n$. Since $R$ contains $2r$ elements on a universe of size $\sigma n$, we can have predecessor searches in time $O(\log\log_{w}(n\sigma/r))$ and $O(r)$ space [10, Thm. 14]. This is the same $O(\log\log_{w}(\sigma+n/r))$ time we obtained in Lemma 1 to carry out the normal backward search operations on the RLFM-index. ∎

Lemma 2 gives us a toe-hold in the suffix array, and we show in this section that a toe-hold is all we need. We first show that, given the position and contents of one cell of the suffix array $\mathit{SA}$ of a text $T$, we can compute the contents of the neighbouring cells in $O(\log\log_{w}(n/r))$ time. It follows that, once we have counted the occurrences of a pattern in $T$, we can locate all the occurrences in $O(\log\log_{w}(n/r))$ time each.

Lemma 3

We can store $O(r)$ words such that, given $p$ and $\mathit{SA}[p]$, we can compute $\mathit{SA}[p-1]$ and $\mathit{SA}[p+1]$ in $O(\log\log_{w}(n/r))$ time.

Proof

We parse $T$ into phrases such that $T[i]$ is the first character in a phrase if and only if $i=1$ or $q=\mathit{SA}^{-1}[i+1]$ is the first or last position of a run in $\mathit{BWT}$ (i.e., $\mathit{BWT}[q]=T[i]$ starts or ends a run). We store an $O(r)$-space predecessor data structure with $O(\log\log_{w}(n/r))$ query time [10, Thm. 14] for the starting phrase positions in $T$ (i.e., the values $i$ just mentioned). We also store, associated with such values $i$ in the predecessor structure, the positions in $T$ of the characters immediately preceding and following $q$ in $\mathit{BWT}$, that is, $N[i]=\langle\mathit{SA}[q-1],\mathit{SA}[q+1]\rangle$.

Suppose we know $\mathit{SA}[p]=k+1$ and want to know $\mathit{SA}[p-1]$ and $\mathit{SA}[p+1]$. This is equivalent to knowing the position $\mathit{BWT}[p]=T[k]$ and wanting to know the positions in $T$ of $\mathit{BWT}[p-1]$ and $\mathit{BWT}[p+1]$. To compute these positions, we find with the predecessor data structure the position $i$ in $T$ of the first character of the phrase containing $T[k]$, take the associated positions $N[i]=\langle x,y\rangle$, and return $\mathit{SA}[p-1]=x+k-i$ and $\mathit{SA}[p+1]=y+k-i$.

To see why this works, let $\mathit{SA}[p-1]=j+1$ and $\mathit{SA}[p+1]=\ell+1$, that is, $j$ and $\ell$ are the positions in $T$ of $\mathit{BWT}[p-1]=T[j]$ and $\mathit{BWT}[p+1]=T[\ell]$. Note that, for all $0\leq t<k-i$, $T[k-t]$ is not the first nor the last character of a run in $\mathit{BWT}$. Thus, by definition of $\mathit{LF}$, $\mathit{LF}^{t}(p-1)$, $\mathit{LF}^{t}(p)$, and $\mathit{LF}^{t}(p+1)$, that is, the $\mathit{BWT}$ positions of $T[j-t]$, $T[k-t]$, and $T[\ell-t]$, are contiguous and within a single run, thus $T[j-t]=T[k-t]=T[\ell-t]$. Therefore, for $t=k-i-1$, $T[j-(k-i-1)]=T[i+1]=T[\ell-(k-i+1)]$ are contiguous in $\mathit{BWT}$, and thus a further $\mathit{LF}$ step yields that $\mathit{BWT}[q]=T[i]$ is immediately preceded and followed by $\mathit{BWT}[q-1]=T[j-(k-i)]$ and $\mathit{BWT}[q+1]=T[\ell-(k-i)]$. That is, $N[i]=\langle\mathit{SA}[q-1],\mathit{SA}[q+1]\rangle=\langle j-(k-i)+1,\ell-(k-i)+1\rangle$ and our answer is correct. ∎

The following lemma shows that the above technique can be generalized. The result is a space-time trade-off allowing us to list each occurrence in constant time at the expense of a slight increase in space usage.

Lemma 4

Let $s>0$. We can store a data structure of $O(rs)$ words such that, given $\mathit{SA}[p]$, we can compute $\mathit{SA}[p-i]$ and $\mathit{SA}[p+i]$ for $i=1,\dots,s^{\prime}$ and any $s^{\prime}\leq s$, in $O(\log\log_{w}(n/r)+s^{\prime})$ time.

Proof

Consider all $\mathit{BWT}$ positions $j_{1}<\dots<j_{t}$ that are at distance at most $s$ from a run border (we say that characters on run borders are at distance $1$), and let $W[1..t]$ be an array such that $W[k]$ is the text position corresponding to $j_{k}$, for $k=1,\dots,t$. Let now $j^{+}_{1}<\dots<j^{+}_{t^{+}}$ be the $\mathit{BWT}$ positions having a run border at most $s$ positions after them, and $j^{-}_{1}<\dots<j^{-}_{t^{-}}$ be the $\mathit{BWT}$ positions having a run border at most $s$ positions before them. We store the text positions corresponding to $j^{+}_{1}<\dots<j^{+}_{t^{+}}$ and $j^{-}_{1}<\dots<j^{-}_{t^{-}}$ in two predecessor structures $P^{+}$ and $P^{-}$, respectively, of size $O(rs)$. We store, for each $i\in P^{+}\cup P^{-}$, its position in $W$, that is, $W[f(i)]=i$.

To answer queries given $\mathit{SA}[p]$, we first compute its $P^{+}$-predecessor $i<\mathit{SA}[p]$ in $O(\log\log_{w}(n/r))$ time, and retrieve $f(i)$. Then, it holds that $\mathit{SA}[p+j]=W[f(i)+j]+(\mathit{SA}[p]-i)$, for $j=0,\dots,s$. Computing $\mathit{SA}[p-j]$ is symmetric (just use $P^{-}$ instead of $P^{+}$).

To see why this procedure is correct, consider the range $\mathit{SA}[p..p+s]$. We distinguish two cases.

(i) $\mathit{BWT}[p..p+s]$ contains at least two distinct characters. Then, $\mathit{SA}[p]-1$ is inside $P^{+}$ (because $p$ is followed by a run break at most $s$ positions away), and is therefore the immediate predecessor of $\mathit{SA}[p]$. Moreover, all $\mathit{BWT}$ positions $[p,p+s]$ are in $j_{1},\dots,j_{t}$ (since they are at distance at most $s$ from a run break), and their corresponding text positions are therefore contained in a contiguous range of $W$ (i.e., $W[f(\mathit{SA}[p]-1)..f(\mathit{SA}[p]-1)+s]$). The claim follows.

(ii) $\mathit{BWT}[p..p+s]$ contains a single character; we say it is unary. Then $\mathit{SA}[p]-1\notin P^{+}$, since there are no run breaks in $\mathit{BWT}[p..p+s]$. Moreover, by the $\mathit{LF}$ formula, the $\mathit{LF}$ mapping applied on the unary range $\mathit{BWT}[p..p+s]$ gives a contiguous range $\mathit{BWT}[\mathit{LF}(p)..\mathit{LF}(p+s)]=\mathit{BWT}[\mathit{LF}(p)..\mathit{LF}(p)+s]$. Note that this corresponds to a parallel backward step on text positions $\mathit{SA}[p]\rightarrow\mathit{SA}[p]-1,\dots,\mathit{SA}[p+s]\rightarrow\mathit{SA}[p+s]-1$. We iterate the application of $\mathit{LF}$ until we end up in a range $\mathit{BWT}[\mathit{LF}^{\delta}(p)..\mathit{LF}^{\delta}(p+s)]$ that is not unary. Then, $\mathit{SA}[\mathit{LF}^{\delta}(p)]-1$ is the immediate predecessor of $\mathit{SA}[p]$ in $P^{+}$, and $\delta$ is their distance (minus one). This means that with a single predecessor query on $P^{+}$ we “skip” all the unary $\mathit{BWT}$ ranges $\mathit{BWT}[\mathit{LF}^{i}(p)..\mathit{LF}^{i}(p+s)]$ for $i=1,\dots,\delta-1$ and, as in case (i), retrieve the contiguous range in $W$ containing the values $\mathit{SA}[p]-\delta,\dots,\mathit{SA}[p+s]-\delta$ and add $\delta$ to obtain the desired $\mathit{SA}$ values. ∎

Combining Lemmas 2 and 4, we obtain the main result of this section. The $O(\log\log_{w}(n/\sigma))$ additional time spent at locating is absorbed by the counting time.

Theorem 3.1

Let $s>0$. We can store a text $T[1..n]$, over alphabet $[1..\sigma]$, in $O(rs)$ words, where $r$ is the number of runs in the $\mathit{BWT}$ of $T$, such that later, given a pattern $P[1..m]$, we can count the occurrences of $P$ in $T$ in $O(m\log\log_{w}(\sigma+n/r))$ time and (after counting) report their $occ$ locations in overall time $O((1+\log\log_{w}(n/r)/s)\cdot occ)$.

In particular, we can locate in $O(m\log\log_{w}(\sigma+n/r)+occ\log\log_{w}(n/r))$ time and $O(r)$ space or, alternatively, in $O(m\log\log_{w}(\sigma+n/r)+occ)$ time and $O(r\log\log_{w}(n/r))$ space.

3.1 Accessing $\mathit{LCP}$

Lemma 4 can be further extended to entries of the $\mathit{LCP}$ array, which we will use later in the article. That is, given $\mathit{SA}[p]$, we compute $\mathit{LCP}[p]$ and its adjacent entries (note that we do not need to know $p$, but just $\mathit{SA}[p]$). The result is also an extension of a representation by Fischer et al. [33]. In Section 6.4 we use different structures that allow us access $\mathit{LCP}[p]$ directly, without knowing $\mathit{SA}[p]$.

Lemma 5

Let $s>0$. We can store a data structure of $O(rs)$ words such that, given $\mathit{SA}[p]$, we can compute $\mathit{LCP}[p-i+1]$ and $\mathit{LCP}[p+i]$, for $i=1,\dots,s^{\prime}$ and any $s^{\prime}\leq s$, in time $O(\log\log_{w}(n/r)+s^{\prime})$.

Proof

The proof follows closely that of Lemma 4, except that now we sample $\mathit{LCP}$ entries corresponding to suffixes following sampled $\mathit{BWT}$ positions. Let us define $j_{1}<\dots<j_{t}$, $j^{+}_{1}<\dots<j^{+}_{t^{+}}$, and $j^{-}_{1}<\dots<j^{-}_{t^{-}}$, as well as the predecessor structures $P^{+}$ and $P^{-}$, exactly as in the proof of Lemma 4. We store $\mathit{LCP}^{\prime}[1..t]=\mathit{LCP}[j_{1}],\dots,\mathit{LCP}[j_{t}]$. We also store, for each $i\in P^{+}\cup P^{-}$, its corresponding position $f(i)$ in $\mathit{LCP}^{\prime}$, that is, $\mathit{LCP}^{\prime}[f(i)]=\mathit{LCP}[\mathit{ISA}[i+1]]$.

To answer queries given $\mathit{SA}[p]$, we first compute its $P^{+}$-predecessor $i<\mathit{SA}[p]$ in $O(\log\log_{w}(n/r))$ time, and retrieve $f(i)$. Then, it holds that $\mathit{LCP}[p+j]=\mathit{LCP}^{\prime}[f(i)+j]-(\mathit{SA}[p]-i-1)$, for $j=1,\dots,s$. Computing $\mathit{LCP}[p-j]$ for $j=0,\dots,s-1$ is symmetric (just use $P^{-}$ instead of $P^{+}$).

To see why this procedure is correct, consider the range $\mathit{SA}[p..p+s]$. We distinguish two cases.

(i) $\mathit{BWT}[p..p+s]$ contains at least two distinct characters. Then, as in case (i) of Lemma 4, $\mathit{SA}[p]-1$ is inside $P^{+}$ and is therefore the immediate predecessor $i=\mathit{SA}[p]-1$ of $\mathit{SA}[p]$. Moreover, all $\mathit{BWT}$ positions $[p,p+s]$ are in $j_{1},\dots,j_{t}$, and therefore values $\mathit{LCP}[p..p+s]$ are explicitly stored in a contiguous range in $\mathit{LCP}^{\prime}$ (i.e., $\mathit{LCP}^{\prime}[f(i)..f(i)+s]$). Note that $(\mathit{SA}[p]-i)=1$, so $\mathit{LCP}^{\prime}[f(i)+j]-(\mathit{SA}[p]-i-1)=\mathit{LCP}^{\prime}[f(i)+j]$ for $j=0,\dots,s$. The claim follows.

(ii) $\mathit{BWT}[p..p+s]$ contains a single character; we say it is unary. Then we reason exactly as in case (ii) of Lemma 4 to define $\delta$ so that $i^{\prime}=\mathit{SA}[\mathit{LF}^{\delta}(p)]-1$ is the immediate predecessor of $\mathit{SA}[p]$ in $P^{+}$ and, as in case (i) of this proof, retrieve the contiguous range $\mathit{LCP}^{\prime}[f(i^{\prime})..f(i^{\prime})+s]$ containing the values $\mathit{LCP}[\mathit{LF}^{\delta}(p)..\mathit{LF}^{\delta}(p+s)]$. Since the skipped $\mathit{BWT}$ ranges are unary, it is then not hard to see that $\mathit{LCP}[\mathit{LF}^{\delta}(p+j)]=\mathit{LCP}[p+j]+\delta$ for $j=1,\dots,s$ (note that we do not include $s=0$ since we cannot exclude that, for some $i<\delta$, $LF^{i}(p)$ is the first position in its run). From the equality $\delta=\mathit{SA}[p]-i^{\prime}-1=\mathit{SA}[p]-\mathit{SA}[\mathit{LF}^{\delta}(p)]$ (that is, $\delta$ is the distance between $\mathit{SA}[p]$ and its predecessor minus one or, equivalently, the number of $\mathit{LF}$ steps virtually performed), we then compute $\mathit{LCP}[p+j]=\mathit{LCP}^{\prime}[f(i^{\prime})+j]-\delta$ for $j=1,\dots,s$. ∎

4 Extracting Substrings and Computing Fingerprints

In this section we consider the problem of extracting arbitrary substrings of $T[1..n]$. Though an obvious solution is to store a grammar-compressed version of $T$ [15], little is known about the relation between the size $g$ of the smallest grammar that generates $T$ (which nevertheless is NP-hard to find [18]) and the number of runs $r$ in its $\mathit{BWT}$ (but see Section 6.5). Another choice is to use block trees [7], which require $O(z\log(n/z))$ space, where $z$ is the size of the Lempel-Ziv parse [61] of $T$. Again, $z$ can be larger or smaller than $r$ [84].

Instead, we introduce a novel representation that uses $O(r\log(n/r))$ space and can retrieve any substring of length $\ell$ from $T$ in time $O(\log(n/r)+\ell\log(\sigma)/w)$. This is similar (though incomparable) with the $O(\log(n/g)+\ell/\log_{\sigma}n)$ time that could be obtained with grammar compression [15, 11], and with the $O(\log(n/z)+\ell/\log_{\sigma}n)$ that could be obtained with block trees. In Section 6.5 we obtain a run-length context-free grammar of asymptotically the same size, $O(r\log(n/r))$, which extracts substrings in time $O(\log(n/r)+\ell)$. The bounds we obtain in this section are thus better. Also, as explained in the Introduction, the $O(\log(n/r))$ additive penalty is near-optimal in general.

We first prove an important result in Lemma 6: any desired substring of $T$ has a primary occurrence, that is, one overlapping a border between phrases. The property is indeed stronger than in alternative formulations that hold for Lempel-Ziv parses [53] or grammars [22]: If we choose a primary occurrence overlapping at its leftmost position, then all the other occurrences of the string suffix must be preceded by the same prefix. This stronger property is crucial to design an optimal locating procedure in Section 5 and an optimal counting procedure in Section 8. The weaker property, instead, is sufficient to design in Theorem 4.1 a data structure reminiscent of block trees [7] for extracting substrings of $T$, which needs to store only some text around phrase borders. Finally, in Lemma 7 we show that a Karp-Rabin fingerprint [54, 36] of any substring of $T$ can be obtained in time $O(\log(n/r))$, which will also be used in Section 5.

Definition 1

We say that a text character $T[i]$ is sampled if and only if $T[i]$ is the first or last character in its $\mathit{BWT}$ run.

Definition 2

We say that a text substring $T[i..j]$ is primary if and only if it contains at least one sampled character.

Lemma 6

Every text substring $T[i..j]$ has a primary occurrence $T[i^{\prime}..j^{\prime}]=T[i..j]$ such that, for some $i^{\prime}\leq p\leq j^{\prime}$, the following hold:

$T[p]$* is sampled.* 2. 2.

$T[i^{\prime}],\dots,T[p-1]$* are not sampled.* 3. 3.

Every text occurrence of $T[p..j^{\prime}]$ is always preceded by the string $T[i^{\prime}..p-1]$.

Proof

We prove the lemma by induction on $j-i$. If $j-i=0$, then $T[i..j]$ is a single character. Every character has a sampled occurrence $i^{\prime}$ in the text, therefore the three properties trivially hold for $p=i^{\prime}$.

Let $j-i>0$. By the inductive hypothesis, $T[i+1..j]$ has an occurrence $T[i^{\prime}+1..j^{\prime}]$ satisfying the three properties for some $i^{\prime}+1\leq p\leq j^{\prime}$. Let $[sp,ep]$ be the $\mathit{BWT}$ range of $T[i+1..j]$. We distinguish three cases.

(i) All characters in $\mathit{BWT}[sp,ep]$ are equal to $T[i]=T[i^{\prime}]$ and are not the first or last in their run. Then, we leave $p$ unchanged. $T[p]$ is sampled by the inductive hypothesis, so Property $1$ still holds. Also, $T[i^{\prime}+1],\dots,T[p-1]$ are not sampled by the inductive hypothesis, and $T[i^{\prime}]$ is not sampled by assumption, so Property $2$ still holds. By the inductive hypothesis, every text occurrence of $T[p..j^{\prime}]$ is always preceded by the string $T[i^{\prime}+1..p-1]$. Since all characters in $\mathit{BWT}[sp,ep]$ are equal to $T[i]=T[i^{\prime}]$, Property $3$ also holds for $T[i..j]$ and $p$.

(ii) All characters in $\mathit{BWT}[sp,ep]$ are equal to $T[i]$ and either $\mathit{BWT}[sp]$ is the first character in its run, or $\mathit{BWT}[ep]$ is the last character in its run (or both). Then, we set $p$ to the text position corresponding to $sp$ or $ep$, depending on which one is sampled (if both are sampled, choose $sp$). The three properties then hold trivially for $T[i..j]$ and $p$.

(iii) $\mathit{BWT}[sp,ep]$ contains at least one character $c\neq T[i]$. Then, there must be a run of $T[i]$’s ending or beginning in $\mathit{BWT}[sp,ep]$, meaning that there is a $sp\leq q\leq ep$ such that $\mathit{BWT}[q]=T[i]$ and the text position $i^{\prime}$ corresponding to $q$ is sampled. We then set $p=i^{\prime}$. Again, the three properties hold trivially for $T[i..j]$ and $p$. ∎

Lemma 6 has several important implications. We start by using it to build a data structure supporting efficient text extraction queries. In Section 5 we will use it to locate pattern occurrences in optimal time.

Theorem 4.1

Let $T[1..n]$ be a text over alphabet $[1..\sigma]$. We can store a data structure of $O(r\log(n/r))$ words supporting the extraction of any length-$\ell$ substring of $T$ in $O(\log(n/r)+\ell\log(\sigma)/w)$ time.

Proof

We describe a data structure supporting the extraction of $\alpha=\frac{w\log(n/r)}{\log\sigma}$ packed characters in $O(\log(n/r))$ time. To extract a text substring of length $\ell$ we divide it into $\lceil\ell/\alpha\rceil$ blocks and extract each block with the proposed data structure. Overall, this will take $O((\ell/\alpha+1)\log(n/r))=O(\log(n/r)+\ell\log(\sigma)/w)$ time.

Our data structure is stored in $O(\log(n/r))$ levels. For simplicity, we assume that $r$ divides $n$ and that $n/r$ is a power of two. The top level (level 0) is special: we divide the text into $r$ blocks $T[1..n/r]\,T[n/r+1..2n/r]\dots T[n-n/r+1..n]$ of size $n/r$. For levels $i>0$, we let $s_{i}=n/(r\cdot 2^{i-1})$ and, for every sampled position $j$ (Definition 1), we consider the two non-overlapping blocks of length $s_{i}$: $X_{i,j}^{1}=T[j-s_{i}..j-1]$ and $X_{i,j}^{2}=T[j..j+s_{i}-1]$. Each such block $X^{k}_{i,j}$, for $k=1,2$, is composed of two half-blocks, $X^{k}_{i,j}=X^{k}_{i,j}[1..s_{i}/2]\,X^{k}_{i,j}[s_{i}/2+1..s_{i}]$. We moreover consider three additional consecutive and non-overlapping half-blocks, starting in the middle of the first, $X^{1}_{i,j}[1..s_{i}/2]$, and ending in the middle of the last, $X^{2}_{i,j}[s_{i}/2+1..s_{i}]$, of the 4 half-blocks just described: $T[j-s_{i}+s_{i}/4..j-s_{i}/4-1],\ T[j-s_{i}/4..j+s_{i}/4-1]$, and $T[j+s_{i}/4..j+s_{i}-s_{i}/4-1]$.

From Lemma 6, blocks at level [math] and each half-block at level $i>0$ have a primary occurrence at level $i+1$. Such an occurrence can be fully identified by the coordinate $\langle\mathit{off},j^{\prime}\rangle$, for $0<\mathit{off}\leq s_{i+1}$ and $j^{\prime}$ sampled position, indicating that the occurrence starts at position $j^{\prime}-s_{i+1}+\mathit{off}$.

Let $i^{*}$ be the smallest number such that $s_{i^{*}}<4\alpha=\frac{4w\log(n/r)}{\log\sigma}$. Then $i^{*}$ is the last level of our structure. At this level, we explicitly store a packed string with the characters of the blocks. This uses in total $O(r\cdot s_{i^{*}}\log(\sigma)/w)=O(r\log(n/r))$ words of space. All the blocks at level 0 and half-block at levels $0<i<i^{*}$ store instead the coordinates $\langle\mathit{off},j^{\prime}\rangle$ of their primary occurrence in the next level. At level $i^{*}-1$, these coordinates point inside the strings of explicitly stored characters.

Let $S=T[i..i+\alpha-1]$ be the text substring to be extracted. Note that we can assume $n/r\geq\alpha$; otherwise all the text can be stored in plain packed form using $n\log(\sigma)/w<\alpha r\log(\sigma)/w\in O(r\log(n/r))$ words and we do not need any data structure. It follows that $S$ either spans two blocks at level 0, or it is contained in a single block. The former case can be solved with two queries of the latter, so we assume, without losing generality, that $S$ is fully contained inside a block at level [math]. To retrieve $S$, we map it down to the next levels (using the stored coordinates of primary occurrences of half-blocks) as a contiguous text substring as long as this is possible, that is, as long as it fits inside a single half-block. Note that, thanks to the way half-blocks overlap, this is always possible as long as $\alpha\leq s_{i}/4$. By definition, then, we arrive in this way precisely to level $i^{*}$, where characters are stored explicitly and we can return the packed text substring. ∎

Using a similar idea, we can compute the Karp-Rabin fingerprint of any text substring in just $O(\log(n/r))$ time. This will be used in Section 5 to obtain our optimal-time locate solution.

Lemma 7

We can store a data structure of $O(r\log(n/r))$ words supporting computation of the Karp-Rabin fingerprint of any text substring in $O(\log(n/r))$ time.

Proof

We store a data structure with $O(\log(n/r))$ levels, similar to the one of Theorem 4.1 but with two non-overlapping children blocks. Assume again that $r$ divides $n$ and that $n/r$ is a power of two. The top level 0 divides the text into $r$ blocks $T[1..n/r]\,T[n/r+1..2n/r]\dots T[n-n/r+1..n]$ of size $n/r$. For levels $i>0$, we let $s_{i}=n/(r\cdot 2^{i-1})$ and, for every sampled position $j$, we consider the two non-overlapping blocks of length $s_{i}$: $X_{i,j}^{1}=T[j-s_{i}..j-1]$ and $X_{i,j}^{2}=T[j..j+s_{i}-1]$. Each such block $X^{k}_{i,j}$ is composed of two half-blocks, $X^{k}_{i,j}=X^{k}_{i,j}[1..s_{i}/2]\,X^{k}_{i,j}[s_{i}/2+1..s_{i}]$. As in Theorem 4.1, blocks at level [math] and each half-block at level $i>0$ have a primary occurrence at level $i+1$, meaning that such an occurrence can be written as $X_{i+1,j^{\prime}}^{1}[L..s_{i+1}]\,X_{i+1,j^{\prime}}^{2}[1..R]$ for some $1\leq L,R\leq s_{i+1}$, and some sampled position $j^{\prime}$ (the special case where the half-block is equal to $X_{i+1,j^{\prime}}^{2}$ is expressed as $L=s_{i+1}+1$ and $R=s_{i+1}$).

We associate with every block at level 0 and every half-block at level $i>0$ the following information: its Karp-Rabin fingerprint $\kappa$, the coordinates $\langle j^{\prime},L\rangle$ of its primary occurrence in the next level, and the Karp-Rabin fingerprints $\kappa(X_{i+1,j^{\prime}}^{1}[L..s_{i+1}])$ and $\kappa(X_{i+1,j^{\prime}}^{2}[1..R])$ of (the two pieces of) its occurrence. At level [math], we also store the Karp-Rabin fingerprint of every text prefix ending at block boundaries, $\kappa(T[1..jr])$ for $j=1,\ldots,n/r$. At the last level, where blocks are of length 1, we only store their Karp-Rabin fingerprint (or we may compute them on the fly).

To answer queries $\kappa(T[i..j])$ quickly, the key point is to show that computing the Karp-Rabin fingerprint of a prefix or a suffix of a block translates into the same problem (prefix/suffix of a block) in the next level, and therefore leads to a single-path descent in the block structure. To prove this, consider computing the fingerprint of the prefix $\kappa(X_{i,j}^{k}[1..R^{\prime}])$ of some block (computing suffixes is symmetric). Note that we explicitly store $\kappa(X_{i,j}^{k}[1..s_{i}/2])$, so we can consider only the problem of computing the fingerprint of a prefix of a half-block, that is, we assume $R^{\prime}\leq s_{i}/2=s_{i+1}$ (the proof is the same for the right half of $X_{i,j}^{k}$). Let $X_{i+1,j^{\prime}}^{1}[L..s_{i+1}]\,X_{i+1,j^{\prime}}^{2}[1..R]$ be the occurrence of the half-block in the next level. We have two cases. (i) $R^{\prime}\geq s_{i+1}-L+1$. Then, $X_{i,j}^{k}[1..R^{\prime}]=X_{i+1,j^{\prime}}^{1}[L..s_{i+1}]\,X_{i+1,j^{\prime}}^{2}[1..R^{\prime}-(s_{i+1}-L+1)]$. Since we explicitly store the fingerprint $\kappa(X_{i+1,j^{\prime}}^{1}[L..s_{i+1}])$, the problem reduces to computing the fingerprint of the block prefix $X_{i+1,j^{\prime}}^{2}[1..R^{\prime}-(s_{i+1}-L+1)]$. (ii) $R^{\prime}<s_{i+1}-L+1$. Then, $X_{i,j}^{k}[1..R^{\prime}]=X_{i+1,j^{\prime}}^{1}[L..L+R^{\prime}-1]$. Even though this is not a prefix nor a suffix of a block, note that $X_{i+1,j^{\prime}}^{1}[L..s_{i+1}]=X_{i+1,j^{\prime}}^{1}[L..L+R^{\prime}-1]\,X_{i+1,j^{\prime}}^{1}[L+R^{\prime}..s_{i+1}]$. We explicitly store the fingerprint of the left-hand side of this equation, so the problem reduces to finding the fingerprint of $X_{i+1,j^{\prime}}^{k}[L+R^{\prime}..s_{i+1}]$, which is a suffix of a block. From both fingerprints we can compute $\kappa(X_{i+1,j^{\prime}}^{1}[L..L+R^{\prime}-1])$.

Note that, in order to combine fingerprints, we also need the corresponding exponents and their inverses (i.e., $\sigma^{\pm\ell}\!\!\mod q$, where $\ell$ is the string length and $q$ is the prime used in $\kappa$). We store the exponents associated with the lengths of the explicitly stored fingerprints at all levels. The remaining exponents needed for the calculations can be retrieved by combining exponents from the next level (with a plain modular multiplication) in the same way we retrieve fingerprints by combining partial results from next levels.

To find the fingerprint of any text substring $T[i..j]$, we proceed as follows. If $T[i..j]$ spans at least two blocks at level 0, then $T[i..j]$ can be factored into $(a)$ a suffix of a block, $(b)$ a central part (possibly empty) of full blocks, and $(c)$ a prefix of a block. Since at level [math] we store the Karp-Rabin fingerprint of every text prefix ending at block boundaries, the fingerprint of $(b)$ can be found in constant time. Computing the fingerprints of $(a)$ and $(c)$, as proved above, requires only a single-path descent in the block structure, taking $O(\log(n/r))$ time each. If $T[i..j]$ is fully contained in a block at level 0, then we map it down to the next levels until it spans two blocks. From this point, the problem translates into a prefix/suffix problem, which can be solved in $O(\log(n/r))$ time. ∎

5 Locating in Optimal Time

In this section we show how to obtain optimal locating time in the unpacked — $O(m+occ)$ — and packed — $O(m\log(\sigma)/w+occ)$ — scenarios, by using $O(r\log(n/r))$ and $O(rw\log_{\sigma}(n/r))$ space, respectively. To improve upon the times of Theorem 3.1 we have to abandon the idea of using the RLFM-index to find the toe-hold suffix array entry, as counting on the RLFM-index takes $\omega(m)$ time. We will use a different machinery that, albeit conceptually based on the $\mathit{BWT}$ properties, does not use it at all. We exploit the idea that some pattern occurrence must cross a run boundary to build a structure that only finds pattern suffixes starting at a run boundary. By sampling more text suffixes around those run boundaries, we manage to find one pattern occurrence in time $O(m+\log(n/r))$, Lemma 10. We then show how the $\mathit{LCP}$ information we obtained in Section 3.1 can be used to extract all the occurrences in time $O(m+occ+\log(n/r))$, in Lemma 12. Finally, by adding a structure that finds faster the patterns shorter than $\log(n/r)$, we obtain the unpacked result in Theorem 5.1. We use the same techniques, but with larger structures, in the packed setting, Theorem 5.2.

We make use of Lemma 6: if the pattern $P[1..m]$ occurs in the text then there must exist an integer $1\leq p\leq m$ such that (1) $P[p..m]$ prefixes a text suffix $T[i+p-1..]$, where $T[i+p-1]$ is sampled, (2) none of the characters $T[i],\dots,T[i+p-2]$ are sampled, and (3) $P[p..m]$ is always preceded by $P[1..p-1]$ in the text. It follows that $T[i..i+m-1]=P$. This implies that we can locate a pattern occurrence by finding the longest pattern suffix prefixing some text suffix that starts with a sampled character. Indeed, those properties are preserved if we enlarge the sampling.

Lemma 8

Lemma 6 still holds if we add arbitrary sampled positions to the original sampling.

Proof

If the leftmost sampled position $T[i+p-1]$ in the pattern occurrence belongs to the original sampling, then the properties hold by Lemma 6. If, on the other hand, $T[i+p-1]$ is one of the extra samples we added, then let $i^{\prime}+p^{\prime}-1$ be the position of the original sampling satisfying the three properties, with $p^{\prime}\geq p$. Properties (1) and (2) hold for the sampled position $i+p-1$ by definition. By property (3) applied to $i^{\prime}+p^{\prime}-1$, we have that $P[p^{\prime}..m]$ is always preceded by $P[1..p^{\prime}-1]$ in the text. Since $p^{\prime}\geq p$, it follows that also $P[p..m]$ is always preceded by $P[1..p-1]$ in the text, that is, property (3) holds for position $i+p-1$ as well. ∎

We therefore add to the sampling the $r$ equally-spaced extra text positions $i\cdot(n/r)+1$, for $i=0,\dots,r-1$; we now have at most $3r$ sampled positions. The task of finding a pattern occurrence satisfying properties (1)–(3) on the extended sampling can be efficiently solved by inserting all the text suffixes starting with a sampled character in a data structure supporting fast prefix search operations and taking $O(r)$ words (e.g., a z-fast trie [3]). We make use of the following lemma.

Lemma 9 ([36, 3])

Let $\mathcal{S}$ be a set of strings and assume we have some data structure supporting extraction of any length-$\ell$ substring of strings in $\mathcal{S}$ in time $f_{e}(\ell)$ and computation of the Karp-Rabin fingerprint of any substring of strings in $\mathcal{S}$ in time $f_{h}$. We can build a data structure of $O(|\mathcal{S}|)$ words such that, later, we can solve the following problem in $O(m\log(\sigma)/w+t(f_{h}+\log m)+f_{e}(m))$ time: given a pattern $P[1..m]$ and $t>0$ suffixes $Q_{1},\dots,Q_{t}$ of $P$, discover the ranges of strings in (the lexicographically-sorted) $\mathcal{S}$ prefixed by $Q_{1},\dots,Q_{t}$.

Proof

Z-fast tries [3, App. H.3] already solve the weak part of the lemma in $O(m\log(\sigma)/w+t\log m)$ time. By weak we mean that the returned answer for suffix $Q_{i}$ is not guaranteed to be correct if $Q_{i}$ does not prefix any string in $\mathcal{S}$: we could therefore have false positives among the answers, but false negatives cannot occur. A procedure for deterministically discarding false positives has already been proposed [36] and requires extracting substrings and their fingerprints from $\mathcal{S}$. We describe this strategy in detail in order to analyze its time complexity in our scenario.

First, we require the Karp-Rabin function $\kappa$ to be collision-free between equal-length text substrings whose length is a power of two. We can find such a function at index-construction time in $O(n\log n)$ expected time and $O(n)$ space [14]. We extend the collision-free property to pairs of equal-letter strings of general length switching to the hash function $\kappa^{\prime}$ defined as $\kappa^{\prime}(T[i..i+\ell-1])=\langle\kappa(T[i..i+2^{\lfloor\log_{2}\ell\rfloor}-1]),\kappa(T[i+\ell-2^{\lfloor\log_{2}\ell\rfloor}..i+\ell-1])\rangle$. Let $Q_{1},\dots,Q_{j}$ be the pattern suffixes for which the prefix search found a candidate node. Order the pattern suffixes so that $|Q_{1}|<\dots<|Q_{j}|$, that is, $Q_{i}$ is a suffix of $Q_{i^{\prime}}$ whenever $i<i^{\prime}$. Let moreover $v_{1},\dots,v_{j}$ be the candidate nodes (explicit or implicit) of the z-fast trie such all substrings below them are prefixed by $Q_{1},\dots,Q_{j}$ (modulo false positives), respectively, and let $t_{i}=string(v_{i})$ be the substring read from the root of the trie to $v_{i}$. Our goal is to discard all nodes $v_{k}$ such that $t_{k}\neq Q_{k}$.

We compute the $\kappa^{\prime}$-signatures of all candidate pattern suffixes $Q_{1},\dots,Q_{t}$ in $O(m\log(\sigma)/w+t)$ time. We proceed in rounds. At the beginning, let $a=1$ and $b=2$. At each round, we perform the following checks:

If $\kappa^{\prime}(Q_{a})\neq\kappa^{\prime}(t_{a})$: discard $v_{a}$ and set $a\leftarrow a+1$ and $b\leftarrow b+1$. 2. 2.

If $\kappa^{\prime}(Q_{a})=\kappa^{\prime}(t_{a})$: let $R$ be the length-$|t_{a}|$ suffix of $t_{b}$, i.e. $R=t_{b}[|t_{b}|-|t_{a}|+1..|t_{b}|]$. We have two sub-cases:

1. (a)

$\kappa^{\prime}(Q_{a})=\kappa^{\prime}(R)$. Then, we set $b\leftarrow b+1$ and $a$ to the next integer $a^{\prime}$ such that $v_{a^{\prime}}$ has not been discarded. 2. (b)

$\kappa^{\prime}(Q_{a})\neq\kappa^{\prime}(R)$. Then, discard $v_{b}$ and set $b\leftarrow b+1$. 3. 3.

If $b=j+1$: let $v_{f}$ be the last node that was not discarded. Note that $Q_{f}$ is the longest pattern suffix that was not discarded; other non-discarded pattern suffixes are suffixes of $Q_{f}$. We extract $t_{f}$. Let $s$ be the length of the longest common suffix between $Q_{f}$ and $t_{f}$. We report as a true match all nodes $v_{i}$ that were not discarded in the above procedure and such that $|Q_{i}|\leq s$.

Intuitively, the above procedure is correct because we deterministically check that text substrings read from the root to the candidate nodes form a monotonically increasing sequence according to the suffix relation: $t_{i}\subseteq_{suf}t_{i^{\prime}}$ for $i<i^{\prime}$ (if the relation fails at some step, we discard the failing node). Comparisons to the pattern are delegated to the last step, where we explicitly compare the longest matching pattern suffix with $t_{f}$. For a full formal proof, see Gagie et al. [36].

For every candidate node we compute a $\kappa^{\prime}$-signature from the set of strings ($O(f_{h})$ time). For the last candidate, we extract a substring of length at most $m$ ($O(f_{e}(m))$ time) and compare it with the longest candidate pattern suffix ($O(m\log(\sigma)/w)$ time). There are at most $t$ candidates, so the verification process takes $O(m\log(\sigma)/w+t\cdot f_{h}+f_{e}(m))$. Added to the time spent to find the candidates in the z-fast trie, we obtain the claimed bounds. ∎

In our case, we use the results stated in Theorem 4.1 and Lemma 7 to extract text substrings and their fingerprints, so we get $f_{e}(m)=O(\log(n/r)+m\log(\sigma)/w)$ and $f_{h}=O(\log(n/r))$. Moreover note that, by the way we added the $r$ equally-spaced extra text samples, if $m\geq n/r$ then the position $p$ satisfying Lemma 8 must occur in the prefix of length $n/r$ of the pattern. It follows that, for long patterns, it is sufficient to search the prefix data structure for only the $t=n/r$ longest pattern suffixes. We can therefore solve the problem stated in Lemma 9 in time $O(m\log(\sigma)/w+\min(m,n/r)(\log(n/r)+\log m))$. Note that, while the fingerprints are obtained with a randomized method, the resulting data structure offers deterministic worst-case query times and cannot fail.

To further speed up operations, for every sampled character $T[i]$ we insert in $\mathcal{S}$ the text suffixes $T[i-j..]$ for $j=0..\tau-1$, for some parameter $\tau$ that we determine later. This increases the size of the prefix-search structure to $O(r\,\tau)$ (excluding the components for extracting substrings and fingerprints), but in exchange it is sufficient to search only for aligned pattern suffixes of the form $P[\ell\cdot\tau+1..m]$, for $\ell=0\ldots\lceil\min(m,n/r)/\tau\rceil-1$, to find any primary occurrence: to find the longest suffix of $P$ that prefixes a string in $\mathcal{S}$, we keep an array $B[1..|\mathcal{S}|]$ storing the shift relative to each element in $\mathcal{S}$; for every sampled $T[i]$ and $j=0\ldots\tau-1$, if $k$ is the rank of $T[i-j..]$ among all suffixes in $\mathcal{S}$, then $B[k]=j$. We build a constant-time range minimum data structure on $B$, which requires only $O(|\mathcal{S}|)=O(r\,\tau)$ bits [32]. Let $[L,R]$ be the lexicographical range of suffixes in $\mathcal{S}$ prefixed by the longest aligned suffix of $P$ that has occurrences in $\mathcal{S}$. With a range minimum query on $B$ in the interval $[L,R]$ we find a text suffix with minimum shift, thereby matching the longest suffix of $P$. By Lemma 8, if $P$ occurs in $T$ then the remaining prefix of $P$ appears to the left of the longest suffix found. However, if $P$ does not occur in $T$ this is not the case. We therefore verify the candidate occurrence of $P$ using Theorem 4.1 in time $f_{e}(m)=O(\log(n/r)+m\log(\sigma)/w)$.

Overall, we find one pattern occurrence in $O(m\log(\sigma)/w+(\min(m,n/r)/\tau+1)(\log m+\log(n/r)))$ time. By setting $\tau=\log(n/r)$, we obtain the following result.

Lemma 10

We can find one pattern occurrence in time $O(m+\log(n/r))$ with a structure using $O(r\log(n/r))$ space.

Proof

If $m<n/r$, then it is easy to verify that $O(m\log(\sigma)/w+(\min(m,n/r)/\tau+1)(\log m+\log(n/r)))=O(m+\log(n/r))$. If $m\geq n/r$, the running time is $O(m\log(\sigma)/w+((n/r)/\log(n/r)+1)\log m)$. The claim follows by noticing that $(n/r)/\log(n/r)=O(m/\log m)$, as $x/\log x=\omega(1)$. ∎

If we choose, instead, $\tau=w\log_{\sigma}(n/r)$, we can approach optimal-time locate in the packed setting.

Lemma 11

We can find one pattern occurrence in time $O(m\log(\sigma)/w+\log(n/r))$ with a structure using $O(rw\log_{\sigma}(n/r))$ space.

Proof

We follow the proof of Lemma 10. The main difference is that, when $m\geq n/r$, we end up with time $O(m\log(\sigma)/w+\log m)$, which is $O(m)$ but not necessarily $O(m\log(\sigma)/w)$. However, if $m\log(\sigma)/w=o(\log m)$, then $m/\log m=o(w/\log\sigma)$ and thus $(n/r)/\log(n/r)=o(w/\log\sigma)$. The space we use, $O(rw\log_{\sigma}(n/r))$, is therefore $\omega(n)$, within which we can include a classical structure that finds one occurrence in $O(m\log(\sigma)/w)$ time (see, e.g., Belazzougui et al. [3, Sec. 7.1]). ∎

Let us now consider how to find the other occurrences. Note that, differently from Section 3, at this point we know the position of one pattern occurrence but we do not know its relative position in the suffix array nor the $\mathit{BWT}$ range of the pattern. In other words, we can extract adjacent suffix array entries using Lemma 4, but we do not know where we are in the suffix array. More critically, we do not know when to stop extracting adjacent suffix array entries. We can solve this problem using $\mathit{LCP}$ information extracted with Lemma 5: it is sufficient to continue extraction of candidate occurrences and corresponding $\mathit{LCP}$ values (in both directions) as long as the $\mathit{LCP}$ is greater than or equal to $m$. It follows that, after finding the first occurrence of $P$, we can locate the remaining ones in $O(occ+\log\log_{w}(n/r))$ time using Lemmas 4 and 5 (with $s=\log\log_{w}(n/r)$). This yields two first results with a logarithmic additive term over the optimal time.

Lemma 12

We can find all the $occ$ pattern occurrences in time $O(m+occ+\log(n/r))$ with a structure using $O(r\log(n/r))$ space.

Lemma 13

We can find all the $occ$ pattern occurrences in time $O(m\log(\sigma)/w+occ+\log(n/r))$ with a structure using $O(rw\log_{\sigma}(n/r))$ space.

To achieve the optimal running time, we must speed up the search for patterns that are shorter than $\log(n/r)$ (Lemma 12) and $w\log_{\sigma}n$ (Lemma 13). We index all the possible short patterns by exploiting the following property.

Lemma 14

There are at most $2rk$ distinct $k$-mers in the text, for any $1\leq k\leq n$.

Proof

From Lemma 6, every distinct $k$-mer appearing in the text has a primary occurrence. It follows that, in order to count the number of distinct $k$-mers, we can restrict our attention to the regions of size $2k-1$ overlapping the at most $2r$ sampled positions (Definition 1). The claim easily follows. ∎

Note that, without Lemma 14, we would only be able to bound the number of distinct $k$-mers by $\sigma^{k}$. We first consider achieving optimal locate time in the unpacked setting.

Theorem 5.1

We can store a text $T[1..n]$ in $O(r\log(n/r))$ words, where $r$ is the number of runs in the $\mathit{BWT}$ of $T$, such that later, given a pattern $P[1..m]$, we can report the $occ$ occurrences of $P$ in optimal $O(m+occ)$ time.

Proof

We store in a path-compressed trie $\mathcal{T}$ all the strings of length $\log(n/r)$ occurring in the text. By Lemma 14, $\mathcal{T}$ has $O(r\log(n/r))$ leaves, and since it is path-compressed, it has $O(r\log(n/r))$ nodes. The texts labeling the edges are represented with offsets pointing inside $r$ strings of length $2\log(n/r)$ extracted around each run boundary and stored in plain form (taking care of possible overlaps). Child operations on the trie are implemented with perfect hashing to support constant-time traversal.

In addition, we use the sampling structure of Lemma 4 with $s=\log\log_{w}(n/r)$. Recall from Lemma 4 that we store an array $W$ such that, given any range $\mathit{SA}[sp..sp+s-1]$, there exists a range $W[i..i+s-1]$ and an integer $\delta$ such that $\mathit{SA}[sp+j]=W[i+j]+\delta$, for $j=0,\dots,s-1$. We store this information on $\mathcal{T}$ nodes: for each node $v\in\mathcal{T}$, whose string prefixes the range of suffixes $\mathit{SA}[sp..ep]$, we store in $v$ the triple $\langle ep-sp+1,i,\delta\rangle$ such that $\mathit{SA}[sp+j]=W[i+j]+\delta$, for $j=0,\dots,s-1$.

Our complete locate strategy is as follows. If $m>\log(n/r)$, then we use the structures of Lemma 12, which already gives us $O(m+occ)$ time. Otherwise, we search for the pattern in $\mathcal{T}$. If $P$ does not occur in $\mathcal{T}$, then its number of occurrences must be zero, and we stop. If it occurs and the locus node of $P$ is $v$, let $\langle occ,i,\delta\rangle$ be the triple associated with $v$. If $occ\leq s=\log\log_{w}(n/r)$, then we obtain the whole interval $\mathit{SA}[sp..ep]$ in time $O(occ)$ by accessing $W[i,i+occ-1]$ and adding $\delta$ to the results. Otherwise, if $occ>s$, a plain application of Lemma 4 starting from the pattern occurrence $W[i]+\delta$ yields time $O(\log\log_{w}(n/r)+occ)=O(occ)$. Thus we obtain $O(m+occ)$ time and the trie uses $O(r\log(n/r))$ space. Considering all the structures, we obtain the main result. ∎

With more space, we can achieve optimal locate time in the packed setting.

Theorem 5.2

We can store a text $T[1..n]$ over alphabet $[1..\sigma]$ in $O(rw\log_{\sigma}(n/r))$ words, where $r$ is the number of runs in the $\mathit{BWT}$ of $T$, such that later, given a packed pattern $P[1..m]$, we can report the $occ$ occurrences of $P$ in optimal $O(m\log(\sigma)/w+occ)$ time.

Proof

As in the proof of Theorem 5.1 we need to index all the short patterns, in this case of length at most $\ell=w\log_{\sigma}(n/r)$. We insert all the short text substrings in a z-fast trie to achieve optimal-time navigation. As opposed to the z-fast trie used in Lemma 9, now we need to perform the trie navigation (i.e., a prefix search) in only $O(m\log(\sigma)/w)$ time, that is, we need to avoid the additive term $O(\log m)$ that was instead allowed in Lemma 9, as it could be larger than $m\log(\sigma)/w$ for very short patterns. We exploit a result by Belazzougui et al. [3, Sec. H.2]: letting $n^{\prime}$ be the number of indexed strings of average length $\ell$, we can support weak prefix search in optimal $O(m\log(\sigma)/w)$ time with a data structure of size $O(n^{\prime}\ell^{1/c}(\log\ell+\log\log n))$ bits, for any constant $c$. Note that, since $\ell=O(w^{2})$, this is $O(n^{\prime})$ space for any $c>2$. We insert in this structure all $n^{\prime}$ text $\ell$-mers. For Lemma 14, $n^{\prime}=O(r\ell)$. It follows that the prefix-search data structure takes space $O(r\ell)=O(rw\log_{\sigma}(n/r))$. This space is asymptotically the same of Lemma 13, which we use to find long patterns. We store in a packed string $V$ the contexts of length $\ell$ surrounding sampled text positions ($O(rw\log_{\sigma}(n/r))$ space); z-fast trie nodes point to their corresponding substrings in $V$. After finding the candidate node on the z-fast trie, we verify it in $O(m\log(\sigma)/w)$ time by extracting a substring from $V$. We augment each trie node as done in Theorem 5.1 with triples $\langle occ,i,\delta\rangle$. The locate procedure works as for Theorem 5.1, except that now we use the z-fast trie mechanism to navigate the trie of all short patterns.∎

6 Accessing the Suffix Array and Related Structures

In this section we show how we can provide direct access to the suffix array $\mathit{SA}$, its inverse $\mathit{ISA}$, and the longest common prefix array $\mathit{LCP}$. Those enable functionalities that go beyond the basic counting, locating, and extracting that are required for self-indexes, which we covered in Sections 3 to 5, and will be used to enable a full-fledged compressed suffix tree in Section 7.

We exploit the fact that the runs that appear in $\mathit{BWT}$ induce equal substrings in the differential suffix array, its inverse, and longest common prefix arrays, $\mathit{DSA}$, $\mathit{DISA}$, and $\mathit{DLCP}$, where we store the difference between each cell and the previous one. Those equal substrings are exploited to grammar-compress the differential arrays. We choose a particular class of grammar compressor that we will prove to produce a grammar of size $O(r\log(n/r))$, on which we can access cells in time $O(\log(n/r))$. The grammar is of an extended type called run-length context-free grammar (RLCFG) [80], which allows rules of the form $X\rightarrow Y^{t}$ that count as size 1. The grammar is based on applying several rounds of a technique called locally consistent parsing, which parses the string into small blocks (which then become nonterminals) in a way that equal substrings are parsed in the same form, therefore not increasing the grammar size. There are various techniques to achieve such a parsing [2], from which we choose a recent one [49, 48] that gives us the best results.

6.1 Compressing with Locally Consistent Parsing

We plan to grammar-compress sequences $W=\mathit{DSA}$, $\mathit{ISA}$, or $\mathit{LCP}$ by taking advantage of the runs in $\mathit{BWT}$. We will apply a locally consistent parsing [49, 48] to obtain a first partition of $W$ into nonterminals, and then recurse on the sequence of nonterminals. We will show that the interaction between the runs and the parsing can be used to bound the size of the resulting grammar, because new nonterminals can be defined only around the borders between runs. We use the following result.

Definition 3

A repetitive area in a string is a maximal run of the same symbol, of length 2 or more.

Lemma 15 ([49])

We can partition a string $S$ into at most $(3/4)|S|$ blocks so that, for every pair of identical substrings $S[i..j]=S[i^{\prime}..j^{\prime}]$, if neither $S[i+1..j-1]$ or $S[i^{\prime}+1..j^{\prime}-1]$ overlap a repetitive area, then the sequence of blocks covering $S[i+1..j-1]$ and $S[i^{\prime}+1..j^{\prime}-1]$ are identical.

Proof

The parsing is obtained by, first, creating blocks for the repetitive areas, which can be of any length $\geq 2$. For the remaining characters, the alphabet is partitioned into two subsets, left-symbols and right-symbols. Then, every left-symbol followed by a right-symbol are paired in a block. It is then clear that, if $S[i+1..j-1]$ and $S[i^{\prime}+1..j^{\prime}-1]$ do not overlap repetitive areas, then the parsing of $S[i..j]$ and $S[i^{\prime}..j^{\prime}]$ may differ only in their first position (if it is part of a repetitive area ending there, or if it is a right-symbol that becomes paired with the preceding one) and in their last position (if it is part of a repetitive area starting there, or if it is a left-symbol that becomes paired with the following one). Jez [49] shows how to choose the pairs so that $S$ contains at most $(3/4)|S|$ blocks. ∎

The lemma ensures a locally consistent parsing into blocks as long as the substrings do not overlap repetitive areas, though the substrings may fully contain repetitive areas.

Our key result is that a round of parsing does not create too many distinct nonterminals if there are few runs in the $\mathit{BWT}$. We state the result in the general form we will need.

Lemma 16

Let $W[1..n]$ be a sequence with $2r$ positions $p_{1}^{-}\leq p_{1}^{+}<p_{2}^{-}\leq p_{2}^{+}<\ldots<p_{r}^{-}\leq p_{r}^{+}$ and let there be a permutation $\pi$ of one cycle on $[1..n]$ so that, whenever $k$ is not in any range $[p_{i}^{-}..p_{i}^{+}]$, it holds (a) $\pi(k-1)=\pi(k)-1$ and (b) $W[\pi(k)]=W[k]$. Then, an application of the parsing scheme of Lemma 15 on $W$ creates only the distinct blocks that it defines to cover all the areas $W[p_{i}^{-}-1..p_{i}^{+}+1]$.

Proof

Assume we carry out the parsing of Lemma 15 on $W$, which cuts $W$ into at most $(3/4)n$ blocks. Consider the smallest $k\geq 1$ along the cycle of $\pi$ such that (1) $j=\pi^{k}(1)$ is the end of a block $[j^{\prime}..j]$ and (2) $[j^{\prime}-1..j+1]$ is disjoint from any $[p_{i}^{-}..p_{i}^{+}]$. Then, by conditions (a) and (b) it follows that $[\pi(j^{\prime}-1)..\pi(j+1)]=[\pi(j^{\prime})-1..\pi(j)+1]=[\pi(j)-(j-j^{\prime})-1..\pi(j)+1]$ and $W[\pi(j^{\prime})-1..\pi(j)+1]=W[j^{\prime}-1..j+1]$.

By the way blocks $W[j^{\prime}..j]$ are formed, it must be $W[j^{\prime}-1]\not=W[j^{\prime}]$ and $W[j+1]\not=W[j]$, and thus $W[\pi(j^{\prime})-1]\not=W[\pi(j^{\prime})]$ and $W[\pi(j)+1]\not=W[\pi(j)]$. Therefore, neither $W[j^{\prime}..j]$ nor $W[\pi(j^{\prime})..\pi(j)]$ may overlap a repetitive area. It thus holds by Lemma 15 that $W[\pi(j^{\prime})..\pi(j)]$ must also be a single block, identical to $W[j^{\prime}..j]$.

Hence position $\pi(j)=\pi^{k+1}(1)$ satisfies condition (1) as well. If it also satisfies (2), then another block identical to $W[j^{\prime}..j]$ ends at $\pi^{k+2}(1)$, and so on, until we reach some $\pi^{k^{\prime}}(1)$ ending a block that intersects some $[p_{i}^{-}-1,p_{i}^{+}+1]$. At this point, we find the next $k^{\prime\prime}>k^{\prime}$ such that $\pi^{k^{\prime\prime}}$ satisfies (1) and (2), and continue the process from $k=k^{\prime\prime}$.

Along the cycle we visit all the positions in $W$. The positions that do not satisfy (2) are those in the blocks that intersect some $W[p_{i}^{-}-1,p_{i}^{+}+1]$. From those positions, the ones that satisfy (1) are the endpoints of the distinct blocks we visit. Therefore, the distinct blocks are precisely those that the parsing creates to cover the areas $W[p_{i}^{-}-1,p_{i}^{+}+1]$. ∎

Now we show that a RLCFG built by successively parsing a string into blocks and replacing them by nonterminals has a size that can be bounded in terms of $r$.

Lemma 17

Let $W[1..n]$ be a sequence with $2r$ positions $p_{i}^{-}$ and $p_{i}^{+}$ satisfying the conditions of Lemma 16. Then, a RLCFG built by applying Lemma 16 in several rounds and replacing blocks by nonterminals, compresses $W$ to size $O(\lambda r\log(n/r))$, where $\lambda=\max_{1\leq i\leq r}(p_{i}^{+}-p_{i}^{-}+1)$.

Proof

We apply a first round of locally consistent parsing on $W$ and create a distinct nonterminal per distinct block of length $\geq 2$ produced along the parsing. In order to represent the blocks that are repetitive areas, our grammar is a RLCFG.

Once we replace the parsed blocks by nonterminals, the new sequence $W^{\prime}$ is of length at most $(3/4)n$. Let $\mu$ map from a position in $W$ to the position of its block in $W^{\prime}$. To apply Lemma 16 again over $W^{\prime}$, we define a new permutation $\pi^{\prime}$, by skipping the positions in $\pi$ that do not fall on block ends of $W$, and mapping the block ends to the blocks (now symbols) in $W^{\prime}$. The positions $p_{i}^{+}$ and $p_{i}^{-}$ are mapped to their corresponding block positions $\mu(p_{i}^{+})$ and $\mu(p_{i}^{-})$. To see that (a) and (b) hold in $W^{\prime}$, let $[p_{i}^{+}+1..p_{i+1}^{-}-1]$ be the zone between two areas $[p^{-},p^{+}]$ of Lemma 16, where it can be applied, and consider the corresponding range $W[\pi(p_{i}^{+}+1)..\pi(p_{i+1}^{-}-1)]=W[\pi(p_{i}^{+})+1..\pi(p_{i+1}^{-})-1]=W[p_{i}^{+}+1..p_{i+1}^{-}-1]$. Then, the locally consistent parsing represented as a RLCFG guarantees that

[TABLE]

Therefore, we can choose the new values $(p_{i}^{+})^{\prime}=\mu(p_{i}^{+}+1)$ and $(p_{i}^{-})^{\prime}=\mu(p_{i}^{-}-1)$ and apply Lemma 16 once again.

Let us now bound the number of nonterminals created by each round. Considering repetitive areas in the extremes, the area $W[p_{i}^{-}-1,p_{i}^{+}+1]$, of length $\ell_{i}=p_{i}^{+}-p_{i}^{-}+3\leq\lambda+2$, may produce up to $2+\lfloor\ell_{i}/2\rfloor$ nonterminals, and be reduced to this new length after the first round. It could also produce no nonterminals at all and retain its original length. In general, for each nonterminal produced, the area shrinks by at least 1: if we create $d$ nonterminals, then $(p_{i}^{+})^{\prime}-(p_{i}^{-})^{\prime}\leq p_{i}^{+}-p_{i}^{-}-d$.

On the other hand, the new extended area $W^{\prime}[(p_{i}^{-})^{\prime}-1..(p_{i}^{+})^{\prime}+1]$ adds two new symbols to that length. Therefore, the area starts with length $\ell_{i}$ and grows by 2 in each new round. Whenever it creates a new nonterminal, it decreases at least by 1. It follows that, after $k$ parsing rounds, each area can create at most $\ell_{i}+2k\leq\lambda+2+2k$ nonterminals, thus the grammar is of size $r(\lambda+2+2k)$. On the other hand, the text is of length $(3/4)^{k}n$, for a total size of $(3/4)^{k}n+r(\lambda+2+2k)$. Choosing $k=\log_{4/3}(n/r)$, the total space becomes $O(\lambda r\log(n/r))$. ∎

6.2 Accessing $\mathit{SA}$

Let us define the differential suffix array $\mathit{DSA}[k]=\mathit{SA}[k]-\mathit{SA}[k-1]$ for all $k>1$, and $\mathit{DSA}[1]=\mathit{SA}[1]$. The next lemma shows that the structure of $\mathit{DSA}$ is suitable to apply Lemmas 16 and 17.

Lemma 18

Let $[x-1,x]$ be within a run of $\mathit{BWT}$. Then $\mathit{LF}(x-1)=\mathit{LF}(x)-1$ and $\mathit{DSA}[\mathit{LF}(x)]=\mathit{DSA}[x]$.

Proof

Since $x$ is not the first position in a run of $\mathit{BWT}$, it holds that $\mathit{BWT}[x-1]=\mathit{BWT}[x]$, and thus $\mathit{LF}(x-1)=\mathit{LF}(x)-1$ follows from the formula of $\mathit{LF}$. Therefore, if $y=\mathit{LF}(x)$, we have $\mathit{SA}[y]=\mathit{SA}[x]-1$ and $\mathit{SA}[y-1]=\mathit{SA}[\mathit{LF}(x-1)]=\mathit{SA}[x-1]-1$; therefore $\mathit{DSA}[y]=\mathit{DSA}[x]$. See the bottom of Figure 1 (other parts are used in the next sections). ∎

Therefore, Lemmas 16 and 17 apply on $W=\mathit{DSA}$, $p_{i}^{-}=p_{i}^{+}=p_{i}$ being the $r$ positions where runs start in $\mathit{BWT}$, and $\pi=\mathit{LF}$. In this case, $\lambda=1$.

Note that the height of the grammar is $k=O(\log(n/r))$, since the nonterminals of a round use only nonterminals from the previous rounds. Our construction ends with a sequence of $O(r)$ symbols, not with a single initial symbol. While we could add $O(r)$ nonterminals to have a single initial symbol and a grammar of height $O(\log n)$, we maintain the grammar in the current form to enable extraction in time $O(\log(n/r))$.

To efficiently extract any value $\mathit{SA}[i]$, we associate with each nonterminal $X$ the sum of the $\mathit{DSA}$ values, $d(X)$, and the number of positions, $l(X)$, it expands to. We also sample one out of $n/r$ positions of $\mathit{SA}$, storing the value $\mathit{SA}[i\cdot(n/r)]$, the pointer to the corresponding nonterminal in the compressed $\mathit{DSA}$ sequence. For each position in the compressed $\mathit{DSA}$ sequence, we also store its starting position, $pos$, in the original sequence, and the corresponding left absolute value, $abs=\mathit{SA}[pos-1]$. Since there cannot be more than $n/r$ nonterminals between two sampled positions, a binary search on the fields $pos$ finds, in time $O(\log(n/r))$, the nonterminal of the compressed $\mathit{DSA}$ sequence we must expand to find any $\mathit{SA}[i]$, and the desired offset inside it, $o\leftarrow i-pos$, as well as the sum of the $\mathit{DSA}$ values to the left, $a\leftarrow abs$. We then descend from the node of this nonterminal in the grammar tree. At each node $X\rightarrow YZ$, we descend to the left child if $l(Y)\geq o$ and to the rigth child otherwise. If we descend to the right child, we update $o\leftarrow i-l(Y)$ and $a\leftarrow a+d(Y)$. For rules of the form $X\rightarrow Y^{t}$, we determine which copy of $Y$ we descend to, $j=\lceil o/l(Y)\rceil$, and update $o\leftarrow o-(j-1)\cdot l(Y)$ and $a\leftarrow a+(j-1)\cdot d(Y)$. When we finally reach the leaf corresponding to $\mathit{DSA}[i]$ (with $o=0$), we return $\mathit{SA}[i]=a+\mathit{DSA}[i]$. The total time is $O(\log(n/r)+k)$. Note that, once a cell is found, each subsequent cell can be extracted in constant time.555To save space, we may store $pos$ only for one out of $k$ symbols in the compressed $\mathit{DSA}$ sequence, and complete the binary search with an $O(k)$-time sequential scan. We may also store one top-level sample out of $(n/r)^{2}$. This retains the time complexity and reduces the total sampling space to $o(r)+O(1)$ words.

Theorem 6.1

Let the $\mathit{BWT}$ of a text $T[1..n]$ contain $r$ runs. Then there exists a data structure using $O(r\log(n/r))$ words that can retrieve any $\ell$ consecutive values of its suffix array in time $O(\log(n/r)+\ell)$.

6.3 Accessing $\mathit{ISA}$

A similar method can be used to access inverse suffix array cells, $\mathit{ISA}[i]$. Let us define $\mathit{DISA}[i]=\mathit{ISA}[i]-\mathit{ISA}[i-1]$ for all $i>1$, and $\mathit{DISA}[1]=\mathit{ISA}[1]$. The role of the runs in $\mathit{SA}$ will now be played by the phrases in $\mathit{ISA}$, which will be defined analogously as in the proof of Lemma 3: Phrases in $\mathit{ISA}$ start at the positions $\mathit{SA}[p]$ such that a new run starts in $\mathit{BWT}[p]$. Instead of $\mathit{LF}$, we define the cycle $\phi(i)=\mathit{SA}[\mathit{ISA}[i]-1]$ if $\mathit{ISA}[i]>1$ and $\phi(i)=\mathit{SA}[n]$ otherwise. We then have the following lemmas.

Lemma 19

Let $[i-1..i]$ be within a phrase of $\mathit{ISA}$. Then it holds $\phi(i-1)=\phi(i)-1$.

Proof

Consider the pair of positions $T[i-1..i]$ within a phrase. Let them be pointed from $\mathit{SA}[x]=i$ and $\mathit{SA}[y]=i-1$, therefore $\mathit{ISA}[i]=x$, $\mathit{ISA}[i-1]=y$, and $\mathit{LF}(x)=y$ (see Figure 1). Now, since $i$ is not a phrase beginning, $x$ is not the first position in a $\mathit{BWT}$ run. Therefore, $\mathit{BWT}[x-1]=\mathit{BWT}[x]$, from which it follows that $\mathit{LF}(x-1)=\mathit{LF}(x)-1=y-1$. Now let $\mathit{SA}[x-1]=j$, that is, $j=\phi(i)$. Then $\phi(i-1)=\mathit{SA}[\mathit{ISA}[i-1]-1]=\mathit{SA}[y-1]=\mathit{SA}[\mathit{LF}(x-1)]=\mathit{SA}[x-1]-1=j-1=\phi(i)-1$. ∎

Lemma 20

Let $[i-1..i]$ be within a phrase of $\mathit{ISA}$. Then it holds $\mathit{DISA}[i]=\mathit{DISA}[\phi(i)]$.

Proof

From the proof of Lemma 19, it follows that $\mathit{DISA}[i]=x-y=\mathit{DISA}[j]=\mathit{DISA}[\phi(i)]$. ∎

As a result, Lemmas 16 and 17 apply with $W=\mathit{DISA}$, $p_{i}^{+}=p_{i}+1$ and $p_{i}^{-}=p_{i}$, where $p_{i}$ are the $r$ positions where phrases start in $\mathit{ISA}$, and $\pi=\phi$. Now $\lambda=2$. We use a structure analogous to that of Section 6.2 to obtain the following result.

Theorem 6.2

Let the $\mathit{BWT}$ of a text $T[1..n]$ contain $r$ runs. Then there exists a data structure using $O(r\log(n/r))$ words that can retrieve any $\ell$ consecutive values of its inverse suffix array in time $O(\log(n/r)+\ell)$.

6.4 Accessing $\mathit{LCP}$, Revisited

In Section 3.1 we showed how to access array $\mathit{LCP}$ efficiently if we can access $\mathit{SA}$. However, for the full suffix tree functionality we will develop in Section 7, we will need operations more sophisticated than just accessing cells, and these will be carried out on a grammar-compressed representation. In this section we show that the differential array $\mathit{DLCP}[1..n]$, where $\mathit{DLCP}[i]=\mathit{LCP}[i]-LCP[i-1]$ if $i>1$ and $\mathit{DLCP}[1]=\mathit{LCP}[1]$, can be represented by a grammar of size $O(r\log(n/r))$.

Lemma 21

Let $[x-2,x]$ be within a run of $\mathit{BWT}$. Then $\mathit{DLCP}[\mathit{LF}(x)]=\mathit{DLCP}[x]$.

Proof

Let $i=\mathit{SA}[x]$, $j=\mathit{SA}[x-1]$, and $k=\mathit{SA}[x-2]$. Then $\mathit{LCP}[x]=lcp(T[i..n],T[j..n])$ and $\mathit{LCP}[x-1]=lcp(T[j..n],T[k..n])$. We know from Lemma 18 that, if $y=\mathit{LF}(x)$, then $\mathit{LF}(x-1)=y-1$ and $\mathit{LF}(x-2)=y-2$. Also, $\mathit{SA}[y]=i-1$, $\mathit{SA}[y-1]=j-1$, and $\mathit{SA}[y-2]=k-1$. Therefore, $\mathit{LCP}[\mathit{LF}(x)]=\mathit{LCP}[y]=lcp(T[\mathit{SA}[y]..n],T[\mathit{SA}[y-1]..n)=lcp(T[i-1..n],T[j-1..n])$. Since $x$ is not the first position in a $\mathit{BWT}$ run, it holds that $T[j-1]=\mathit{BWT}[x-1]=\mathit{BWT}[x]=T[i-1]$, and thus $lcp(T[i-1..n],T[j-1..n])=1+lcp(T[i..n],T[j..n])=1+\mathit{LCP}[x]$. Similarly, $\mathit{LCP}[\mathit{LF}(x)-1]=\mathit{LCP}[y-1]=lcp(T[\mathit{SA}[y-1]..n],T[\mathit{SA}[y-2]..n)=lcp(T[j-1..n],T[k-1..n])$. Since $x-1$ is not the first position in a $\mathit{BWT}$ run, it holds that $T[k-1]=\mathit{BWT}[x-2]=\mathit{BWT}[x-1]=T[j-1]$, and thus $lcp(T[j-1..n],T[k-1..n])=1+lcp(T[j..n],T[k..n])=1+\mathit{LCP}[x-1]$. Therefore $\mathit{DLCP}[y]=\mathit{LCP}[y]-\mathit{LCP}[y-1]=(1+\mathit{LCP}[x])-(1+\mathit{LCP}[x-1])=\mathit{DLCP}[x]$. ∎

Therefore, Lemmas 16 and 17 appliy with $W=\mathit{DLCP}$, $p_{i}^{+}=p_{i}+2$ and $p_{i}^{-}=p_{i}$, where $p_{i}$ are the $r$ positions where runs start in $\mathit{BWT}$, and $\pi=\mathit{LF}$. In this case, $\lambda=3$. We use a structure analogous to that of Section 6.2 to obtain the following result.

Theorem 6.3

Let the $\mathit{BWT}$ of a text $T[1..n]$ contain $r$ runs. Then there exists a data structure using $O(r\log(n/r))$ words that can retrieve any $\ell$ consecutive values of its longest common prefix array in time $O(\log(n/r)+\ell)$.

6.5 Accessing the Text, Revisited

In Section 4 we devised a data structure that uses $O(r\log(n/r))$ words and extracts any substring of length $\ell$ from $T$ in time $O(\log(n/r)+\ell\log(\sigma)/w)$. We now obtain a result that, although does not improve upon the representation of Section 4, is an easy consequence of our results on accessing $\mathit{SA}$, $\mathit{ISA}$, and $\mathit{LCP}$, and will be used in Section 10 to shed light on the relation between run-compression, grammar-compression, Lempel-Ziv compression, and bidirectional compression.

Let us define $\phi$ just as in Section 6.3; we also define phrases in $T$ exactly as those in $\mathit{ISA}$. We can then use Lemma 19 verbatim on $T$ instead of $\mathit{ISA}$, and also translate Lemma 20 as follows.

Lemma 22

Let $T[i-1..i]$ be within a phrase. Then it holds $T[i-1]=T[\phi(i)-1]$.

Proof

From the proof of Lemma 19, it follows that $T[i-1]=\mathit{BWT}[x]=\mathit{BWT}[x-1]=T[j-1]=T[\phi(i)-1]$. ∎

We can then apply Lemmas 16 and 17 again, with $W=T$, $p_{i}^{+}=p_{i}+1$ and $p_{i}^{-}=p_{i}-1$ (so $\lambda=3$), where $p_{i}$ are the $r$ positions where phrases start in $T$, and $\pi=\phi$. Therefore, there exists a data structure (analogous to that of Section 6.2) of size $O(r\log(n/r))$ that gives access to any substring $T[i..i+\ell-1]$ in time $O(\log(n/r)+\ell)$.

7 A Run-Length Compressed Suffix Tree

In this section we show how to implement a compressed suffix tree within $O(r\log(n/r))$ words, which solves a large set of navigation operations in time $O(\log(n/r))$. The only exceptions are going to a child by some letter and performing level ancestor queries on the tree depth. The first compressed suffix tree for repetitive collections built on runs [65], but just like the self-index, it needed $O(n/s)$ space to obtain $O(s\log n)$ time in key operations like accessing $\mathit{SA}$. Other compressed suffix trees for repetitive collections appeared later [1, 76, 27], but they do not offer formal space guarantees (see later). A recent one, instead, uses $O(\overline{e})$ words and supports a number of operations in time typically $O(\log n)$ [5]. The two space measures are not comparable.

7.1 Compressed Suffix Trees without Storing the Tree

Fischer et al. [33] showed that a rather complete suffix tree functionality including all the operations in Table 3 can be efficiently supported by a representation where suffix tree nodes $v$ are identified with the suffix array intervals $\mathit{SA}[v_{l}..v_{r}]$ they cover. Their representation builds on the following primitives:

Access to arrays $\mathit{SA}$ and $\mathit{ISA}$, in time we call $t_{\mathit{SA}}$. 2. 2.

Access to array $\mathit{LCP}[1..n]$, in time we call $t_{\mathit{LCP}}$. 3. 3.

Three special queries on $\mathit{LCP}$:

1. (a)

Range Minimum Query, $\mathrm{RMQ}(i,j)=\arg\min_{i\leq k\leq j}\mathit{LCP}[k]$, choosing the leftmost one upon ties, in time we call $t_{\mathrm{RMQ}}$. 2. (b)

Previous/Next Smaller Value queries, $\mathrm{PSV}(i)=\max(\{k<i,\mathit{LCP}[k]<\mathit{LCP}[i]\}\cup\{0\})$ and $\mathrm{NSV}(i)=\min(\{k>i,\mathit{LCP}[k]<\mathit{LCP}[i]\}\cup\{n+1\})$, in time we call $t_{\mathrm{SV}}$.

An interesting finding of Fischer et al. [33] related to our results is that array $\mathit{PLCP}$, which stores the $\mathit{LCP}$ values in text order, can be stored in $O(r)$ words and accessed efficiently; therefore we can compute any $\mathit{LCP}$ value in time $t_{\mathit{SA}}$ (see also Fischer [31]). We obtained a generalization of this property in Section 3.1. They [33] also show how to represent the array $\mathit{TDE}[1..n]$, where $\mathit{TDE}[i]$ is the tree-depth of the lowest common ancestor of the $(i-1)$th and $i$th suffix tree leaves ($\mathit{TDE}[1]=0$). Fischer et al. [33] represent its values in text order in an array $\mathit{PTDE}$, which just like $\mathit{PLCP}$ can be stored in $O(r)$ words and accessed efficiently, giving access to $\mathit{TDE}$ in time $t_{\mathit{SA}}$. They use $\mathit{TDE}$ to compute operations TDepth and LAQT efficiently.

Abeliuk et al. [1] show that primitives $\mathrm{RMQ}$, $\mathrm{PSV}$, and $\mathrm{NSV}$ can be implemented using a simplified variant of range min-Max trees (rmM-trees) [77], consisting of a perfect binary tree on top of $\mathit{LCP}$ where each node stores the minimum $\mathit{LCP}$ value in its subtree. The three primitives are then computed in logarithmic time. They show that the slightly extended primitives $\mathrm{PSV}^{\prime}(i,d)=\max(\{k<i,\mathit{LCP}[k]<d\}\cup\{0\})$ and $\mathrm{NSV}^{\prime}(i,d)=\min(\{k>i,\mathit{LCP}[k]<d\}\cup\{n+1\})$, can be computed with the same complexity $t_{\mathrm{SV}}$ of the basic $\mathrm{PSV}$ and $\mathrm{NSV}$ primitives, and they can be used to simplify some of the operations of Fischer et al. [33].

The resulting time complexities are given in the second column of Table 4, where $t_{\mathit{LF}}$ is the time to compute function $\mathit{LF}$ or its inverse, or to access a position in $\mathit{BWT}$. Operation WLink, not present in Fischer et al. [33], is trivially obtained with two $\mathit{LF}$-steps. We note that most times appear multiplied by $t_{\mathit{LCP}}$ in Fischer et al. [33] because their $\mathrm{RMQ}$, $\mathrm{PSV}$, and $\mathrm{NSV}$ structures do not store $\mathit{LCP}$ values inside, so they need to access the array all the time; this is not the case when we use rmM-trees. The times of NSibling and LAQS owe to improvements obtained with the extended primitives $\mathrm{PSV}^{\prime}$ and $\mathrm{NSV}^{\prime}$ [1]. The time for Child($v,a$) is obtained by binary searching among the $\sigma$ minima of $\mathit{LCP}[v_{l},v_{r}]$, and extracting the desired letter (at position SDepth$(v)+1$) to compare with $a$. Each binary search operation can be done with an extended primitive $\mathrm{RMQ}^{\prime}(i,j,m)$ that finds the $m$th left-to-right occurrence of the minimum in a range. This is easily done in $t_{\mathrm{RMQ}}$ time on a rmM-tree that stores, in addition, the number of times the minimum of each node occurs below it [77]. Finally, the complexities of TDepth and LAQT make use of array $\mathit{TDE}$. While Fischer et al. [33] use an $\mathrm{RMQ}$ operation to compute TDepth, we note that TDepth$(v)=1+\max(\mathit{TDE}[v_{l}],\mathit{TDE}[v_{r}+1])$, because the suffix tree has no unary nodes (they used this simpler formula only for leaves).666We observe that LAQT can be solved exactly as LAQS, with the extended $\mathrm{PSV}^{\prime}/\mathrm{NSV}^{\prime}$ operations, now defined on the array $\mathit{TDE}$ instead of on $\mathit{LCP}$. However, an equivalent to Lemma 21 for the differential $\mathit{TDE}$ array does not hold, and therefore we cannot build the structures on its grammar within the desired space bounds.

7.2 Exploiting the Runs

An important result of Abeliuk et al. [1] is that they represent $\mathit{LCP}$ differentially, that is, array $\mathit{DLCP}$, in grammar-compressed form. Further, they store the rmM-tree information in the nonterminals, that is, a nonterminal $X$ expanding to a substring $D$ of $\mathit{DLCP}$ stores the minimum $m(X)=\min_{0\leq k\leq|D|}\sum_{i=1}^{k}D[i]$ and its position $p(X)=\arg\min_{0\leq k\leq|D|}\sum_{i=1}^{k}D[i]$. Thus, instead of a perfect rmM-tree, they conceptually use the grammar tree as an rmM-tree. They show how to adapt the algorithms on the perfect rmM-tree to run on the grammar, and thus solve primitives $\mathrm{RMQ}$, $\mathrm{PSV}^{\prime}$, and $\mathrm{NSV}^{\prime}$, in time proportional to the grammar height.

Abeliuk et al. [1], and also Fischer et al. [33], claim that RePair compression [60] reaches size $O(r\log(n/r))$. This is an incorrect result borrowed from González et al. [43, 44], where it was claimed for $\mathit{DSA}$. The proof fails for a reason we describe in Appendix 0.A. In Section 6 we have shown that, instead, the use of locally consistent parsing does offer a guarantee of $O(r\log(n/r))$ words, with a (run-length) grammar height of $O(\log(n/r))$, for $\mathit{DSA}$, $\mathit{DISA}$, and $\mathit{DLCP}$.777The reason of failure is subtle and arises pathologically; in our preliminary experiments RePair actually compresses to about half the space of locally consistent parsing in typical repetitive texts.

The third column of Table 4 gives the resulting complexities when $t_{\mathit{SA}}=\log(n/r)$, $t_{\mathit{LF}}=\log\log_{w}(n/r)$, and $t_{\mathit{LCP}}=\log\log_{w}(n/r)+t_{\mathit{SA}}=\log(n/r)$, as we have established in previous sections. To complete the picture, we describe how to compute the extended primitives $\mathrm{RMQ}^{\prime}$ and $\mathrm{PSV}^{\prime}/\mathrm{NSV}^{\prime}$ on the grammar, in time $t_{\mathrm{RMQ}}=t_{\mathrm{SV}}=\log(n/r)$. While analogous procedures have been described before [1, 77], some particularities in our structures deserve a complete description.

Note that the three primitives can be solved on $\mathit{DLCP}$ or on $\mathit{LCP}$, as they refer to relative values. The structure we use for $\mathit{DLCP}$ in Section 6.4 is formed by a compressed sequence $\mathit{DLCP}^{\prime}$ of length $O(r)$, plus a run-length grammar of size $O(r\log(n/r))$ and of height $h=O(\log(n/r))$.

The first part of our structure is built as follows. Let $\mathit{DLCP}[i_{m},i_{m+1}-1]$ be the area to which $\mathit{DLCP}^{\prime}[m]$ expands; then we build the array $M[m]=\min_{i_{m}\leq k<i_{m+1}}\mathit{LCP}[k]$. We store a succinct $\mathrm{RMQ}$ data structure on $M$, which requires just $O(r)$ bits and answers $\mathrm{RMQ}$s on $M$ in constant time, without need to access $M$ [32, 77].

As in Section 6.2, in order to give access to $\mathit{DLCP}$, we store for each nonterminal $X$ the number $l(X)$ of positions it expands to and its total difference $d(X)$. We also store, to compute $\mathrm{RMQ}^{\prime}$, the values $m(X)$, $p(X)$, and also $n(X)$, the number of times $m(X)$ occurs inside the expansion of $X$. We also store the sampling information described in Section 6.2, now to access $\mathit{DLCP}$.

To compute $\mathrm{RMQ}^{\prime}(i,j,m)$, we use a mechanism analogous to the one described in Section 6.2 to access $\mathit{DLCP}[i..j]$, but do not traverse the cells one by one. Instead, we determine that $\mathit{DLCP}[i..j]$ is contained in the area $\mathit{DLCP}[i^{\prime}..j^{\prime}]$ that corresponds to $\mathit{DLCP}^{\prime}[x..y]$ in the compressed sequence, partially overlapping $\mathit{DLCP}^{\prime}[x]$ and $\mathit{DLCP}^{\prime}[y]$ and completely covering $\mathit{DLCP}^{\prime}[x+1..y-1]$ (there are various easy particular cases we ignore). We first obtain in constant time the minimum position of the central area, $z=\mathrm{RMQ}(x+1,y-1)$, using the $O(r)$-bit structure. We must then obtain the minimum values in $\mathit{DLCP}^{\prime}[x]\langle i^{\prime}-i+1,l(\mathit{DLCP}^{\prime}[x])\rangle$, $\mathit{DLCP}^{\prime}[z]\langle 1,l(\mathit{DLCP}^{\prime}[z])\rangle$, and $\mathit{DLCP}^{\prime}[y]\langle 1,l(\mathit{DLCP}^{\prime}[y])+j-j^{\prime}\rangle$, where $X\langle a,b\rangle$ refers to the range $[a..b]$ in the expansion of nonterminal $X$.

To find the minimum in $\mathit{DLCP}^{\prime}[w]\langle a,b\rangle$, where $\mathit{DLCP}^{\prime}[w]=X$, we identify the at most $2k$ maximal nodes of the grammar tree that cover the range $[a..b]$ in the expansion of $X$. Let these nodes be $X_{1},X_{2},\ldots,X_{2k}$. We then find the minimum of $m(X_{1})$, $d(X_{1})+m(X_{2})$, $d(X_{1})+d(X_{2})+m(X_{3})$, and so on, in $O(k)$ time. Once the minimum is identified at $X_{s}$, we obtain the absolute value by extracting $\mathit{LCP}[i_{w}-1+l(X_{1})+\ldots+l(X_{s-1})+p(X_{s})]$.

Once we have the three minima, the smallest of the three is $\mu$, the value of $\mathrm{RMQ}(i,j)$. To solve $\mathrm{RMQ}^{\prime}(i,j,m)$, we first compute $\mu$ and then find its $m$th occurrence through $\mathit{DLCP}^{\prime}[x]\langle i^{\prime}-i+1,l(\mathit{DLCP}^{\prime}[x])\rangle$, $\mathit{DLCP}^{\prime}[x+1..y-1]$, and $\mathit{DLCP}^{\prime}[y]\langle 1,l(\mathit{DLCP}^{\prime}[y])+j-j^{\prime}\rangle$. To process $X\langle a,b\rangle$, we scan again $X_{1},X_{2},\ldots,X_{2k}$. For each $X_{s}$, if $d(X_{1})+\ldots+d(X_{s-1})+m(X_{s})=m$, we subtract $n(X_{s})$ from $m$. When the result is below 1, we enter the children of $X_{s}$, $Y_{1}$ and $Y_{2}$, doing the same process we ran on $X_{1},X_{2},\ldots$ on $Y_{1},Y_{2}$ to find the child where $m$ falls below 1. We recursively enter this child of $X$, and so on, until reaching the precise position in $X\langle a,b\rangle$ with the $m$th occurrence of $\mu$. The position itself is computing by adding all the lengths $l(X_{i})$, $l(Y_{i})$, etc. we skip along the process. All this takes time $O(k)=O(\log(n/r))$.

We might find the answer as we traverse $\mathit{DLCP}^{\prime}[x]\langle i^{\prime}-i+1,l(\mathit{DLCP}^{\prime}[x])\rangle$. If, however, there are only $m^{\prime}<m$ occurrences of $\mu$ in there, and the minimum in $\mathit{DLCP}^{\prime}[x+1..y-1]$ is also $\mu$, we must find the $(m-m^{\prime})$th occurrence of the minimum in $\mathit{DLCP}^{\prime}[x+1..y-1]$. This can also be done in constant time with the $O(r)$-bit structure [77]. If, in turn, there are only $m^{\prime\prime}<m-m^{\prime}$ occurrences of $\mu$ inside, we must find the $(m-m^{\prime}-m^{\prime\prime})$th occurrence of $\mu$ in $\mathit{DLCP}^{\prime}[y]\langle 1,l(\mathit{DLCP}^{\prime}[y])+j-j^{\prime}\rangle$.

Our grammar also has rules of the form $X\rightarrow Y^{t}$. It is easy to process several copies of $Y$ in constant time, even if they span only some of the $t$ copies of $X$. For example, the minimum in $Y^{s}$ is $m(Y)$ if $d(Y)\geq 0$, and $(s-1)\cdot d(Y)+m(Y)$ otherwise. Later, to find where $\mu$ occurs, we have that it can only occur in the first copy if $d(Y)>0$, only in the last if $d(Y)<0$, and in every copy if $d(Y)=0$. In the latter case, if $s\cdot n(Y)<m$, then the $m$th occurrence is not inside the copies; otherwise it occurs inside the $\lceil m/n(Y)\rceil$th copy. The other operations are trivially derived.

Therefore, we can compute any query $\mathrm{RMQ}^{\prime}(i,j,m)$ in time $O(\log(n/r))$ plus $O(1)$ accesses to $\mathit{LCP}$; therefore $t_{\mathrm{RMQ}}=\log(n/r)$.

Queries $\mathrm{PSV}^{\prime}(i,d)$ and $\mathrm{NSV}^{\prime}(i,d)$ are solved analogously. Let us describe $\mathrm{NSV}^{\prime}$; $\mathrm{PSV}^{\prime}$ is similar. Let $\mathit{DLCP}[i..n]$ intersect $\mathit{DLCP}^{\prime}[x..y]$, which expands to $\mathit{DLCP}[i^{\prime}..n]$, and let us subtract $\mathit{LCP}[i^{\prime}-1]$ from $d$ to have it in relative form. We consider $\mathit{DLCP}^{\prime}[x]\langle i-i^{\prime}+1,l(\mathit{DLCP}^{\prime}[x])\rangle=X\langle a,b\rangle$, obtaining the nonterminals $X_{1},X_{2},\ldots,X_{2k}$ that cover $X\langle a,b\rangle$, and find the first $X_{s}$ where $d(X_{1})+\ldots+d(X_{s-1})+m(X_{s})<d$. Then we enter the children of $X_{s}$ to find the precise point where we fall below $d$, as before. If we do not fall below $d$ inside $X\langle a,b\rangle$, we must run the query on $\mathit{DLCP}^{\prime}[x+1..y]$ for $d-d(X_{1})-\ldots-d(X_{2k})$. Such a query boils down to the forward-search queries on large trees considered by Navarro and Sadakane [77, Sec. 5.1]. They build a so-called left-to-right minima tree where they carry out a level-ancestor-problem path decomposition [12], and have a precedessor structure on the consecutive values along such paths. In our case, the union of all those paths has $O(r)$ elements and the universe is of size $2n$; therefore predecessor queries can be carried out in $O(r)$ space and $O(\log\log_{w}(n/r))$ time [10, Thm. 14]. Overall, we can also obtain time $t_{\mathrm{SV}}=\log(n/r)$ within $O(r\log(n/r))$ words of space.

Theorem 7.1

Let the $\mathit{BWT}$ of a text $T[1..n]$, over alphabet $[1..\sigma]$, contain $r$ runs. Then a compressed suffix tree on $T$ can be represented using $O(r\log(n/r))$ words, and it supports the operations with the complexities given in Table 4.

8 Counting in Optimal Time

Powered by the results of the previous sections, we can now show how to achieve optimal counting time, both in the unpacked and packed settings.

Theorem 8.1

We can store a text $T[1..n]$ in $O(r\log(n/r))$ words, where $r$ is the number of runs in the $\mathit{BWT}$ of $T$, such that later, given a pattern $P[1..m]$, we can count the occurrences of $P$ in optimal $O(m)$ time.

Proof

By Lemma 10, we find one pattern occurrence in $O(m+\log(n/r))$ time with a structure of $O(r\log(n/r))$ words. By Theorem 6.2, we can compute the corresponding suffix array location $p$ in $O(\log(n/r))$ time with a structure of $O(r\log(n/r))$ words. Our goal is to compute the $\mathit{BWT}$ range $[sp,ep]$ of the pattern; then the answer is $ep-sp+1$. Let $\mathit{LCE}(i,j)=lcp(T[\mathit{SA}[i]..],T[\mathit{SA}[j]..])$ denote the length of the longest common prefix between the $i$-th and $j$-th lexicographically smallest text suffixes. Note that $p\in[sp,ep]$ and, for every $1\leq i\leq n$, $\mathit{LCE}(p,i)\geq m$ if and only if $i\in[sp,ep]$. On the other hand, it holds $\mathit{LCE}(i,j)=\min_{i<k\leq j}\mathit{LCP}[i]$ [89]. We can then find the area with the primitives $\mathrm{PSV}^{\prime}$ and $\mathrm{NSV}^{\prime}$ defined in Section 7: $sp=\max(1,\mathrm{PSV}^{\prime}(p,m))$ and $ep=\mathrm{NSV}^{\prime}(p,m)-1$. These primitives are computed in time $O(\log(n/r))$ and need $O(r\log(n/r))$ space. This gives us count in $O(m+\log(n/r))$ time and $O(r\log(n/r))$ words. To speed up counting for patterns shorter than $\log(n/r)$, we index them using a path-compressed trie as done in Theorem 5.1. We store in each explicit trie node the number of occurrences of the corresponding string to support the queries for short patterns. By Lemma 14, the size of the trie and of the text substrings explicitly stored to support path compression is $O(r\log(n/r))$. Our claim follows. ∎

Theorem 8.2

We can store a text $T[1..n]$ over alphabet $[1..\sigma]$ in $O(rw\log_{\sigma}(n/r))$ words, where $r$ is the number of runs in the $\mathit{BWT}$ of $T$, such that later, given a packed pattern $P[1..m]$, we can count the occurrences of $P$ in optimal $O(m\log(\sigma)/w)$ time.

Proof

By Lemma 11, we find one pattern occurrence in $O(m\log(\sigma)/w+\log(n/r))$ time with a structure of $O(rw\log_{\sigma}(n/r))$ words. By Theorem 6.2, we compute the corresponding suffix array location $p$ in $O(\log(n/r))$ time with a structure of $O(r\log(n/r))$ words. As in the proof of Theorem 8.1, we retrieve the $\mathit{BWT}$ range $[sp,ep]$ of the pattern with the primitives $\mathrm{PSV}^{\prime}$ and $\mathrm{NSV}^{\prime}$ of Section 7, and then return $ep-sp+1$. Overall, we can now count in $O(m\log(\sigma)/w+\log(n/r))$ time. To speed up counting patterns shorter than $w\log_{\sigma}n$, we index them using a z-fast trie [3, Sec. H.2] offering $O(m\log(\sigma)/w)$-time prefix queries, as done in Theorem 5.2. The trie takes $O(rw\log_{\sigma}(n/r))$ space. We store in each explicit trie node the number of occurrences of the corresponding string. The total space is dominated by $O(rw\log_{\sigma}(n/r))$. ∎

9 Experimental results

In this section we report on preliminary experiments that are nevertheless sufficient to expose the orders-of-magnitude time/space savings offered by our structure (more precisely, the simple variant developed in Section 3) compared with the state of the art.

9.1 Implementation

We implemented the structure of Theorem 3.1 with $s=1$ using the sdsl library [40]. For the run-length FM-index, we used the implementation described by Prezza [84, Thm. 28] (suffix array sampling excluded), taking $(1+\epsilon)r\log(n/r)+r(\log\sigma+2)$ bits of space for any constant $\epsilon>0$ (in our implementation, $\epsilon=0.5$) and supporting $O(\log(n/r)+\log\sigma)$-time LF mapping. This structure employs Huffman-compressed wavelet trees (sdsl’s wt_huff) to represent run heads, as in our experiments they turned out to be comparable in size and faster than Golynski et al.’s structure [42], which is implemented in sdsl’s wt_gmr. Our locate machinery is implemented as follows. We store two gap-encoded bitvectors U and D marking with a bit set text positions that are the last and first in their $\mathit{BWT}$ run, respectively. These bitvectors are implemented using sdsl’s sd_vector, take overall $2r(\log(n/r)+2)$ bits of space, and answer queries in $O(\log(n/r))$ time. We moreover store two permutations, DU and RD. DU maps the (D-ranks of) text positions corresponding to the last position of each $\mathit{BWT}$ run to the (U-rank of the) first position of the next run. RD maps ranks of $\mathit{BWT}$ runs to the (D-ranks of) text positions associated with the last position of the corresponding $\mathit{BWT}$ run. DU and RD are implemented using Munro et al.’s representation [71], take $(1+\epsilon^{\prime})r\log r$ bits each for any constant $\epsilon^{\prime}>0$, and support map and inverse in $O(1)$ time. These structures are sufficient to locate each pattern occurrence in $O(\log(n/r))$ time with the strategy of Theorem 3.1. We choose $\epsilon^{\prime}=\epsilon/2$. Overall, our index takes at most $r\log(n/r)+r\log\sigma+6r+(2+\epsilon)r\log n\leq(3+\epsilon)r\log n+6r$ bits of space for any constant $\epsilon>0$ and, after counting, locates each pattern occurrence in $O(\log(n/r))$ time. Note that this space is $(2+\epsilon)r\log n+O(r)$ bits larger than an optimal run-length $\mathit{BWT}$ representation, and since we store $2r$ suffix array samples, this is just $\epsilon\,r\log n+O(n)$ bits over the optimum (i.e., RLBWT + samples). In the following, we refer to our index as r-index. The code is publicly available [83].

9.2 Experimental Setup

We compared r-index with the state-of-the-art index for each compressibility measure: lzi [20] ($z$), slp [20] ($g$), rlcsa [90] ($r$), and cdawg [86] ($e$). We tested rlcsa using three suffix array sample rates per dataset: the rate $X$ resulting in the same size for rlcsa and r-index, plus rates $X/2$ and $X/4$. We measured memory usage and locate times per occurrence of all indexes on 1000 patterns of length 8 extracted from four repetitive datasets, also published with our implementation:

DNA:

an artificial dataset of 629145 copies of a DNA sequence of length 1000 (Human genome) where each character was mutated with probability $10^{-3}$;

boost:

a dataset consisting of concatenated versions of the GitHub’s boost library;

einstein:

a dataset consisting of concatenated versions of Wikipedia’s English Einstein page;

world_leaders:

a collection containing all pdf files of CIA World Leaders from January 2003 to December 2009 downloaded from the Pizza&Chili corpus.

Memory usage (Resident Set Size, RSS) was measured using /usr/bin/time between index loading time and query time. This choice was made because, due to the datasets’ high repetitiveness, the number $occ$ of pattern occurrences was very large. This impacts sharply on the working space of indexes such as lzi and slp, which report the occurrences in a recursive fashion. When considering this extra space, these indexes always use more space than r-index, but we prefer to emphasize the relation between the index sizes and their associated compressibility measure. The only existing implementation of cdawg works only on DNA files, so we tested it only on the DNA dataset.

9.3 Results

The results of our experiments are summarized in Figure 2. On all datasets, r-index significantly deviates from the space-time curve on which all other indexes are aligned. We locate occurrences one to three orders of magnitude faster than all other indexes except cdawg, which however is one order of magnitude larger. It is also clear that r-index dominates all practical space-time tradeoffs of rlcsa (other tradeoffs are too space- or time- consuming to be practical). The smallest indexes, lzi and slp, save very little space with respect to r-index at the expense of being one to two orders of magnitude slower.

10 New Bounds on Grammar and Lempel-Ziv Compression

In this section we obtain various new relations between repetitiveness measures, inspired in our construction of RLCFGs of size $O(r\log(n/r))$ of Section 6. We consider general bidirectional schemes, which encompass grammar compression and Lempel-Ziv parsings, and are arguably the most general mechanism to capture repetitiveness. We first prove that the $\mathit{BWT}$ runs induce a bidirectional scheme on $T$. We then obtain upper and lower bounds relating the size of the smallest bidirectional scheme with the number of $\mathit{BWT}$ runs, the size of the smallest grammar, and the number of phrases in the Lempel-Ziv parse.

We will use $g$ to denote the size of the smallest CFG and $g_{rl}$ to denote the size of the smallest RLCFG. Similarly, we will use $z$ for the size of the Lempel-Ziv parse that allows overlaps between source and target (i.e., the original one [61]) and $z_{no}$ the size of the Lempel-Ziv parse that does not allow overlaps. Both $z$ [61] and $z_{no}$ [93] are optimal left-to-right parsings allowing or not allowing overlaps, respectively. Therefore we have $g_{rl}\leq g$ and $z\leq z_{no}$. Storer and Szymanski [93] define a number of so-called macro schemes, which allow expressing a text $T$ in terms of copying blocks of it elsewhere and having some positions given explicitly. Lempel-Ziv is a particular case, equivalent to the scheme they call OPM/L (i.e., “original pointer macro to the left”, where phrases can be copied from anywhere at the left) in the case of allowing overlaps, and otherwise OPM/TR/L (i.e., adding the restriction called “topological recursion”, where a block cannot be used to define itself). If we remove the restriction of pointers pointing only to the left, then we can recreate $T$ by copying blocks from anywhere else in $T$. Those are called bidirectional (macro) schemes. We are interested in the most general one, OPM, where sources and targets can overlap and the only restriction is that every character of $T$ can be eventually deduced from copies of sources to targets. Finding the optimal OPM scheme, of $b$ macros ($b$ is called $\Delta_{OPM}$ in their paper [93]), is NP-complete [38], whereas linear-time algorithms to find the optimal unidirectional schemes, of sizes $z=\Delta_{OPM/L}$ or $z^{R}=\Delta_{OPM/R}$ (i.e., $z$ for the reversed $T$), are well-known [93].888What they call Lempel-Ziv, producing $LZ$ phrases, does not allow source/target overlaps, so it is our $z_{no}$. Further, little is known about the relation between the optimal bidirectional and unidirectional parsings, except that for any constant $\epsilon>0$ there is an infinite family of strings for which $b<(\frac{1}{2}+\epsilon)\cdot\min(z,z^{R})$ [93, Cor. 7.1]. Given the difficulty of finding an optimal bidirectional parsing, the question of how much worse can unidirectional parsings be is of interest.

10.1 Lower Bounds on $r$ and $z$

In this section we exhibit an infinite family of strings for which $z=\Omega(b\log n)$, which shows that the gap between bidirectionality and unidirectionality is significantly larger than what was previously known. The idea is to show that the phrases we defined in previous sections (i.e., starting at positions $\mathit{SA}[p]$ where $p$ starts a $\mathit{BWT}$ run) induce a valid bidirectional macro scheme of size $2r$, and then use Fibonacci strings as the family where $z=\Omega(r\log n)$ [84].

Definition 4

Let $p_{1},p_{2},\ldots,p_{r}$ be the positions that start runs in $\mathit{BWT}$, and let $s_{1}<s_{2}<\ldots<s_{r}$ be the positions $\{\mathit{SA}[p_{i}],1\leq i\leq r\}$ in $T$ where phrases start (note that $s_{1}=1$ because \mathit{BWT}[\mathit{ISA}[1]]=\$$ is a size-1 run). Assume s_{r+1}=n+1$. Let also$\phi(i)=\mathit{SA}[\mathit{ISA}[i]-1]$if$\mathit{ISA}[i]>1$and$\phi(i)=\mathit{SA}[n]$otherwise. Then we define the *macro scheme of the$\mathit{BWT}$* as follows:

For each $1\leq i\leq r$, $T[\phi(s_{i})..\phi(s_{i+1}-2)]$ is copied from $T[s_{i}..s_{i+1}-2]$. 2. 2.

For each $1\leq i\leq r$, $T[\phi(s_{i+1}-1)]$ is stored explicitly.

Lemma 23

The macro scheme of the $\mathit{BWT}$ is a valid bidirectional macro scheme, and thus $r\geq b$.

Proof

Lemma 19, proved for $\mathit{ISA}$, applies verbatim on $T$, since we define the phrases identically. Thus $\phi(j-1)=\phi(j)-1$ if $[j-1..j]$ is within a phrase. From Lemma 22, it also holds $T[j-1]=T[\phi(j)-1]$. Therefore, we have that $\phi(s_{i}+k)=\phi(s_{i})+k$ for $0\leq k<s_{i+1}-s_{i}-1$, and therefore $T[\phi(s_{i}),\ldots,\phi(s_{i+1}-2)]$ is indeed a contiguous range. We also have that $T[\phi(s_{i})..\phi(s_{i+1}-2)]=T[s_{i}..s_{i+1}-2]$, and therefore it is correct to make the copy. Since $\phi$ is a permutation, every position of $T$ is mentioned exactly once as a target in points 1 and 2.

Finally, it is easy to see that we can recover the whole $T$ from those $2r$ directives. We can, for example, follow the cycle $\phi^{k}(n)$, $k=0,\ldots,n-1$ (note that $T[\phi^{0}(n)]=T[n]$ is stored explicitly), and copy $T[\phi^{k}(n)]$ to $T[\phi^{k+1}(n)]$ unless the latter is explicitly stored. ∎

We are now ready to obtain the lower bound on bidirectional versus unidirectional parsings. We recall that, with $z$, we refer to the Lempel-Ziv parsing that allows source/target overlaps.

Theorem 10.1

There is an infinite family of strings over an alphabet of size 2 for which $z=\Omega(b\log n)$.

Proof

Consider the family of the Fibonacci stings, $F_{1}=a$, $F_{2}=b$, and $F_{k}=F_{k-1}F_{k-2}$ for all $k>2$. As shown by Prezza [84, Thm. 25], for $F_{k}$ we have $r=O(1)$ [67] and $z=\Theta(\log n)$ [30]. By Lemma 23, it also holds $b=O(1)$, and therefore $z=\Omega(b\log n)$. ∎

10.2 Upper bounds on $g$ and $z$

We first prove that $g_{rl}=O(b\log(n/b))$, and then that $z\leq 2g_{rl}$. For the first part, we will show that Lemmas 16 and 17 can be applied to any bidirectional scheme, which will imply the result.

Let a bidirectional scheme partition $T[1..n]$ into $b$ chunks $B_{1},\ldots,B_{b}$, such that each $B_{i}=T[t_{i}..t_{i}+\ell_{i}-1]$ is either $(1)$ copied from another substring $T[s_{i}..s_{i}+\ell_{i}-1]$ with $s_{i}\not=t_{i}$, which may overlap $T[t_{i}..t_{i}+\ell_{i}-1]$, or $(2)$ formed by $\ell_{i}=1$ explicit symbol

We define the function $f:[1..n]\rightarrow[1..n]$ so that, in case (1), $f(t_{i}+j)=s_{i}+j$ for all $0\leq j<\ell_{i}$, and in case (2), $f(t_{i})=-1$. Then, the bidirectional scheme is valid if there is an order in which the sources $s_{i}+j$ can be copied onto the targets $t_{i}+j$ so that we can rebuild the whole of $T$.

Being a valid scheme is equivalent to saying that $f$ has no cycles, that is, there is no $k>0$ and $p$ such that $f^{k}(p)=p$: Initially we can set all the explicit positions (type (2)), and then copy sources with known values to their targets. If $f$ has no cycles, we will eventually complete all the positions in $T$ because, for every $T[p]$, there is a $k>0$ such that $f^{k}(p)=-1$, so we can obtain $T[p]$ from the symbol explicitly stored for $T[f^{k-1}(p)]$.

Consider a locally consistent parsing of $W=T$ into blocks. We will count the number of different blocks that appear, as this is equal to the number of nonterminals produced in the first round. We will charge to each chunk $B$ the first and the last block that intersects it. Although a block overlapping one or more consecutive chunk boundaries will be charged several times, we do not charge more than $2b$ overall. On the other hand, we do not charge the other blocks, which are strictly contained in a chunk, because they will be charged somewhere else, when they appear intersecting an extreme of a chunk. We show this is true in both types of blocks:

If the block is a pair of left- and right-alphabet symbols, $W[p..p+1]=ab$, then it holds $[f(p-1)..f(p+2)]=[f(p)-1..f(p)+2]$ because $W[p..p+1]$ is strictly contained in a chunk. Moreover, $W[f(p)-1..f(p)+2]=W[p-1..p+2]$. That is, the block appears again at $[f(p)..f(p)+1]$, surrounded by the same symbols. Thus by Lemma 15, the locally consistent parsing also forms a block with $W[f(p)..f(p+1)]$. If this block is not strictly contained in another chunk, then it will be charged. Otherwise, by the same argument, $W[f(p)-1..f(p+1)+1]=W[f(p)-1..f(p)+2]$ will be equal to $W[f^{2}(p)-1..f^{2}(p)+2]$ and a block will be formed with $W[f^{2}(p)..f^{2}(p)+1]$. Since $f$ has no cycles, there is a $k>0$ for which $f^{k}(p)=-1$. Thus for some $l<k$ it must be that $W[f^{l}(p)-1..f^{l}(p)+2]$ is not contained in a chunk. At the smallest such $l$, the block $W[f^{l}(p)..f^{l}(p)+1]$ will be charged to the chunk whose boundary it touches. Therefore, $W[p..p+1]$ is already charged to some chunk and we do not need to charge it at $W[p..p+1]$. 2. 2.

If the block is a maximal run $W[p..p+\ell-1]=a^{\ell}$, then it also holds $[f(p-1)..f(p+\ell)]=[f(p)-1..f(p)+\ell]$, because all the area $[p-1..p+\ell]$ is within the same chunk. Moreover, $W[f(p-1)..f(p+\ell)]=ba^{\ell}c=W[p-1..p+\ell]$ with $b\not=a$ and $c\not=a$, because the run is maximal. It follows that the block $a^{\ell}$ also appears in $W[f(p)..f(p+\ell-1)]$, since the parsing starts by forming blocks with the maximal runs. If $W[f(p)..f(p+\ell-1)]$ is not strictly contained in a chunk, then it will be charged, otherwise we can repeat the argument with $W[f^{2}(p)-1..f^{2}(p)+\ell]$. Once again, since $f$ has no cycles, the block will eventually be charged at some $[f^{l}(p)..f^{l}(p+\ell-1)]$, so we do not need to charge it at $W[p..p+\ell-1]$.

Therefore, we produce at most $2b$ distinct blocks, and the RLCFG has at most $2b$ nonterminals. For the second round, we replace all the blocks of length $2$ or more by their corresponding nonterminals. The new sequence, $W^{\prime}$, is guaranteed to have length at most $(3/4)n$ by Lemma 15. We define a new bidirectional scheme on $W^{\prime}$, as follows:

The symbols that were explicit in $W$ are also explicit in $W^{\prime}$. 2. 2.

The nonterminals associated with the blocks of $W$ that intersect the first or last position of a chunk in $W$ (i.e., those that were charged to the chunks) are stored as explicit symbols. 3. 3.

For the non-explicit chunks $B_{i}=W[t_{i}..t_{i}+\ell_{i}-1]$ of $W$, let $B^{\prime}_{i}$ be obtained by trimming from $B_{i}$ the first and last block that overlaps $B_{i}$. Then $B^{\prime}_{i}$ appears inside $W[s_{i}..s_{i}+\ell_{i}-1]$, where the same sequence of blocks is formed because of the locally consistent parsing. The sequence of nonterminals associated with the blocks of $B_{i}^{\prime}$ therefore forms a chunk in $W^{\prime}$, pointing to the identical sequence of nonterminals that appear as blocks inside $W[s_{i}..s_{i}+\ell_{i}-1]$.

Let the original bidirectional scheme be formed by $b_{1}$ chunks of type (1) and $b_{2}$ of type (2), thus $b=b_{1}+b_{2}$. Now $W^{\prime}$ has at most $b_{1}$ chunks of type (1) and $b_{2}+2b_{1}$ chunks of type (2). After $k$ rounds, the sequence is of length at most $(3/4)^{k}n$ and it has at most $b_{1}$ chunks of type (1) and $b_{2}+2kb_{1}$ chunks of type (2), so we have generated at most $b_{2}+2kb_{1}\leq 2bk$ nonterminals. Therefore, if we choose to perform $k=\log_{4/3}(n/b)$ rounds, the sequence will be of length at most $b$ at the grammar size will be $O(b\log(n/b))$. To complete the process, we add $O(b)$ nonterminals to reduce the sequence to a single initial symbol.

Theorem 10.2

Let $T[1..n]$ have a bidirectional scheme of size $b$. Then there exists a run-length context-free grammar of size $g_{rl}=O(b\log(n/b))$ that generates $T$.

With Theorem 10.2, we can also bound the size $z$ of the Lempel-Ziv parse [61] that allows overlaps. The size without allowing overlaps is known to be bounded by the size of the smallest CFG, $z_{no}\leq g$ [88, 18]. We can easily see that $z\leq 2g_{rl}$ also holds by extending an existing proof [18, Lem. 9] to handle the run-length rules. Let us call left-to-right parse to any parsing of $T$ where each new phrase is a letter or it occurs previously in $T$.

Theorem 10.3

Let a RLCFG of size $g_{rl}$ expand to a text $T$. Then the Lempel-Ziv parse (allowing overlaps) of $T$ produces $z\leq 2g_{rl}$ phrases.

Proof

Consider the parse tree of $T$, where all internal nodes representing any but the leftmost occurrence of a nonterminal are pruned and left as leaves. The number of nodes in this tree is precisely $g_{rl}$. We say that the internal node of nonterminal $X$ is its definition. Our left-to-right parse of $T$ is a sequence $Z[1..z]$ obtained by traversing the leaves of the pruned parse tree left to right. For a terminal leaf, we append the letter to $Z$. For a leaf representing nonterminal $X$, such that the subtree of its definition generated $Z[i..j]$, we append to $Z$ a reference to the area $T[x..y]$ expanded by $Z[i..j]$.

Rules $X\rightarrow Y^{t}$ are handled as follows. First, we expand them to $X\rightarrow Y\cdot Y^{t-1}$, that is, the node for $X$ has two children for $Y$, and it is annotated with $t-1$. Since the right child of $X$ is not the first occurrence of $Y$, it must be a leaf. The left child of $X$ may or may not be a leaf, depending on whether $Y$ occurred before or not. Now, when our leaf traversal reaches the right child $Y$ of a node $X$ indicating $t-1$ repetitions, we append to $Z$ a reference to $T[x..y+(t-2)(y-x+1)]$, where $T[x..y]$ is the area expanded by the first child of $X$. Note that source and target overlap if $t>2$. Thus a left-to-right parse of size $2g_{rl}$ exists, and Lempel-Ziv is the optimal left-to-right parse [61, Thm. 1]. ∎

We then obtain a result on the long-standing open problem of finding the approximation ratio of Lempel-Ziv compared to the smallest bidirectional scheme (the bound this is tight as a function of $n$, according to Theorem 10.1).

Theorem 10.4

Let $T[1..n]$ have a bidirectional scheme of size $b$. Then the Lempel-Ziv parsing of $T$ allowing overlaps has $z=O(b\log(n/b))$ phrases.

We can also derive upper bounds for $g$, the size of the smallest CFG, and for $z_{no}$, the size of the Lempel-Ziv parse that does not allow overlaps. It is sufficient to combine the results that $z_{no}\leq g$ [88, 18] and that $g=O(z\log(n/z))$ [39, Lem. 8] with the previous results.

Theorem 10.5

Let $T[1..n]$ have a bidirectional scheme of size $b$. Then there exists a context-free grammar of size $g=O(b\log^{2}(n/b))$ that generates $T$.

Theorem 10.6

Let $T[1..n]$ have a bidirectional scheme of size $b$. Then the Lempel-Ziv parsing of $T$ without allowing overlaps has $z_{no}=O(b\log^{2}(n/b))$ phrases.

10.3 Map of the Relations between Repetitiveness Measures

Figure 3 (left) illustrates the known asymptotic bounds that relate various repetitiveness measures: $z$, $z_{no}$, $r$, $g$, $g_{rl}$, $b$, and $e$ (the size of the CDAWG [16]). We do not include $m$, the number of maximal matches [6], because it can be zero for all the $n!$ strings of length $n$ with all distinct symbols [6], and thus it is below the Kolmogorov complexity. Yet, we use the fact that $m\leq e$ to derive other lower bounds on $e$.

The bounds $e\geq\max(m,z,r)$ and $e=\Omega(g)$ are from Belazzougui et al. [6, 5], $z_{no}\leq g=O(z_{no}\log(n/z_{no}))$ is a classical result [88, 18] and it also holds $g=O(z\log(n/z))$ [39, Lem. 8]; $b\leq z$ holds by definition [93]. The others were proved in this section.

There are also several lower bounds on further possible upper bounds, for example, there are text families for which $g=\Omega(g_{rl}\log n)$ and $z_{no}=\Omega(z\log n)$ (i.e., T=a^{n-1}\$$); g=\Omega(z_{no}\log n/\log\log n)[[47](#bib.bib47)];e\geq m=\Omega(\max(r,z)\cdot n)[[6](#bib.bib6)] and thuse=\Omega(g\cdot n/\log n)$since$g=O(z\log n)$;$\min(r,z)=\Omega(m\cdot n)[[6](#bib.bib6)];r=\Omega(z_{no}\log n)[[6](#bib.bib6), [84](#bib.bib84)];z=\Omega(r\log n)[[84](#bib.bib84)];r=\Omega(g\log n/\log\log n)$(since on a de Bruijn sequence of order$k$on a binary alphabet we have$r=\Theta(n)[[6](#bib.bib6)],z=O(n/\log n)$, and thus$g=O(z\log(n/z))=O(n\log\log n/\log n)). Those are shown in the right of Figure [3](#S10.F3). We are not aware of a separation between z$and$g_{rl}$. From the upper bounds that hold for every string family, we can also deduce that, for example, there are text families where$r=\Omega(z\log n)$and thus$r=\Omega(b\log n)$(since$r=\Omega(z_{no}\log n)$);${g,g_{rl},z_{no}}=\Omega(r\log n)$(since$z=\Omega(r\log n)$) and thus$z=\Omega(b\log n)$(since$r\geq b$, see Theorem 10.1).

Thus, there are no simple dominance relations between $r$ and $z$, $z_{no}$, $m$, $g$, or $g_{rl}$. Experimental results, on the other hand [65, 58, 6, 21], show that it typically holds $z\approx z_{no}<m<r\approx g\ll e$.

Appendix 0.A Failure in the Analysis based on RePair and Relatives

We show that the proofs that RePair or other related compressors reach a grammar of size $O(r\log(n/r))$ in the literature [43, 44] are not correct. The mistake is propagated in other papers [33, 1].

The root of their problem is the possibly inconsistent parsing of equal runs, that is, the same text may be parsed into distinct nonterminals, with the inconsistencies progressively growing from the run breaks. Locally consistent parsing avoids this problem. Although the run breaks do grow from one round to the next, locally consistent parsing reduces the text from $n$ to $(3/4)n$ symbols in each round, parsing the runs in the same way (except for a few symbols near the ends). Thus the text decreases fast compared to the growth in the number of runs.

Note that we are not proving that those compressors can actually produce a grammar of size $\omega(r\log(n/r))$ on some particular string family; we only show that the proofs in the literature do not ensure that they always produce a grammar of size $O(r\log(n/r))$. In fact, our experiments show that both are rather competitive (especially RePair), so it is possible that they actually reach this space bound and the right proof is yet to be found.

0.A.1 RePair

The argument used to claim that RePair compression reached $O(r\log(n/r))$ space was simple [44]. As we show in Lemma 16 (specialized with Lemma 18), as we traverse the permutation $\pi=\mathit{LF}$, consecutive pairs of symbols can only change when they fall in different runs; therefore there are only $r$ different pairs. Since RePair iteratively replaces the most frequent pair by a nonterminal [60], the chosen pair must appear at least $n/r$ times. Thus the reduced text is of size $n-n/r=n(1-1/r)$. Since collapsing a pair of consecutive symbols into a nonterminal does not create new runs, we can repeat this process on the reduced text, where there are still $r$ different pairs. After $k$ iterations, the grammar is of size $2k$ and the text is of size $n(1-1/r)^{k}$, and the result follows.

To prepare for the next iteration, they do as follows. When they replace a pair $ab$ by a nonterminal $A$, they replace $a$ by $A$ and $b$ by a hole, which we write “  ”. Then they remove all the positions in $\pi$ that fall into holes, that is, if $W[\pi(i)]=\underline{~{}~{}}$, they replace $\pi(i)\leftarrow\pi(\pi(i))$, as many times as necessary. Finally, holes are removed and the permutation is mapped to the new sequence. Then they claim that the number of places where $W[i..i+1]\not=W[\pi(i)..\pi(i)+1]$ is still $r$.

However, this is not so. Let us regard the chains [33] in $W$, which are contiguous sections of the cycle of $\pi$ where $W[i..i+1]=W[\pi(i)..\pi(i)+1]$. The chains are also $r$, since we find the $r$ run breaks in another order. Consider a chain where the pair, $ab$, always falls at the end of runs, alternatively followed by $c$ or by $d$. We write this as $ab|c$ and $ab|d$, using the bar to denote run breaks. Now assume that the current step of RePair chooses the pair $bd$ because it is globally the most frequent. Then every $b|d$ will be replaced by $B|\underline{~{}~{}}$, and the chain of $ab$s will be broken into many chains of length 1, since $ab$ is now followed by $aB$ and $aB$ is followed by $ab$. Since each run end may break a chain once, we have $2r$ chains (and runs) after one RePair step. Effectively, we are adding a new run break preceding each existing one: $ab|c$ becomes $a|b|c$ and $a|b|d$ becomes $a|B|\underline{~{}~{}}$.

If the runs grow by $r$ at each iteration, the analytical result does not follow. The size of $W$ after $k$ replacements becomes $n\prod_{i=1}^{k}(1-1/(ir))$. This is a function that does tend to zero, but very slowly. In particular, after $k=c\cdot r$ replacements, and assuming only $p\cdot r$ of the runs grow (i.e., the factors are $(1-1/(r(1+(i-1))p)$, the product is $\frac{n\Gamma(cr+(r-1)/(rp))}{\Gamma(cr+1/p)\Gamma((r-1)/(rp))}$, which tends to $n$ as $r$ tends to infinity for any constants $c$ and $p$. That is, the text has not been significantly reduced after adding $O(r)$ rules.

Even if we avoid replacing pairs that cross run breaks, there are still problems. Consider a substring $W[i..i+2]=abc$, where we form the rule $A\rightarrow ab$ and replace the string by $A\underline{~{}~{}}c$. Now let $W[j..j+1]=|bc$ start a run (so it does not have to be preceded by $a$), and that $\pi(j)=i+1$. Now, since $i+1$ is a hole, we will replace $\pi(j)\leftarrow\pi(i+1)$, which breaks the chain at the edge $j\rightarrow i+1$. We have effectively created a new run starting at $W[j+1]$, $|b|c$, and the number of runs could increase by $r$ in this way.

RePair can be thought of as choosing the longest chain at each step, and as such it might insist on a long chain that is affected by all the $r$ run breaks while reducing the text size only by $n/r$. The next method avoids this (somewhat pathological) worst-case by processing all the chains at each iteration, so that the $r$ run breaks are spread across all the chains, and the text size is divided by a constant factor at each round. Still, their parsing is even less consistent and the number of runs may grow faster.

0.A.2 Following $\pi$

They [44] also propose another method, whose proof is also subtly incorrect. They follow the cycle of $\pi=\mathit{LF}$ (or its inverse, $\Psi$). For each chain of equal pairs $ab$, they create the rule $A\rightarrow ab$ and replace every $ab$ by $A\underline{~{}~{}}$ along the chain. If some earlier replacement in the cycle changed some $a$ to $\underline{~{}~{}}$ or some $b$ to a nonterminal $B$, they skip that pair. When the chain ends (i.e., the next pair was not $ab$ before the cycle of changes started), they switch to the new chain (and pair). This also produces only $r$ pairs in the first round, and reduces the text to at most $2n/3$, so after several rounds, a grammar of size $O(r\log(n/r))$ would be obtained.

The problem is that the number of runs may grow fast from one iteration to the next, due to inconsistent parsing. Consider that we are following a chain, replacing the pair $B\rightarrow ba$, until we hit a run break and start a new chain for the pair $C\rightarrow cb$. At some point, this new chain may reach a substring $cba$ that was converted to $cB\underline{~{}~{}}$ along the previous chain. Actually, this $ba$ must have been the first in the chain of $ba$, because otherwise the $b$ must come from the previous element in the chain of $ba$. The chain for $cb$ follows for some more time, without replacing $cb$ by $C\underline{~{}~{}}$ because it collides with the $B$. Thus this chain has been cut into two, with pairs $C\underline{~{}~{}}$ and $cB$.

After finishing with this chain, we enter a new chain $D\rightarrow dc$. At some point, this chain may find a $dcb$ that was changed to $dC\underline{~{}~{}}$ (this is the beginning of the chain of $cb$, as explained before), and thus it cannot change the pair. For some time, all the pairs $dc$ we find overlap those $cb$ of the previous chain, until we reach the point where the chain of $cb$ “met” the chain of $ba$. Since those $cb$ were not changed to $C\underline{~{}~{}}$, we can now again convert $dc$ to $D\underline{~{}~{}}$. This chain has then been cut into three, with pairs $D\underline{~{}~{}}$, $dC$, and then again $D\underline{~{}~{}}$.

Now we may start a new chain $E\rightarrow ed$, which after some time touches an $edc$ that was converted to $eD\underline{~{}~{}}$, follows besides the chain of $dc$ and reaches the point where $dc$ was not converted to $D\underline{~{}~{}}$ and thus it starts generating the pairs $E\underline{~{}~{}}$ again. Then it reaches the point where $dc$ was again converted to $D\underline{~{}~{}}$ and thus it produces pairs $eD$ again. Thus the chain for $ed$ was split into four. Note, however, that in the process we removed one of the three chains created by $dc$, since we replaced all those $edC\underline{~{}~{}}$ by $E\underline{~{}~{}}C\underline{~{}~{}}$. So we have created 2 chains from $cb$, 2 from $dc$, and 4 from $ed$. A new replacement $F\rightarrow fe$ will create 5 chains and remove one from $ed$, so we have 2, 2, 3, 5. Yet a new replacement $G\rightarrow gf$ will create 6 chains and remove 2 from $fe$, so we have 2, 2, 3, 3, 6.

The number of new chains (and hence runs) may then grow quadratically, thus at the end of the round we may have $\Theta(r^{2})$ runs and at least $n/2$ symbols. The sum is optimized for $k=\frac{\log n}{1+2\log r}$ rounds, where the size of the grammar is $\Theta(n^{1-\frac{1}{2\log r+1}})$, which even for $r=2$ is $\Theta(n^{2/3})$.

Note, again, that we are not proving that, for a concrete family of strings, this method does not produce a grammar of size $O(r\log(n/r))$. We are just exposing a failure of the proof in the literature [44], where it is said that the runs do not increase without proving it, as a consistent parsing of the runs is assumed. Indeed, two runs, $|abcdef|x$ and $|abcdefg|$, where the first leads to the second by $\pi$, may be parsed in completely different forms, $aBDF$ and $ACEg$, if the chains we follow are $F\rightarrow fx$, $E\rightarrow ef$, $D\rightarrow de$, $C\rightarrow cd$, $B\rightarrow bc$, and $A\rightarrow ab$.