Fast, Small, and Simple Document Listing on Repetitive Text Collections

TL;DR

This paper introduces a simple, space-efficient document listing index tailored for repetitive text collections, significantly improving retrieval speed and space usage by exploiting document array repetitiveness and grammar compression.

Contribution

The paper presents a novel document listing index that leverages grammar compression of the document array to efficiently handle highly repetitive collections, outperforming existing methods.

Findings

Index achieves faster query times compared to alternatives.

Significant reduction in space usage for repetitive collections.

Effective exploitation of document array repetitiveness through grammar compression.

Abstract

Document listing on string collections is the task of finding all documents where a pattern appears. It is regarded as the most fundamental document retrieval problem, and is useful in various applications. Many of the fastest-growing string collections are composed of very similar documents, such as versioned code and document collections, genome repositories, etc. Plain pattern-matching indexes designed for repetitive text collections achieve orders-of-magnitude reductions in space. Instead, there are not many analogous indexes for document retrieval. In this paper we present a simple document listing index for repetitive string collections of total length $n$ that lists the $ndoc$ distinct documents where a pattern of length $m$ appears in time $\mathcal{O}(m+ndoc \cdot \log n)$. We exploit the repetitiveness of the document array (i.e., the suffix array coarsened to document identifiers) to grammar-compress it while precomputing the answers to nonterminals, and store them in grammar-compressed form as well. Our experimental results show that our index sharply outperforms existing alternatives in the space/time tradeoff map.

Figures

Tables

Table 1. Table 1. Statistics for document collections (small, medium, and large variants): Collection name; Size in megabytes; RLCSA bits per symbol (bps); Docs , number of documents; Doc size , average document length; number of Patterns ; Occs , average number of occurrences; Doc occs , average number of document occurrences; Occs/doc , average ratio of occurrences to document occurrences.

\rowfont Collection	Size	RLCSA	Docs	Doc size	Patterns	Occs	Doc occs	Occs/doc
	($n$)	(bps)	($D$)	($n/D$)		($\mathrm{𝑜𝑐𝑐}$)	($\mathrm{𝑛𝑑𝑜𝑐}$)	($\frac{\mathrm{𝑜𝑐𝑐}}{\mathrm{𝑛𝑑𝑜𝑐}}$)
Page	110	0.18	$60$	$\mathrm{1\hspace{0.17em}919\hspace{0.17em}382}$	$7658$	$781$	$3$	242.75
	641	0.11	$190$	$\mathrm{3\hspace{0.17em}534\hspace{0.17em}921}$	$\mathrm{14\hspace{0.17em}286}$	$2601$	$6$	444.79
	1037	0.13	$280$	$\mathrm{3\hspace{0.17em}883\hspace{0.17em}145}$	$\mathrm{20\hspace{0.17em}536}$	$2889$	$7$	429.04
Revision	110	0.18	$8834$	$\mathrm{13\hspace{0.17em}005}$	$7658$	$776$	$371$	2.09
	640	0.11	$\mathrm{31\hspace{0.17em}208}$	$\mathrm{21\hspace{0.17em}490}$	$\mathrm{14\hspace{0.17em}284}$	$2592$	$1065$	2.43
	1035	0.13	$\mathrm{65\hspace{0.17em}565}$	$\mathrm{16\hspace{0.17em}552}$	$\mathrm{20\hspace{0.17em}536}$	$2876$	$1188$	2.42
Influenza	137	0.32	$\mathrm{100\hspace{0.17em}000}$	$1436$	$269$	$\mathrm{532\hspace{0.17em}739}$	$\mathrm{88\hspace{0.17em}525}$	6.02
	321	0.26	$\mathrm{227\hspace{0.17em}356}$	$1480$	$269$	$\mathrm{1\hspace{0.17em}248\hspace{0.17em}428}$	$\mathrm{202\hspace{0.17em}437}$	6.17

Full text

1. Introduction

Document retrieval is a family of problems aimed at retrieving documents from a set that are relevant to a query pattern. In a rather general setting, both documents and patterns are arbitrary strings. This encompasses the well-known application of natural language and Web searching, but also many others of interest in bioinformatics, software development, and multimedia retrieval, to name a few [22].

The most fundamental document retrieval problem, on top of which more sophisticated ones are built, is document listing. This problem aims at simply returning the list of documents where the pattern appears. An obvious solution to document listing resorts to pattern matching: find all the $\mathit{occ}$ positions where the pattern appears, and then return the $\mathit{ndoc}$ different documents where those positions lie. This solution requires time $\Omega(\mathit{occ})$ and the output is of size $\operatorname{\mathcal{O}}(\mathit{ndoc})$, so the approach is very inefficient if $\mathit{ndoc}\ll\mathit{occ}$ (i.e., if the pattern appears many times in the same documents). A better solution, which however applies only in natural language settings, resorts to inverted indexes [1]. These restrict the possible patterns to sequences of words and store the list of the documents where each word appears, thereby solving document listing via intersections of the lists of the pattern words.

Muthunkishnan [20] designed the first linear-space and optimal-time index for general string collections. Given a collection of total length $n$, he builds an index of $\operatorname{\mathcal{O}}(n)$ words that lists the $\mathit{ndoc}$ documents where a pattern of length $m$ appears in time $\operatorname{\mathcal{O}}(m+\mathit{ndoc})$. While linear space is deemed as sufficiently small in classic scenarios, the solution is impractical for very large text collections unless one resorts to disk, which is orders of magnitude slower. Sadakane [26] showed how to reduce the space of Muthukrishnan’s index to that of the statistically-compressed text plus $\operatorname{\mathcal{O}}(n)$ bits, while raising the time complexity to only $\operatorname{\mathcal{O}}(m+\mathit{ndoc}\cdot\lg n)$ if the appropriate underlying pattern-matching index is used [2].

The sharp growth of text collections is a concern in many recent applications, outperforming Moore’s Law in some cases [27]. Fortunately, many of the fastest-growing text collections are highly repetitive: each document can be obtained from a few large blocks of other documents. These collections arise in different areas, such as repositories of genomes of the same species (which differ from each other by a small percentage only) like the 100K-genome project222https://www.genomicsengland.co.uk/about-genomics-england/the-100000-genomes-project, software repositories that store all the versions of the code arranged in a tree or acyclic graph like GitHub333https://github.com/search?q=is:public, versioned document repositories where each document has a timeline of versions like Wikipedia444https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia, etc. On such text collections, statistical compression is ineffective [14] and even $\operatorname{\mathcal{O}}(n)$ bits of extra space can be unaffordable.

Repetitiveness is the key to tackle the fast growth of these collections: their amount of new material grows much slower than their size. For example, version control systems compress those collections by storing the list of edits with respect to some reference document that is stored in plain form, and reconstruct it by applying the edits to the reference version. Much more challenging, however, is to index those collections in small space so as to support fast pattern matching or document retrieval tasks. To date, there exist several pattern matching indexes for repetitive text collections (see a couple of studies [21, 10] and references therein). However, there are not many document retrieval indexes for repetitive text collections [5, 8, 23]. Most of these indexes [26, 8] rely on a pattern-matching index needs $\Omega(n)$ bits in order to offer $\operatorname{\mathcal{O}}(\lg n)$ time per retrieved document.

In this paper we introduce new simple and efficient document listing indexes aimed at highly repetitive text collections. Like various preceding indexes, we achieve $\operatorname{\mathcal{O}}(m+\mathit{ndoc}\cdot\lg n)$ search time, yet our indexes are way faster and/or smaller than previous ones on various repetitive datasets, because they escape from the space/time tradeoff of the pattern-matching index. Our main idea is as follows: we use the document array $\mathit{DA}[1..n]$ [20], which projects the entries of the suffix array [19] to the document where each position belongs. Document listing boils down to listing the distinct integers in a range $\mathit{DA}[sp..ep]$, where $sp$ and $ep$ are found in time $\operatorname{\mathcal{O}}(m)$. Array $\mathit{DA}$ must be grammar-compressible since the differential suffix array is known to be so on repetitive texts [11, 10]. We then build a balanced binary context-free grammar that generates (only) $\mathit{DA}$. This allows us retrieve any individual cell of $\mathit{DA}$ in time $\operatorname{\mathcal{O}}(\lg n)$ and any range $\mathit{DA}[sp..ep]$ in time $\operatorname{\mathcal{O}}(ep-sp+\lg n)$. We can then implement existing indexes [26, 8] within much less space and without affecting their time complexities. Further, we propose a new simple index based on the grammar-compressed array $\mathit{DA}$. Our compression guarantees that any range $\mathit{DA}[sp..ep]$ is covered by $\operatorname{\mathcal{O}}(\lg n)$ nonterminals. For each nonterminal of the grammar, we store the list of the distinct documents appearing in it. The set of all the lists is grammar-compressed as well, as done in previous work [5, 8]. We then merge the lists of the $\operatorname{\mathcal{O}}(\lg n)$ nonterminals that cover $\mathit{DA}[sp..ep]$, in time $\operatorname{\mathcal{O}}(\mathit{ndoc}\cdot\lg n)$.

2. Preliminaries

A document $\mathit{T}$ is a sequence of symbols over an alphabet $\Sigma=[1..\sigma]$, terminated by a special symbol \mathit{\}$that is lexicographically smaller than any symbol of$\Sigma$.

A collection $\mathcal{D}$ is a set of $\mathit{d}$ documents $\mathcal{D}=\{\mathit{T}_{1},\dots,\mathit{T}_{\mathit{d}}\}$. $\mathcal{D}$ is commonly represented as the concatenation of all its documents, $\mathcal{T}=\mathit{T}_{1}\mathit{T}_{2}\dots\mathit{T}_{\mathit{d}}$, of length $|\mathcal{T}|=\mathit{n}$.

A pattern $\mathit{P}$ is a string over the same alphabet $\Sigma$ with length $|\mathit{P}|=\mathit{m}$. It occurs $\mathit{occ}$ times in $\mathcal{T}$, and appears in $\mathit{ndoc}$ documents.

Text indexes. The suffix tree [28] of a string $\mathcal{T}$ is a compressed digital tree storing all the suffixes $\mathcal{T}[i..\mathit{n}]$, for all $1\leq i\leq\mathit{n}$. The suffix tree node reached by following the symbols of a pattern $P$

is called the locus of $P$ and is the ancestor of all the $\mathit{occ}$ leaves corresponding to the positions of $P$ in $\mathcal{T}$. The suffix tree uses $\operatorname{\mathcal{O}}(n\lg n)$ bits and lists all the occurrences of $\mathit{P}$ in time $\operatorname{\mathcal{O}}(\mathit{m}+\mathit{occ})$.

The suffix array [19] $\mathit{SA}[1..n]$ of a string $\mathcal{T}[1..n]$ is a permutation of the starting positions of all the suffixes of $\mathcal{T}$ in lexicographic order, $\mathcal{T}[\mathit{SA}[i],\mathit{n}]<\mathcal{T}[\mathit{SA}[i+1],\mathit{n}]$ for all $1\leq i<\mathit{n}$. $\mathit{SA}$ can be binary searched to obtain the range $\mathit{SA}[\mathit{sp}..\mathit{ep}]$ of all the suffixes prefixed by $\mathit{P}$ (note $\mathit{occ}=ep-sp+1$). Thus the occurrences of $P$ can be listed in time $\operatorname{\mathcal{O}}(\mathit{m}\lg\mathit{n}+\mathit{occ})$. The suffix array takes $\mathit{n}\lg\mathit{n}$ bits.

Compressed suffix arrays ($\operatorname{CSA}$s) [24] are space-efficient representations of the suffix array. They find the interval $[\mathit{sp}..\mathit{ep}]$ corresponding to $\mathit{P}[1..m]$ in time $\operatorname{t_{search}(\mathit{m})}$, and access any cell $\mathit{SA}[i]$ in time $\operatorname{t_{lookup}(\mathit{n})}$. Their size in bits, $|\operatorname{CSA}|$, is usually bounded by $\operatorname{\mathcal{O}}(\mathit{n}\lg\sigma)$.

Grammar compression. Grammar compression of a string $S[1..n]$ replaces it by a context-free grammar (CFG) $\mathcal{G}$ that uniquely generates $S$.

This CFG $\mathcal{G}$ may require less space than the original sequence $S$, especially when $S$ is repetitive.

Finding the smallest CFG $\mathcal{G}^{*}$ generating the input $S$ is NP-hard [16], but various $\operatorname{\mathcal{O}}(\lg(n/|\mathcal{G}^{*}|))$-approximations exist. In particular, we are interested in approximations that are binary (i.e., the maximum arity of the parse tree is 2) and balanced (i.e., any substring is covered by $\operatorname{\mathcal{O}}(\lg n)$ maximal nodes of the parse tree) [25, 3, 13].

3. Related Work

Muthukrishnan [20] proposed the first optimal-time linear-space solution to the document listing problem. He defines the document array $\mathit{DA}[1..n]$ of $\mathcal{T}$, where $\mathit{DA}[i]$ stores the identifier of the document to which $\mathcal{T}[\mathit{SA}[i]]$ belongs. The document listing problem is then translated into computing the $\mathit{ndoc}$ distinct identifiers in the interval $\mathit{DA}[\mathit{sp}..\mathit{ep}]$ corresponding to the pattern $P$. He uses a suffix tree to find $sp$ and $ep$ in time $\operatorname{\mathcal{O}}(m)$, and then an algorithm that finds the $\mathit{ndoc}$ distinct numbers in the range in time $\operatorname{\mathcal{O}}(\mathit{ndoc})$.

Sadakane [26] adapts the method of Muthukrishnan to use much less space. He replaces the suffix tree by a CSA, and mimics the algorithm to find the distinct numbers in $\mathit{DA}[sp..ep]$ using only $\operatorname{\mathcal{O}}(n)$ bits of space. Within $|\operatorname{CSA}|+\operatorname{\mathcal{O}}(n)$ bits, he performs document listing in time $\operatorname{\mathcal{O}}(\operatorname{t_{search}(\mathit{m})}+\mathit{ndoc}\cdot\operatorname{t_{lookup}(\mathit{n})})$. Using a particular CSA [2] the space is $n\lg\sigma+o(n\lg\sigma)+\operatorname{\mathcal{O}}(n)$ bits and the time is $\operatorname{\mathcal{O}}(m+\mathit{ndoc}\cdot\lg n)$.

There are many other classical and compact indexes for document listing. We refer the reader to a survey [22] and focus on those aimed at repetitive text collections.

Gagie et al. [8] proposed a technique adapting Sadakane’s solution to highly repetitive collections. They show that the technique to find the distinct elements of $\mathit{DA}[sp..ep]$ can be applied almost verbatim on an array they call interleaved longest-common-prefix array ($\mathit{ILCP}$). On repetitive collections, this array can be decomposed into a small number $\rho$ of equal values, which allows them represent it in little space. The ILCP index requires $|\operatorname{CSA}|+\operatorname{\mathcal{O}}((\rho+\mathit{d})\lg\mathit{n})$ bits of space and solves document listing in time $\operatorname{\mathcal{O}}(\operatorname{t_{search}(\mathit{m})}+\mathit{ndoc}\cdot\operatorname{t_{lookup}(\mathit{n})})$.

Gagie et al. [8] proposed another radically different approach, called Precomputed Document Lists (PDL). The idea is to store the list of the documents where (the corresponding substring of) each suffix tree node appears. Then the search consists of finding the locus of $P$ and returning its list. To reduce space, however, only some sampled nodes store their lists, and so document listing requires merging the lists of the maximal sampled nodes descending from the locus node. To further save space, the lists are grammar-compressed, which is effective when the collection $\mathcal{D}$ is repetitive.

To bound the query time, the deepest sampled nodes cover at most $\mathit{b}$ leaves, and a factor $\beta$ restricts the work done per document to be merged in the unions of the lists. The index then requires $|\operatorname{CSA}|+\operatorname{\mathcal{O}}((\mathit{n}/\mathit{b})\lg\mathit{n})$ bits, and the document listing time is $\operatorname{\mathcal{O}}(\operatorname{t_{search}(\mathit{m})}+\mathit{ndoc}\cdot\beta\cdot h+\mathit{b}\cdot\operatorname{t_{lookup}(\mathit{n})})$, where $h$ is the height of the suffix tree.

A problem in all the revisited $\operatorname{CSA}$-based solutions are the extra $\Theta((\mathit{n}\lg\mathit{n})/\operatorname{t_{lookup}(\mathit{n})})$ bits that must be included in $|\operatorname{CSA}|$ in order to get $\Theta(\operatorname{t_{lookup}(\mathit{n})})$ time per document. This space does not decrease with repetitiveness, forcing all these indexes to use $\Omega(n)$ bits to obtain time $\operatorname{\mathcal{O}}(\operatorname{t_{search}(\mathit{m})}+\mathit{ndoc}\cdot\lg n)$, for example.

Claude and Munro [5] propose the first index for document listing based on grammar compression, which escapes from the problem above. They extend a grammar-based pattern-matching index [6] by storing the list of the documents where each nonterminal appears. Those lists are grammar-compressed as well. The index searches for the minimal nonterminals that contain $P$ and merges their lists. While it does not offer relevant space or query time guarantees, the index performs well in practice. Navarro [23] extends this index in order to obtain space guarantees and $\operatorname{\mathcal{O}}(m^{2}+m\lg^{2}n)$ time, but the scheme is difficult to implement.

4. Our Document Listing Index

Like most of the previous work, we solve the document listing problem by computing the $\mathit{ndoc}$ distinct documents in the interval $\mathit{DA}[\mathit{sp}..\mathit{ep}]$ corresponding to the pattern $P$, found with a $\operatorname{CSA}$ in time $\operatorname{\mathcal{O}}(\operatorname{t_{search}(\mathit{m})})$. Instead of also using the $\operatorname{CSA}$ to compute the values of $\mathit{DA}$ (and thus facing the problem of using $\Theta((n\lg n)/\operatorname{t_{lookup}(\mathit{n})})$ bits to compute a cell in time $\Theta(\operatorname{t_{lookup}(\mathit{n})})$, as it happens in previous work [26, 8]), we store the array $\mathit{DA}$ directly, yet in grammar-compressed form. This is promising because the suffix array of repetitive collections is known to have large areas $\mathit{SA}[i..i+\ell]$ that appear shifted by 1 elsewhere, $\mathit{SA}[j..j+\ell]$, that is, $\mathit{SA}[i+k]=\mathit{SA}[j+k]+1$ for all $0\leq k\leq\ell$ [18, 10]. Except for the $d$ entries of $\mathit{SA}$ that point to the ends of the documents, it also holds that $\mathit{DA}[i+k]=\mathit{DA}[j+k]$. Grammar compression is then expected to exploit those large repeated areas in $\mathit{DA}$.

To answer the queries efficiently, we use an idea similar to the one introduced in PDL [8] and the Grammar-index [5]: precomputing and storing the answers of document listing queries, and grammar-compressing those lists as well. An important difference with them is that PDL stores lists for suffix tree nodes and the Grammar-index stores lists for nonterminals of the grammar of $\mathcal{T}$. Our index, instead, stores lists for the nonterminals of the grammar of $\mathit{DA}$. This is much simpler because we do not store a suffix tree topology (like PDL) nor a complex grammar-based pattern-matching index (like the Grammar-index): we simply find the interval $\mathit{DA}[sp..ep]$ using the $\operatorname{CSA}$, fetch the nonterminals covering it, and merge their lists. By using a binary balanced grammar on $\mathit{DA}$, we ensure that any document is obtained in the merging only $\operatorname{\mathcal{O}}(\lg n)$ times, which leads to our worst-case bound of $\operatorname{\mathcal{O}}(\mathit{ndoc}\cdot\lg n)$. PDL and the Grammar-index cannot offer such a logarithmic-time guarantee.

4.1. Structure

The first component of our index is a $\operatorname{CSA}$ suitable for repetitive collections, of which we are only interested in the functionality of finding the interval $\mathit{SA}[sp..ep]$ corresponding to a pattern $P[1..m]$. For example, we can use the Run-Length CSA (RLCSA) variant of Gagie et al. [10], which offers times $\operatorname{t_{search}(\mathit{m})}=\operatorname{\mathcal{O}}(m\lg\lg_{w}\sigma)$ within $\operatorname{\mathcal{O}}(r\lg n)$ bits, or $\operatorname{t_{search}(\mathit{m})}=\operatorname{\mathcal{O}}(m)$ within $\operatorname{\mathcal{O}}(r\lg(n/r)\lg n)$ bits, where $r$ is the number of equal-letter runs in the Burrows-Wheeler Transform of $\mathcal{T}$. This also upper-bounds the number of areas $\mathit{SA}[i..i+\ell]$ into which $\mathit{SA}$ can be divided such that each area appears elsewhere shifted by 1 [17].

The second component is the grammar $\mathcal{G}$ that generates $\mathit{DA}[1..\mathit{n}]$, which must be binary and balanced. Such grammars can be built so as to ensure that their total size is $\operatorname{\mathcal{O}}(r\lg(n/r)\lg n)$ bits [9], which is of the same order of the first component.

The third component are the lists $D_{v}$ of the distinct documents that appear in the expansion of each nonterminal $v$ of $\mathcal{G}$. These lists are stored in ascending order to merge them easily. To reduce their size, the set of sequences $D_{1},\ldots,D_{g}$ are grammar-compressed as a whole in a new grammar $\mathcal{G}^{\prime}$, ensuring that no nonterminal of $\mathcal{G}^{\prime}$ expands beyond a list $D_{v}$. Each list $D_{v}$ can then be obtained in optimal time, $\operatorname{\mathcal{O}}(|D_{v}|)$, from a nonterminal of $\mathcal{G}^{\prime}$.

4.2. Document listing

Given a pattern $P[1..m]$, we use the $\operatorname{CSA}$ to find the range $[\mathit{sp}..\mathit{ep}]$ where the occurrences of $P$ lie in the suffix array, in time $\operatorname{\mathcal{O}}(\operatorname{t_{search}(\mathit{m})})$. We then find the maximal nodes of the parse tree of $\mathit{DA}$ that cover $\mathit{DA}[\mathit{sp}..\mathit{ep}]$. Finally, we decompress the lists of the nonterminals corresponding to those maximal nodes, and compute their union.

Since $\mathcal{G}$ is binary and balanced, there are $\operatorname{\mathcal{O}}(\lg n)$ maximal nonterminals that cover $\mathit{DA}[sp..ep]$ in the parse tree. By storing the length to which each nonterminal of $\mathcal{G}$ expands, we can easily find those $\operatorname{\mathcal{O}}(\lg n)$ maximal nonterminals in time $\operatorname{\mathcal{O}}(\lg n)$, by (virtually) descending in the parse tree from the initial symbol of $\mathcal{G}$ towards the area $\mathit{DA}[sp..ep]$.

To merge the $\operatorname{\mathcal{O}}(\lg n)$ lists of documents in ascending order, we use an atomic heap [7] (see practical considerations in the next section). This data structure performs insert and extractmin operations in constant amortized time, when storing $\operatorname{\mathcal{O}}(\lg^{2}\mathit{n})$ elements. We then insert the heads of the $\operatorname{\mathcal{O}}(\lg\mathit{n})$ lists in the atomic heap, extract the minimum, and insert the next element of its list. If we extract the same document many times, we report only one copy. We then expand and merge the lists $D_{v_{1}},\dots,D_{v_{k}}$ in time $\operatorname{\mathcal{O}}(|D_{v_{1}}|+\dots+|D_{v_{k}}|)$.

Since each distinct document we report may appear in the $\operatorname{\mathcal{O}}(\lg\mathit{n})$ lists, our document listing solution takes time $\operatorname{\mathcal{O}}(\operatorname{t_{search}(\mathit{m})}+\mathit{ndoc}\cdot\lg\mathit{n})$. By using the RLCSA that occupies $\operatorname{\mathcal{O}}(r\lg(n/r)\lg n)$ bits, the total time is $\operatorname{\mathcal{O}}(\mathit{m}+\mathit{ndoc}\cdot\lg\mathit{n})$.

4.3. Plugging-in other indexes

Our grammar-compressed $\mathit{DA}$, without the lists $D_{v}$, can be used to replace the CSA component that requires $\Theta((n\lg n)/\operatorname{t_{lookup}(\mathit{n})})$ bits to compute a cell in time $\Theta(\operatorname{t_{lookup}(\mathit{n})})$. These indexes actually access cells $\mathit{SA}[i]$ in order to obtain $\mathit{DA}[i]$. Our grammar-compressed $\mathit{DA}$ offers $\operatorname{\mathcal{O}}(\lg n)$ access time within $O(r\lg(n/r)\lg n)$ bits of space.

Therefore, we can implement Sadakane’s solution [26], as well as ILCP and PDL [8] all answering in time $\operatorname{\mathcal{O}}(m+\mathit{ndoc}\cdot\lg n)$, and replacing the $\operatorname{\mathcal{O}}((n\lg n)/\operatorname{t_{lookup}(\mathit{n})})$ part of their $|\operatorname{CSA}|$ space by $O(r\lg(n/r)\lg n)$ bits (which also accounts for the RLCSA variant that finds $[sp..ep]$ in time $\operatorname{\mathcal{O}}(m)$. We can also implement the brute-force solution in time $O(m+\mathit{occ}+\lg n)$ and $O(r\lg(n/r)\lg n)$ bits by extracting the whole $\mathit{DA}[sp..ep]$.

5. Practical Considerations

5.1. Compressed suffix array

We use a practical RLCSA [18, called RLFM+ in there] that uses $(r\lg\sigma+2r\lg(n/r))(1+o(1))$ bits of space and offers search time $\operatorname{t_{search}(\mathit{m})}$ in $\operatorname{\mathcal{O}}(m\lg r)\subseteq\operatorname{\mathcal{O}}(m\lg n)$. Since we do not need to compute cells of $\mathit{SA}$ with this structure, we do not need to spend the $\operatorname{\mathcal{O}}((n\lg n)/\operatorname{t_{lookup}(\mathit{n})})$ bits, and as a result the contribution of the RLCSA to the total space is negligible.

5.2. Grammar compressor

We choose Re-Pair [15] to obtain both $\mathcal{G}$ and $\mathcal{G}^{\prime}$, since it performs very well in practice. Re-Pair repeatedly replaces the most frequent pair of adjacent symbols with a new nonterminal, until every pair is unique. Upon ties in frequency, we give priority to the pairs whose symbols have been generated earlier, which in practice yielded rather balanced grammars in all the cases we have tried.

Re-Pair yields a binary grammar, but the top-level is a sequence of terminals and nonterminals. We then complete the grammar by artificially adding a parse tree on top of the final sequence left by Re-Pair. To minimize the height of the resulting grammar, we merge first the pairs of nonterminals with shorter parse trees.

We store the $\mathit{g}$ grammar rules as an array $G$ taking $2\mathit{g}\lg(\mathit{g}+\mathit{d})$ bits of space, so that if $A_{i}$ is the $i$th nonterminal of the grammar, it holds that $A_{i}\rightarrow A_{G[2i]}A_{G[2i+1]}$.

When building $\mathcal{G}^{\prime}$, we concatenate all the lists $D_{v}$ and separate them with unique numbers larger than $d$, to ensure that Re-Pair will not produce nonterminals that cross from one list to another. After running Re-Pair, we remove the separators but do not complete the grammars, as all we need is to decompress any $D_{v}$ in optimal time. We represent all the reduced sets $D^{\prime}_{v}$ as a sequence $D^{\prime}$, marking the beginning of each set in a bitvector $B$. The beginning of $D_{v}^{\prime}$ is found with operation $select(B,v)$, which finds the $v$th 1 in $B$. This operation can be implemented in constant time using $o(|B|)$ further bits [4].

5.3. Sampling

The largest component of our index is the set of compressed lists $D_{v}^{\prime}$. To reduce this space, we will store those lists only for some sampled nonterminals $v$ of $\mathcal{G}$. The list of a nonsampled nonterminal $v$ is then obtained by merging those of the highest sampled descendants of $v$ in the parse tree, which yields a space/time tradeoff.

We use a strategy similar to PDL [8], based on parameters $b$ and $\beta$. We define a sampled tree by sampling some nodes from the parse tree. First, no leaf $v$ of the sampled tree can have an expansion larger than $b$, so that we spend time $\operatorname{\mathcal{O}}(b\lg b)$ to obtain its sorted list directly from $\mathcal{G}$. To this aim, we sample all the nonterminals $v$ of $\mathcal{G}$ with parent $w$ such that $|D_{v}|\leq b<|D_{w}|$. Those are the leaves of the sampled tree, which form a partition of $\mathit{DA}$.

Second, for any nonsampled node $v$ with $|D_{v}|>b$, we must be able to build $D_{v}$ by merging other precomputed lists of total length $\leq\beta|D_{v}|$. This implies that generating $D_{v}$ costs $\operatorname{\mathcal{O}}(\beta\lg n)$ times more than having $D_{v}^{\prime}$ stored and just decompressing it.

We first assume that the sampled tree contains all the ancestors of the sampled leaves and then proceed bottom-up in the sampled tree, removing some nodes from it. Any node $v$ with parent $w$ and children $u_{1},...,u_{k}$ is removed if $\sum_{i=1}^{k}|D_{u_{i}}|\leq\beta\cdot|D_{v}|$. In this case, the nodes $u_{i}$ become children of $w$ in the sampled tree.

At query time, if a node $v$ of interest is not sampled, we collect all the lists of its highest sampled descendants. Therefore, on a parse tree of height $h$ we may end up merging many more than the original $\operatorname{\mathcal{O}}(h)$ lists $D_{1},\ldots,D_{k}$, but have the guarantee that the merged lists add up to size at most $\beta\cdot(|D_{1}|+\cdots+|D_{k}|)$. To merge the lists we use a classical binary heap instead of an atomic heap, so the cost per merged element is $\operatorname{\mathcal{O}}(\lg n)$.

We may then spend $k\cdot b\lg b=\operatorname{\mathcal{O}}(hb\lg b)$ time in extracting and sorting the lists $D_{v}$ of size below $b$. The other lists $D_{v}$ may lead to merging $\beta|D_{v}|$ elements. The total cost over the $k=\operatorname{\mathcal{O}}(h)$ lists is then $\operatorname{\mathcal{O}}(hb\lg b+\beta(|D_{1}|+\cdots+|D_{k}|)\lg n)\subseteq\operatorname{\mathcal{O}}(hb\lg b+\mathit{ndoc}\cdot\beta h\lg n)$. In terms of complexity, if we choose for example $b=\operatorname{\mathcal{O}}(\lg n/\lg\lg n)$, $\beta=\operatorname{\mathcal{O}}(1)$, and the grammar is balanced, $h=\operatorname{\mathcal{O}}(\lg n)$, then the total cost of merging is $\operatorname{\mathcal{O}}(\mathit{ndoc}\cdot\lg^{2}n)$.

6. Experiments and Results

We evaluate different variants of our indexes and compare them with the state of the art. We use the experimental framework proposed by Gagie et al. [8].

6.1. Document collections

To test various kinds of repetitiveness scenarios, we performed several experiments with real and synthetic datasets. We used the same document collections tested by Gagie et al. [8], available at https://jltsiren.kapsi.fi/rlcsa. Table 1 summarizes some statistics on the collections and the patterns used in the queries.

Real collections. Page and Revision are collections formed by all the revisions of some selected pages from the Wikipedia in Finnish language.

In Page, there is a document for each selected article, that also includes all of its revisions. In the case of Revision, each page revision becomes a separate document. Influenza is another repetitive collection composed of sequences of the H. influenzae virus genomes.

Synthetic collections. We also used two types of synthetic collections to explore the effect of collection repetitiveness on document listing performance in more detail.

Concat and Version are similar to Page and Revision, respectively. We use 10 and 100 base documents of length $1000$ each, extracted at random from the English file of Pizza&Chili (http://pizzachili.dcc.uchile.cl). Besides, we include variants of each base document, generated using different mutation probabilities ($0.001$, $0.003$, $0.01$, and $0.03$). A mutation is a replacement by a different random symbol. In collection Version, each variant becomes a separate document. In Concat, all variants of the same base document are concatenated into a single document.

Queries. The query patterns for Page and Revision datasets are Finnish words of length $\geq 5$ that occur in the collections.

For Influenza, the queries are substrings of length $4$ extracted from the dataset. In the case of Concat and Version, the patterns are terms selected from an MSN query log. See Gagie et al. [8] for a more detailed description.

6.2. Compared indexes

Grammar-Compressed Document Array (GCDA). This is our main proposal. We use the balanced variant of the Re-Pair compressor implemented by Navarro555https://www.dcc.uchile.cl/gnavarro/software/repair.tgz. To sample the parse tree, we test several parameter configurations for the block size $\mathit{b}$ and factor $\beta$.

Brute force (Brute). This family of algorithms is the most basic and simple solution to the document listing problem.

They use a $\operatorname{CSA}$ to retrieve all the document identifiers in $\mathit{DA}[\mathit{sp}..\mathit{ep}]$, sort them, and report each of them once. Brute-L uses the $\operatorname{CSA}$ to extract the values $\mathit{DA}[i]$. Brute-D, instead, uses an explicit document array $\mathit{DA}$. Finally, Brute-C is our variant using the grammar-compressed $\mathit{DA}$. From the grammar tree of height $h$ and storing the length of the expansion of each nonterminal, we extract the range $\mathit{DA}[sp..ep]$ in time $\operatorname{\mathcal{O}}(h+ep-sp)$.

Sadakane (Sada). Sada-L is the original index proposed by Sadakane [26].

Sada-D speeds up the query time by explicitly storing $\mathit{DA}$. Sada-C stores $\mathit{DA}$ in grammar-compressed form, where each individual cell $\mathit{DA}[i]$ is extracted in time $\operatorname{\mathcal{O}}(h)$.

Interleaved Longest Common Prefix (ILCP). ILCP-L is an implementation of the ILCP index proposed by Gagie et

al. [8] using a run-length encoded ILCP array. ILCP-D is a variant that uses the document array instead of the $\operatorname{CSA}$ functionality. ILCP-C uses, instead, our grammar-compressed $\mathit{DA}$, which accesses any cell in time $\operatorname{\mathcal{O}}(h)$.

Precomputed Document Lists (PDL). PDL-BC and PDL-RP are implementations of the PDL algorithm

proposed by Gagie et al. [8]. The first one uses a Web graph compressor [12] on the set of lists, whereas PDL-RP uses Re-Pair compression. Both variants use block size $\mathit{b}=256$ and factor $\beta=16$, as recommended by their authors.

Grammar-based (Grammar). This is an implementation of the index by Claude and Munro

[5]. It uses Re-Pair on the collection $\mathcal{T}$ and on the set of lists. This index is the only tested solution that does not use a $\operatorname{CSA}$.

We implemented GCDA on C++, using several succinct data structures from the SDSL library666https://github.com/simongog/sdsl-lite. We used existing C++ implementations of the indexes Brute, Sada, ILCP and PDL, which were tested by Gagie et al. [8]777https://jltsiren.kapsi.fi/software/doclist.tgz, and modified the versions -C by using $\mathit{DA}$ in grammar-compressed instead of in plain form.

All tested indexes except Grammar use a suffix array to compute the interval $[sp..ep]$ corresponding to pattern $P$. We used a RLCSA implementation888https://jltsiren.kapsi.fi/rlcsa that is optimized for highly repetitive text collections. To compute entries $\mathit{SA}[i]$, the RLCSA uses a suffix array sampling, which requires significant space as explained. Our index does not use this operation, but it is required for the indexes Brute-L, Sada-L, ILCP-L, and both variants of PDL. We use 32 as the value for this sample rate, as it gave good results in previous tests [8]. The exception is Brute-L, which uses a RLCSA optimized to extract whole ranges $\mathit{SA}[sp..ep]$ [10].999https://github.com/nicolaprezza/r-index The column RLCSA of Table 1 gives the space used by the RLCSA without suffix array samples.

Our machine has two Intel(R) Xeon(R) CPU E5-2407 processors running at $2.40\text{\,}\mathrm{GHz}$ and $250\text{\,}\mathrm{GiB}$ of RAM. The operating system was Debian with Linux kernel 4.9.0-8-amd64. All indexes were compiled using g++ version 6.3.0 with flags -O3 -DNDEBUG.

6.3. Tuning our main index

Figure 1 shows the tradeoff between time and space of GCDA on small real collections. We tested GCDA with 4 different sizes of block $\mathit{b}$: $128$, $256$, $512$, and $1024$. For each block size, we used 3 different factors $\beta$ ($4$, $8$, and $16$), which are represented with increasing color darkness in the plots. The configuration $\mathit{b}=512$ and $\beta=4$ shows to be a good general-purpose choice of parameter values, and we stick to it from now on.

The lower-right plot of Figure 2 shows the space required by the main components of our index. As the number of documents in the collection grows and their size decreases, the weight of the grammar-compressed $\mathit{DA}$, and even more, of the grammar-compressed lists of documents, becomes dominant. Note also that Influenza is the least repetitive collection.

6.4. Comparison on real collections

Figure 2 shows the tradeoff between time and space for all tested indexes on the real collections. Our main index, GCDA, and the -C variants of the other indexes we adapted, are clearly dominant in a large portion of the space/time map. Most of the previous indexes are way slower, way larger, or both, than ours. The best previous tradeoffs, PDL-BC and PDL-RP [8], are much closer, but still they are almost always slower and larger than GCDA.

For all versions of Page, where there are few large documents and our grammars compress very well, GCDA requires only $0.48$–$0.56$ bits per symbol (bps) and answers queries in less than 16 microseconds. The index using the least space is Grammar, which requires $0.21$–$0.35$ bps. Grammar is way out of the plot, however, because it requires 1.2–3.4 milliseconds to solve the queries, that is, 205–235 times slower than GCDA.101010As in previous work [8], Grammar was not built on the largest dataset of Page. The next smallest index is our variant Brute-C, which uses 0.35–0.55 bps and is generally smaller than GCDA, but slower by a factor of 2.6–6.7. Brute-L, occupying 0.38–0.60 bps, is also smaller in some cases, but much slower (180–1080 microseconds, out of the plot). GCDA sharply outperforms all the other indexes in space, and also in time (only Sada-D is 6% faster in the small collection, yet using 18 times more space). The closest competitors, PDL-BC and PDL-RP, are 4.4–5.0 times larger and $2.8$–$5.0$ times slower than GCDA.

In the case of Revision, where there are more and smaller documents, GCDA uses 0.73–0.88 bps and answers queries in less than 150 microseconds. Again Grammar uses the least space, 0.26–0.42 bps, but once again at the price of being $8$–$30$ times slower than GCDA. The case of Brute-L is analogous: 0.38–0.60 bps but over 8 times slower than GCDA. Instead, our variant Brute-C is a relevant competitor, using 0.45–0.76 bps and being less than 60% slower than GCDA. The other relevant index is our variant ILCP-C, using almost the same space and time of GCDA. The group GCDA/Brute-C/ILCP-C forms a clear sweetpoint in this collection. The closest competitors, again PDL-BC and PDL-RP, are 3.1–3.8 times larger and 1.2–1.9 times slower than GCDA.

Influenza, with many small documents, is the worst case for the indexes. GCDA uses 4.46–4.67 bps and answers queries within 115 milliseconds. Many indexes are smaller than GCDA, but only our variants form a relevant space/time tradeoff: ILCP-C uses 2.88–3.37 bps, Brute-C uses 2.42–2.86 bps, and Sada-C uses 4.96–5.40 bps. All the -C variants obtain competitive times, and ILCP-C even dominates GCDA (it answers queries within 65 milliseconds, taking less than 60% of the time of GCDA). The other indexes outperforming GCDA in time are -D variants, which are at least 3.7 times larger than GCDA and 5.2 times larger than ILCP-C.

6.5. Comparison on synthetic collections

Figure 3 compares the indexes on synthetic collections. These allow us study how the indexes evolve as the repetitiveness decreases, in a scenario of few large documents (Concat) and many smaller documents (Version). We combine in a single plot the results for different mutation rates of a given collection and number of base documents. The plots show the increasing mutation rates using variations of the same color, from lighter to darker. All the -L variants and Grammar are omitted because they were significantly slower.

On collection Concat, GCDA essentially outperforms all the other indexes. In the case of the version composed by $10$ base documents, our index obtains the best space/time tradeoff by a wide margin. Only Brute-C is smaller than GCDA, but 8–9 times slower. On the other hand, various indexes are slightly faster than GCDA, but much larger (from Sada-D, which is up to 30% faster but 7 times larger, to Sada-C, which is 15% faster but at least 4 times larger). With the other variant of Concat ($100$ base documents), our index offers the best space and time for all mutation rates. Only PDL-RP is 6% faster in its best case, but 2.2 times larger. Further, GCDA retains its space/time performance as repetitiveness decreases, whereas the competing indexes worsen fast in one or both aspects.

On Version, composed by $10\,000$ documents of length $1000$, GCDA is also a dominant solution, retaining its time performance as repetitiveness decreases and outperforming all the -D variants in space up to a mutation rate of 1%. Other competing indexes are our variants Brute-C and ILCP-C (the only one dominating GCDA in some cases), as well as PDL-BC and PDL-RP in the case of 100 base documents. The strange behavior of the PDL indexes in both collections with 10 base documents is briefly discussed in the original article [8].

7. Conclusions

We have presented simple and efficient indexes for document listing on repetitive string collections. They find the $\mathit{ndoc}$ documents where a pattern of length $m$ appears in a collection of size $n$ in time $\operatorname{\mathcal{O}}(\mathit{m}+\mathit{ndoc}\cdot\lg\mathit{n})$. The indexes uses grammar-compression of the document array, and perform better as the collection is more repetitive.

Our experimental results show that our main index, GCDA, outperforms the best previous solutions by a fair margin in time and/or space on various repetitive collections. From the previous indexes, only PDL [8] gets close, but it is almost always dominated by GCDA in both space and time. GCDA performs well in space for mutation rates up to 1%, whereas its query time is mostly insensitive to the repetitiveness. Other previous solutions (especially ILCP [8] and brute force) that we adapted to run on our grammar-compressed document array also display unprecedented performance on repetitive texts, competing with GCDA.

For the final version of this paper, we plan to combine the PDL indexes with a grammar-compressed document array as well, which we omitted for lack of time. A line of future work is to further reduce the space of GCDA and our index variants that use the grammar-compressed document array, by using a more clever encoding of the grammars that may nearly halve their space at a modest increase in time [11]. Another line is to extend the index to support top-$k$ document retrieval, that is, find the $k$ documents where $P$ appears most often. For example, following previous ideas [8], we can store the list of documents where each nonterminal appears in decreasing order of frequency, and use algorithms developed for inverted indexes [1] on the $\operatorname{\mathcal{O}}(\lg n)$ lists involved in a query. The frequency of the candidates can be efficiently counted on repetitive collections [8].