Regular Expression Search on Compressed Text

TL;DR

This paper introduces a new algorithm for efficiently searching regular expressions in compressed text, significantly reducing search time compared to existing methods by leveraging specialized data structures.

Contribution

It presents a novel algorithm and data structures for regular expression search in compressed text, achieving near-optimal complexity and improved performance over prior approaches.

Findings

The algorithm reports match counts in linear time relative to compressed size.

The implementation --zearch-- is up to 25% faster than current state-of-the-art methods.

Efficient data structures enable near-optimal complexity bounds.

Abstract

We present an algorithm for searching regular expression matches in compressed text. The algorithm reports the number of matching lines in the uncompressed text in time linear in the size of its compressed version. We define efficient data structures that yield nearly optimal complexity bounds and provide a sequential implementation --zearch-- that requires up to 25% less time than the state of the art.

Tables

Table 1. Table 1: Sizes (in MB) of the compressed files and (de)compression times (in seconds). Maximum compression levels enabled. (Blue = best; bold black = second best; red = worst).

			Compressed size				Compression time				Decompression time
		File	\asciifamilyLZW	\asciifamilyrepair	\asciifamilyzstd	\asciifamilylz4	\asciifamilyLZW	\asciifamilyrepair	\asciifamilyzstd	\asciifamilylz4	\asciifamilyLZW	\asciifamilyrepair	\asciifamilyzstd	\asciifamilylz4
Uncompressed size	1 MB	Logs	0.19	0.08	0.07	0.12	0.04	0.19	0.51	0.03	0.02	0.01	0.01	0.004
		Subtitles	0.36	0.13	0.11	0.15	0.04	0.25	0.3	0.03	0.02	0.01	0.01	0.004
		Books	0.42	0.34	0.27	0.43	0.04	0.29	0.42	0.08	0.02	0.02	0.01	0.004
	500 MB	Logs	96	38	33	65	16.9	123.2	819.1	13.3	7.8	5.5	1.1	0.64
		Subtitles	191	66	55	114	19.9	169.3	415.2	22.8	8.6	8.2	1.2	0.81
		Books	206	153	129	216	20.2	198.6	646.3	40.6	8.6	9.7	2.0	0.8

Table 2. Table 2: Analysis of the values s ~ ℓ subscript ~ 𝑠 ℓ \tilde{s}_{\ell} and s ( σ ) i subscript 𝑠 subscript 𝜎 𝑖 s_{(σ)_{i}} obtained when considering different regular expressions to search Subtitles (100 MB uncompressed long). The fifth column of the first row indicates that when considering the expression “ \asciifamily I .* you”, for 75% of the grammar rules we have s ~ ℓ ≤ 5 subscript ~ 𝑠 ℓ 5 \tilde{s}_{\ell}\leq 5 while s 3 = 512 superscript 𝑠 3 512 s^{3}=512 .

Expression	$s$	$s^3$	percentiles for ${\tilde{s}}_{\ell }$					$s^2$	percentiles for $s_{{\left(\sigma \right)}_i}$
			$50$ %	$75$ %	$95$ %	$98$ %	$100$ %		$50$ %	$75$ %	$95$ %	$98$ %	$100$ %
“\asciifamilyI .* you”	8	512	3	13	15	17	28	64	3	3	5	5	8
“\asciifamily.*[A-Za-z ]{5}”	7	343	14	25	48	48	48	49	11	14	14	14	14
“\asciifamily.*[A-Za-z ]{10}”	12	1728	29	51	86	95	98	144	16	26	29	29	29
“\asciifamily.*[A-Za-z ]{20}”	22	10648	57	87	132	153	198	484	23	38	52	58	59
“\asciifamily(((((.)*.)*.)*.)*.)*”	6	216	12	29	209	209	209	36	29	29	29	29	29

Equations

$${\text{asciifamilyN}}_{\tau }$$

$${\text{asciifamilyR}}_{\tau }$$

$$\sharp\text{\emph{matching lines} in }u={\text{{\asciifamily M}}_{τ}}+\left\{\begin{array}[]{ll}1&\text{if }{\text{{\asciifamily L}}_{τ}}\\ 0&otherwise\end{array}\\ \right.+\left\{\begin{array}[]{ll}1&\text{if }{\text{{\asciifamily N}}_{τ}}\land{\text{{\asciifamily R}}_{τ}}\\ 0&otherwise\end{array}\\ \right.$$

Full text

1 Introduction

The growing amount of information handled by modern systems demands efficient techniques both for compression, to reduce the storage cost, and for regular expression searching, to speed up querying. A type of query that is supported out of the box by many tools111Tools such as \asciifamilygrep, \asciifamilyripgrep, \asciifamilyawk and \asciifamilyag, among others, can be used to report the number of matching lines in a text. is counting: compute how many lines of the input text contain a match for the expression. When the text is given in compressed form, the fastest approach in practice is to query the uncompressed text as it is recovered by the decompressor.

We present an algorithm for counting the lines in a compressed text containing a match for a regular expression whose runtime does not depend on the size $N$ of the uncompressed text. Instead, it runs in time linear in the size of its compressed version. Furthermore, the information computed for counting can be used to perform an on-the-fly, lazy decompression to recover the matching lines from the compressed text. Note that, for reporting the matching lines, the dependency on $N$ in unavoidable.

The salient features of our approach are:

Generality.

Our algorithm is not tied to any particular grammar-based compressor. Instead, we consider the compressed text is given by a straight line program (SLP): a context-free grammar generating the uncompressed text and nothing else.

Finding the smallest SLP $g$ generating a text of length $N$ is an NP-hard problem, as shown by Charikar et al. [1], for which grammar-based compressors such as LZ78 [2], LZW [3], RePair [4] and Sequitur [5] produce different approximations. For instance, Hucke et al. [6] showed that the LZ78 algorithm produces a representation of size Ω\bigl{(}{|{g}|}{\cdot}(N/\log{N})^{2/3}\bigr{)} and the representation produced by the RePair algorithm has size Ω\bigl{(}{|{g}|}{\cdot}(\log{N}/\log\log{N})\bigr{)}. Since it is defined over SLPs, our algorithm applies to all such approximations, including $g$ itself.

Nearly optimal data structures.

We define data structures enabling the algorithm to run in time linear in the size of the compressed text. With these data structures our algorithm runs in $\mathcal{O}(p{\cdot}s^{3})$ time using $\mathcal{O}(p{\cdot}s^{2})$ space where $p$ is the size of the compressed text and $s$ is the size of the automaton built from the expression. When the automaton is deterministic, the complexity drops to $\mathcal{O}(p{\cdot}s)$ time and $\mathcal{O}(p{\cdot}s)$ space. Abboud et al. [7] showed that there is no combinatorial222Interpreted as any practically efficient algorithm that does not suffer from the issues of Fast Matrix Multiplication such as large constants and inefficient memory usage. algorithm improving thesetime complexity bounds beyond polylog factors, hence our algorithm is nearly optimal.

Efficient implementation.

We present \asciifamilyzearch, a purely sequential implementation of our algorithm which uses the above mentioned data structures.333\asciifamilyzearch can optionally report the matching lines. The experiments show that \asciifamilyzearch requires up to $25\scalebox{0.8}{\%}$ less time than the state of the art: running \asciifamilyhyperscan on the uncompressed text as it is recovered by \asciifamilylz4 (in parallel). Furthermore, when the grammar-based compressor achieves high compression ratio (above 13:1), running \asciifamilyzearch on the compressed text is as fast as running \asciifamilyhyperscan directly on the uncompressed text. This is the case, for instance, when working with automatically generated log files.

2 Notation

An alphabet $Σ$ is a nonempty finite set of symbols. A string $w$ is a finite sequence of symbols of $Σ$ where the empty sequence is denoted $ε$. Let ${|{w}|}$ denote the length of $w$ that we abbreviate to $\textdagger$ when $w$ is clear from the context. Further define $(w)_{i}$ as the $i$-th symbol of $w$ if $1≤i≤\textdagger$ and $ε$ otherwise. Similarly, $(w)_{i,j}$ denotes the substring, also called factor, of $w$ between the $i$-th and the $j$-th symbols, both included.

A finite state automaton (FSA or automaton for short) is a tuple $A=(Q,Σ,I,F,δ)$ where $Q$ is the (finite) set of states; $I\subseteq Q$ are the initial states; $Σ$ is the alphabet; $F\subseteq Q$ are the final states; and $δ\subseteq Q\times Σ\times Q$ are the transitions444Our definition prescribes $ε$-transitions which can be removed in $\mathcal{O}(s^{2})$ time adding no state.. The notions of accepting run, accepted string and language of an automaton (denoted ${\mathcal{L}(A)}$) are defined as expected. For clarity, we assume through the paper that $I$ and $F$ are disjoint. Otherwise $ε\in{\mathcal{L}(A)}$, therefore, for every string there is a factor ($ε$) in ${\mathcal{L}(A)}$.

A Straight Line Program, hereafter SLP, is a tuple $P=(V,Σ,\mathcal{R})$ where $V$ is the set of variables $\{X_{1},X_{2},…,X_{{|{V}|}}\}$, $Σ$ is the alphabet and $\mathcal{R}$ is the set of rules $\{X_{i}\textrightarrow α_{i}β_{i}\mid α_{i},β_{i}\in(Σ\cup\{X_{1},…,X_{i-1}\}\}$. We refer to $X_{{|{V}|}}\textrightarrow\alpha_{|{V}|}β_{|{V}|}$ as the axiom rule. We write $ρ⇒σ$, with $ρ,σ\in(Σ\cup V)^{*}$, if there exists $(ρ)_{i}=X$ and $(X\textrightarrow αβ)\in\mathcal{R}$ such that $σ=(ρ)_{1,i-1}αβ(ρ)_{i+1,{\dagger}}$. Denote the reflexo-transitive closure of $\Rightarrow$ by $⇒^{*}$. Clearly, the language generated by an SLP consists of a single string $w∈Σ^{*}$ and, by definition, ${|{w}|}>1$. Since ${\mathcal{L}(P)}=\{w\}$ we identify $w$ with ${\mathcal{L}(P)}$.

3 Counting Algorithm

Let $\return$ denote the new-line delimiter and $\widehat{Σ}=Σ{\setminus}\{\return\}$. Given a string $w\in Σ^{+}$ compressed as an SLP $P=(V,Σ,\mathcal{R})$ and an automaton $A=(Q,\widehat{Σ},I,F,δ)$ built from a regular expression, Algorithm 1 reports the number of lines in $w$ containing a match for the expression. Note that we deliberately ignore matches across lines.

As an overview, our algorithm computes some counting information for each alphabet symbol of the grammar (procedure init_automaton) which is then propagated, in a bottom-up manner, to the axiom rule. Such propagation is achieved by iterating through the grammar rules (loop in line 13) and combining, for each rule, the information for the symbols on the right hand side to obtain the information for the variable on the left (procedure count). Finally, the output of the counting is computed from the information propagated to the axiom symbol (line 20).

Define a line as a maximal factor of $w$ each symbol of which belongs to $\widehat{Σ}$, a closed line as a line which is not a prefix nor a suffix of $w$ and a matching line as a line in $\widehat{{\mathcal{L}(A)}}$, where $\widehat{{\mathcal{L}(A)}}=\widehat{Σ}^{*}{\cdot}{\mathcal{L}(A)}{\cdot}\widehat{Σ}^{*}$. The counting information of $τ\in(V\cup Σ)$, with $τ\Rightarrow^{*}u$ and $u\in Σ^{+}$, is the tuple ${\mathcal{C}_{τ}}={\left\langle{\text{{\asciifamily N}}_{τ}},{\text{{\asciifamily L}}_{τ}},{\text{{\asciifamily R}}_{τ}},{\text{{\asciifamily M}}_{τ}}\right\rangle}$ where

[TABLE]

Note that ${\text{{\asciifamily N}}_{τ}}$, ${\text{{\asciifamily L}}_{τ}}$ and ${\text{{\asciifamily R}}_{τ}}$ are boolean values while ${\text{{\asciifamily M}}_{τ}}$ is an integer. It follows from the definition that the number of matching lines in $u$, with $τ\Rightarrow^{*}u$, is given by the number of closed matching lines (${\text{{\asciifamily M}}_{τ}}$) plus the prefix of $u$ iff it is a matching line (${\text{{\asciifamily L}}_{τ}}$) and the suffix of $u$ iff it is a matching line (${\text{{\asciifamily R}}_{τ}}$) different from the prefix (${\text{{\asciifamily N}}_{τ}})$. Since whenever ${\text{{\asciifamily N}}_{τ}}=\text{\emph{false}}$ we have ${\text{{\asciifamily L}}_{τ}}={\text{{\asciifamily R}}_{τ}}$, it follows that

[TABLE]

Computing the counting information of $τ$ requires deciding membership of certain factors of $u$ in $\widehat{{\mathcal{L}(A)}}$. To solve such queries we adapt an algorithm of Esparza et al. [8] designed to decide whether the languages generated by a context-free grammar and an automaton intersect. The resulting algorithm iterates through the rules $(X{\textrightarrow}αβ)\in\mathcal{R}$ applying the following operation: add $(q_{1},X,q_{2})$ to $δ$ iff

(a) $(q_{1},α,q^{\prime}),(q^{\prime},β,q_{2})\in δ$,

(b) $(q_{1},β,q_{2})\in δ\text{ with }q_{1}\in I$, or

(c) $(q_{1},α,q_{2})\in δ\text{ with }q_{2}\in F$.

This operation corresponds to lines 15 to 17 of Algorithm 1. As a result, after processing the rule for $τ$, we have $(q_{1},τ,q_{2})\in δ$ iff the automaton moves from $q$ to $q^{\prime}$ reading

(a) $u$,

(b) a suffix of $u$ and $q_{1}\in I$, or

(c) a prefix of $u$ and $q_{2}\in F$.

Procedures count and init_automaton are quite straightforward, the main difficulty being the computation of ${\text{{\asciifamily M}}_{X}}$ which we explain next. Let $x,y\in Σ^{+}$ be the strings generated by $α$ and $β$, respectively. Given rule $X{\textrightarrow}αβ$, $X$ generates all the matching lines generated by $α$ and $β$ plus, possibly, a “new” matching line of the form $z{=}(x)_{i,{\dagger}}(y)_{1,j}$ with $1{<}i{\leq}{|{x}|}$ and $1{\leq}j{<}{|{y}|}$. Such an extra matching line appears iff both $α$ and $β$ generate a $\return$ symbol and either the suffix of $x$ or the prefix of $y$ matches the expression or there is a new match $m\in z$ with $m\notin x$, $m\notin y$ (line 18).

Example 1**.**

Let $A$ be an automaton with ${\mathcal{L}(A)}=\{ab,ba\}$. Consider the grammar rule $X{\textrightarrow}αβ$ with $α\Rightarrow^{*}ba\return a$ and $β\Rightarrow^{*}b\return aba$. Then $X\Rightarrow^{*}ba\return ab\return aba$.

The matching lines generated by $α$, $β$ and $X$ are, respectively, $\{ba\}$, $\{aba\}$ and $\{ba,ab,aba\}$. Furthermore ${\mathcal{C}_{α}}{=}{\left\langle\text{\emph{true}},\text{\emph{true}},\text{\emph{false}},0\right\rangle}$ and ${\mathcal{C}_{β}}{=}{\left\langle\text{\emph{true}},\text{\emph{false}},\text{\emph{true}},0\right\rangle}$.

Applying function count we find that ${\mathcal{C}_{X}}={\left\langle\text{\emph{true}},\text{\emph{true}},\text{\emph{true}},1\right\rangle}$. Therefore the number of matching lines is $1{+}1{+}1{=}3$, as expected. $\Diamond$

Note that the counting information computed by Algorithm 1 can be used to uncompress only the matching lines by performing a top-down processing of the SLP. For instance, given $X{\textrightarrow}αβ$ with ${\mathcal{C}_{X}}={\left\langle\text{\emph{true}},\text{\emph{true}},\text{\emph{false}},0\right\rangle}$ and ${\mathcal{C}_{α}}={\left\langle\text{\emph{true}},\text{\emph{true}},\text{\emph{false}},0\right\rangle}$, there is no need to decompress the string generated by $β$ since we are certain it is not part of any matching line (otherwise we should have ${\text{{\asciifamily M}}_{X}}>0$ or ${\text{{\asciifamily R}}_{X}}=\text{\emph{true}}$).

Next, we describe the data structures used to implement Algorithm 1 with nearly optimal complexity.

\SubSection

Data Structures We assume the alphabet symbols, variables and states are indexed and use the following data structures, illustrated in Figure 1: an array $\mathcal{A}$ with $p{+}{|{Σ}|}$ elements, where $p$ is the number of rules of the SLP, and two $s\times s$ matrices $\mathcal{M}$ and $\mathcal{N}$ where $s$ is the number of states of the automaton.

Each element $\mathcal{A}[i]$ contains the information related to variable $X_{i}$, i.e. ${\mathcal{C}_{X_{i}}}$ and the list of transitions labeled with $X_{i}$, ${\text{{\asciifamily E}}_{X_{i}}}$. We store ${\mathcal{C}_{X}}$ using one bit for each ${\text{{\asciifamily N}}_{X}}$, ${\text{{\asciifamily L}}_{X}}$ and ${\text{{\asciifamily R}}_{X}}$ and an integer for ${\text{{\asciifamily M}}_{X}}$. For each rule $X_{\ell}\textrightarrow α_{\ell}β_{\ell}$ the matrix $\mathcal{N}$ is set so that row $i$ contains all the states reachable from $q_{i}$ with a transition labeled with $β_{\ell}$, i.e. $\{q_{j}\mid(q_{i},β_{\ell},q_{j})\in δ\}$. If there are less than $s$ such states we use a sentinel value (${-}1$ in Figure 1). Finally, each element $\mathcal{M}[i][j]$ stores the index $\ell$ of the last variable for which $(q_{i},X_{\ell},q_{j})$ was added to $δ$. Note that since rules are processed one at a time, matrices $\mathcal{N}$ and $\mathcal{M}$ can be reused for all rules.

These data structures provide $\mathcal{O}(1)$ runtime for the following operations:

• •

Accessing the information corresponding to $α_{\ell}$ and $β_{\ell}$ at line 14 (using $\mathcal{A}$).

• •

Accessing the list of pairs $(q,q^{\prime})$ with $(q,α_{\ell},q^{\prime})\in δ$ at line 15 (using ${\text{{\asciifamily E}}_{X_{i}}}$).

• •

Accessing the list of states $q_{2}$ with $(q^{\prime},β_{\ell},q_{2})\in δ$ at line 16 (using $\mathcal{N}$).

• •

Inserting a pair $(q,q^{\prime})$ in ${\text{{\asciifamily E}}_{X_{i}}}$ at line 17 (using $\mathcal{M}$).

As a result, Algorithm 1 runs in $\mathcal{O}(p{\cdot}s^{3})$ time using $\mathcal{O}(p{\cdot}s^{2})$ space when the FSA built from the regular expression is non deterministic and it runs in $\mathcal{O}(p{\cdot}s)$ time and $\mathcal{O}(p{\cdot}s)$ space when the FSA is deterministic (each row of $\mathcal{N}$ stores up to one state).

Abboud et al. proved [7, Thm. 3.2] that, under the Strong Exponential Time Hypothesis, there is no combinatorial algorithm deciding whether a grammar-compressed text contains a match for a deterministic FSA running in $\mathcal{O}((p{\cdot}s)^{1-ε})$ time with $ε{>}0$. For non deterministic FSA, they proved [7, Thm. 4.2] that, under the $k$-Clique Conjecture, there is no combinatorial algorithm running in $\mathcal{O}((p{\cdot}s^{3})^{1-ε})$ time. Therefore, our algorithm is nearly optimal both for deterministic and non deterministic FSA.

4 Implementation

We implemented Algorithm 1, using the data structures described in the previous section, in a tool named \asciifamilyzearch555https://github.com/pevalme/zearch. This tool works on \asciifamilyrepair666https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/re-pair/repair110811.tar.gz-compressed text and, beyond counting the matching lines, it can also report them by partially decompressing the input file. The implementation consists of less than 2000 lines of C code.

The choice of this particular compressor, which implements the RePair algorithm [4], is due to the little effort required to adapt Algorithm 1 to the specific grammar built by \asciifamilyrepair and the compression it achieves (see Table 1). However \asciifamilyzearch can handle any grammar-based scheme by providing a way to recover the SLP from the input file. Recall that we assume the alphabet symbols, variables and states are indexed. For \asciifamilyrepair-compressed text, the indexes of the alphabet symbols are $0,1,…,255$ ($Σ$ is fixed777Our algorithm also applies to larger alphabets, such as UTF8, without altering its complexity) and the indexes of the variables are $256…p{+}256$. Grammar-based compressors encode the grammar so that rule $X{\textrightarrow}αβ$ appears always after the rules with $α$ and $β$ on the left hand side. Thus, each iteration of the loop in line 14 reads a subsequent rule from the compressed input file.

We translate the input regular expression into an $ε$-free FSA using the automata library \asciifamilylibfa888http://augeas.net/libfa/index.html which applies Thompson’s algorithm [9] with on-the-fly $ε$-removal.

5 Empirical Evaluation

Next we present a summary of the experiments carried out to assess the performance of \asciifamilyzearch. The details of the experiments, including the runtime and number of matching lines reported for each expression on each file and considering more tools, file sizes and regular expressions are available on-line999https://pevalme.github.io/zearch/graphs/index.html.

All tools for regular expression searching considered in this benchmark are used to count the matching lines without reporting them. To simplify the terminology, we refer to counting the matching lines as searching, unless otherwise stated.

\SubSection

Tools Our benchmark compares the performance of \asciifamilyzearch against the fastest implementations we found for

(i) searching the compressed text without decompression,

(ii) searching the uncompressed text,

(iii) decompressing the text without searching and (iv) searching the uncompressed text as it is recovered by the decompressor.

For searching the compressed text we consider \asciifamilyGNgrep, the tool developed by Navarro [10] for searching \asciifamilyLZW-compressed text. To the best of our knowledge, this is the only existing tool departing from the decompress and search approach.

For searching uncompressed text we consider \asciifamilygrep and \asciifamilyhyperscan. We improve the performance of \asciifamilygrep by compiling it without perl regular expression compatibility, which is not supported by \asciifamilyzearch. We used the library \asciifamilyhyperscan by means of the tool (provided with the library) \asciifamilysimplegrep, which we modified101010https://gist.github.com/pevalme/f94bedc9ff08373a0301b8c795063093 to efficiently read data either from stdin or an input file.

For (de)compressing the files we use \asciifamilyzstd and \asciifamilylz4 which are among the best lossless compressors111111https://quixdb.github.io/squash-benchmark/, being \asciifamilylz4 considerably faster while \asciifamilyzstd achieves better compression. We use both tools with the highest compression level, which has little impact on the time required for decompression.

We use versions \asciifamilygrep v3.3, \asciifamilyhyperscan v5.0.0, \asciifamilylz4 v1.8.3 and \asciifamilyzstd v1.3.6 running in an Intel Xeon E5640 CPU 2.67 GHz with 20 GB RAM which supports SIMD instructions up to SSE4-2. We restrict to ASCII inputs and set LC_ALL=C for all experiments, which significantly improves the performance of \asciifamilygrep. Since both \asciifamilyhyperscan and \asciifamilyGNgrep count positions of the text where a match ends, we extend each regular expression (when used with these tools) to match the whole line. We made this decision to ensure all tools solve the same counting problem and therefore produce the same output.

\SubSection

Files and Regular Expressions Our benchmark consists of an automatically generated Log121212http://ita.ee.lbl.gov/html/contrib/NASA-HTTP.html of HTTP requests, English Subtitles [11], and a concatenation of English Books131313https://web.eecs.umich.edu/~lahiri/gutenberg_dataset.html. Table 1 shows how each compressor behaves on these files.

We first run each experiment 3 times as warm up so that the files are loaded in memory. Then we measure the running time 30 times and compute the confidence interval (with 95% confidence) for the running time required to count the number of matching lines for a regular expression in a certain file using a certain tool. We consider the point estimate of the confidence interval and omit the margin of error which never exceeds the $9\scalebox{0.8}{\%}$ of the point estimate for the reported experiments. Figure 2 summarizes the obtained results when considering, for all files, the regular expressions: “what”, “HTTP”, “.”, “I .* you ”, “ [a-z]{4} ”, “ [a-z]*[a-z]{3} ”, “[0-9]{4}”, “[0-9]{2}/(Jun|Jul|Aug)/[0-9]{4}”. For clarity, we report only on the most relevant tools among the ones considered. For \asciifamilylz4 and \asciifamilyzstd, we report the time required to decompress the file and send the output to \asciifamily/dev/null.

\SubSection

Analysis of the Results. Figure 2 and Table 1 show that the performance of \asciifamilyzearch improves with the compression ratio. This is to be expected since \asciifamilyzearch processes each grammar rule exactly once and better compression results in less rules to be processed. In consequence, \asciifamilyzearch is the fastest tool for counting matching lines in compressed Log files while it is the second slowest one for the Books.

In particular, \asciifamilyzearch is more than $25\scalebox{0.8}{\%}$ faster than any other tool working on compressed Log files. Actually \asciifamilyzearch is competitive with \asciifamilygrep and \asciifamilyhyperscan, even though these tools operate on the uncompressed text. These results are remarkable since \asciifamilyhyperscan, unlike \asciifamilyzearch, uses algorithms specifically designed to take advantage of SIMD parallelization.141414According to the documentation, \asciifamilyhyperscan requires, at least, support for SSSE3.

Finally, the fastest tool for counting matching lines in compressed Subtitles and Books, \asciifamilylz4—hyperscan, applies to files larger than the ones obtained when compressing the data with \asciifamilyrepair (see Table 1). However, when considering a better compressor such as \asciifamilyzstd, which achieves slightly more compression than \asciifamilyrepair, the decompression becomes slower. As a result, \asciifamilyzearch outperforms \asciifamilyzstd—hyperscan by more than $7\scalebox{0.8}{\%}$ for Subtitles files and $50\scalebox{0.8}{\%}$ for Logs.

6 Fine-Grained Complexity

The grammars produced by \asciifamilyrepair break the definition of SLP in behalf of the compression by allowing the axiom rule to have more than two symbols on the right hand side. This is due to the fact that the axiom rule is built with the remains of the input text after creating all grammar rules. Typically ${|{σ}|}\geq{|{\mathcal{R}}|}$ so the way in which the axiom is processed heavily influences the performance of \asciifamilyzearch.

Furthermore, our experiments show that the performance of \asciifamilyzearch is typically far from its worst case complexity. This is because the worst case scenario assumes each string generated by a grammar variable labels a path between each pair of states of the automaton. However, we only observed such behavior in contrived examples.

\SubSection

Processing the Axiom Rule.

Algorithm 1 could process the axiom rule $X_{{|{V}|}}{\textrightarrow}σ$ by building an SLP with rules $\{S_{1}{\textrightarrow}(σ)_{1}(σ)_{2}\}\cup\{S_{i}{\textrightarrow}S_{i{-}1}(σ)_{i{+}1}\mid i=2\ldots{|{σ}|}{-}2\}\cup\{X_{{|{V}|}}{\textrightarrow}S_{{|{σ}|}{-}2}(σ)_{\dagger}\}$. However it is more efficient to compute the set of states reachable from the initial ones when reading the string generated with $S_{1}$ and update this set for each symbol $(σ)_{i}$. To perform the counting note that ${\mathcal{C}_{S_{i}}}$ is only used to compute ${\mathcal{C}_{S_{i+1}}}$ and can be discarded afterwards. This yields an algorithm running in $\mathcal{O}\left(p{\cdot}s^{3}{+}{|{σ}|}{\cdot}s^{2}\right)$ time using $\mathcal{O}\left(p{\cdot}s^{2}\right)$ space where $p$ is the number of rules of the input grammar and $X_{{|{V}|}}{\textrightarrow}σ$ its axiom.

\SubSection

Complexity in Terms of Operations Performed by the Algorithm Define $s_{τ,q}={|{\{q^{\prime}\mid(q,τ,q^{\prime})\in δ\}}|}$ and $s_{τ}=\sum_{q\in Q}s_{τ,q}$. Let us recall the complexity of Algorithm 1 according to the described data structures. The algorithm iterates over the $p$ rules of the grammar and, for each of them,

1. initializes matrix $\mathcal{N}$ with $s_{β_{\ell}}$ elements151515We need to set up to $s$ sentinel values for the rows in $\mathcal{N}$ not used for storing $s_{β_{\ell}}$ and

2. iterates through $\mathcal{N}[q^{\prime}][0…s_{β_{\ell},q^{\prime}}]$ for each pair $(q_{1},q^{\prime})\in{\text{{\asciifamily E}}_{α_{\ell}}}$.

Then it processes the axiom rule iterating, for each symbol $(σ_{i})$, through $s_{(σ)_{i}}$ transitions. These are all the operations performed by the algorithm with running time dependent on the size of the input. Hence, Algorithm 1 runs in $\mathcal{O}\left(\sum_{\ell=1}^{{|{V}|}}\tilde{s}_{\ell}+\sum_{i=1}^{|{σ}|}s_{(σ)_{i}}\right)$ time where $\tilde{s}_{\ell}=s_{β_{\ell}}+s+\sum_{(q_{1},q^{\prime})\in{\text{{\asciifamily E}}_{α_{\ell}}}}\left(1{+}s_{β_{\ell},q^{\prime}}\right)$. Note that $\tilde{s}_{\ell}\leq s^{3}$ and $s_{(σ)_{i}}\leq s^{2}$.

In the experiments we observed that $\tilde{s}_{\ell}$ and $s_{(σ_{i})}$ are usually much smaller than $s^{3}$ and $s^{2}$, respectively, as reported in Table 2. Indeed, \asciifamilyzearch exhibits almost linear behavior with respect to the size of the FSA built from the expression. Nevertheless, there are regular expressions that trigger the worst case behavior (last row in Table 2), which cannot be avoided due to the result of Abboud et al. [7] describe before.

7 Related Work

Regular expression matching on compressed text is a well-studied problem consisting on deciding whether a string generated by an SLP matches a regular expression (represented as an automaton). Plandowsky et al. [12] reduced the problem to a series of matrix multiplications, showing it can be solved in $\mathcal{O}(ps^{3})$ time ($\mathcal{O}(ps)$ for deterministic automata) where $p$ is the size of the grammar and $s$ is the size of the automaton. Independently, Esparza et al. [8] defined an algorithm to solve a number of decision problems involving automata and context-free grammars which, when restricted to SLPs, results in a particular implementation of Plandowsky’s approach. Removing the function count from Algorithm 1 and replacing the return statement of the algorithm by $\left((I\times\{X_{|{V}|}\}\times F)\cap δ:\text{\emph{true}}\ \mathbin{?}\ \text{\emph{false}}\right)$ we are left with an efficient implementation of Plandowsky’s approach.

The first algorithm searching grammar-compressed text for regular expression matches is due to Navarro [10], \asciifamilyGNgrep, and it is defined for LZ78/LZW compressed text. His algorithm reports all $k$ positions in the uncompressed text at which a substring matching the expression ends (but not the matches) in $\mathcal{O}(2^{s}{+}s{\cdot}p{+}k{\cdot}s{\cdot}\log{s})$ time using $\mathcal{O}(2^{s}{+}p{\cdot}s)$ space. The algorithm computes the number of matches by enumerating these positions. To the best of our knowledge this is the only algorithm searching compressed text for regular expression matches that has been implemented and evaluated in practice. Bille et al. [13] improved the result of Navarro by defining a data structure of size $o(p)$ to represent LZ78 compressed texts that offers a time-space trade off (reflected in parameter $τ$) for finding all occurrences of a regular expression in a LZ78 compressed text. Their algorithm operates in $\mathcal{O}(p{\cdot}s{\cdot}(s{+}\tau){+}k{\cdot}s{\cdot}\log{s})$ time using $\mathcal{O}(p{\cdot}s^{2}/\tau{+}p{\cdot}s)$ space, where $1\leq τ\leq p$. Note that these approaches to search compressed text exhibit running time linear in the size of the uncompressed text which might be exponentially larger than its compressed version. In consequence, they are not optimal nor competitive with the state of the art, as show by Figure 2.

8 Conclusions and Future Work

We have presented the first algorithm for counting the number of lines in a grammar-compressed text containing a match for a regular expression. The algorithm applies to any grammar-based compression scheme and is nearly optimal for regular expression matching on compressed text. Furthermore, we described the data structures required to achieve nearly optimal complexity and used them to implement a (sequential) tool that significantly outperforms the (parallel) state of the art to solve this problem. Indeed, when the grammar-based compressor achieves hight compression ratio, which is the case for automatically generated Log files, \asciifamilyzearch uses up to $25\scalebox{0.8}{\%}$ less time than \asciifamilylz4—hyperscan, even outperforming \asciifamilygrep and being competitive with \asciifamilyhyperscan.

Finally, Algorithm 1 allows for a conceptually simple parallelization since any set of rules such that no variable appearing on the left hand side of a rule appears on the right hand side of another, can be processed simultaneously. Indeed, a theoretical result by Ullman et al. [14] on the parallelization of Datalog queries can be used to show that counting the number of lines in a grammar-compressed text containing a match for a regular expression is in $\mathcal{NC}^{2}$ when the automaton built from the expression is acyclic. We are optimistic about the possibilities of a parallel version of \asciifamilyzearch.

9 References