Constructing Antidictionaries in Output-Sensitive Space

TL;DR

This paper introduces an output-sensitive method for constructing antidictionaries of multiple words efficiently in space, especially when the set of minimal absent words remains small relative to the total input size.

Contribution

It presents a novel incremental algorithm that computes minimal absent words in output-sensitive space, reducing memory usage for large datasets like genomes.

Findings

Algorithm operates in O(kn + sum of minimal absent words) time

Uses space proportional to input and output sizes, suitable for large data

Experimental results confirm theoretical efficiency and practicality

Abstract

A word $x$ that is absent from a word $y$ is called minimal if all its proper factors occur in $y$. Given a collection of $k$ words $y_1,y_2,\ldots,y_k$ over an alphabet $\Sigma$, we are asked to compute the set $\mathrm{M}^{\ell}_{y_{1}\#\ldots\#y_{k}}$ of minimal absent words of length at most $\ell$ of word $y=y_1\#y_2\#\ldots\#y_k$, $\#\notin\Sigma$. In data compression, this corresponds to computing the antidictionary of $k$ documents. In bioinformatics, it corresponds to computing words that are absent from a genome of $k$ chromosomes. This computation generally requires $\Omega(n)$ space for $n=|y|$ using any of the plenty available $\mathcal{O}(n)$-time algorithms. This is because an $\Omega(n)$-sized text index is constructed over $y$ which can be impractical for large $n$. We do the identical computation incrementally using output-sensitive space. This goal is reasonable when $||\mathrm{M}^{\ell}_{y_{1}\#\ldots\#y_{N}}||=o(n)$, for all $N\in[1,k]$. For instance, in the human genome, $n \approx 3\times 10^9$ but $||\mathrm{M}^{12}_{y_{1}\#\ldots\#y_{k}}|| \approx 10^6$. We consider a constant-sized alphabet for stating our results. We show that all $\mathrm{M}^{\ell}_{y_{1}},\ldots,\mathrm{M}^{\ell}_{y_{1}\#\ldots\#y_{k}}$ can be computed in $\mathcal{O}(kn+\sum^{k}_{N=1}||\mathrm{M}^{\ell}_{y_{1}\#\ldots\#y_{N}}||)$ total time using $\mathcal{O}(\mathrm{MaxIn}+\mathrm{MaxOut})$ space, where $\mathrm{MaxIn}$ is the length of the longest word in $\{y_1,\ldots,y_{k}\}$ and $\mathrm{MaxOut}=\max\{||\mathrm{M}^{\ell}_{y_{1}\#\ldots\#y_{N}}||:N\in[1,k]\}$. Proof-of-concept experimental results are also provided confirming our theoretical findings and justifying our contribution.

Figures

Equations

$$M_y=$$

$$\displaystyle\cup\{c\in\Sigma\mid\mbox{$c$ does not occur in $y$}\}.$$

$$\sum _{N=1}^k\sum _{i=1}^{N-1}(|y_N|+|y_i|)\le \sum _{N=1}^k\sum _{i=1}^k(|y_N|+|y_i|)=\sum _{N=1}^k\sum _{i=1}^k|y_N|+\sum _{N=1}^k\sum _{i=1}^k|y_i|=2k(|y_1|+\cdots +|y_k|).$$

Full text

1 Introduction

The word $x$ is an absent word of the word $y$ if it does not occur in $y$. The absent word $x$ of $y$ is called minimal if and only if all its proper factors occur in $y$. The set of all minimal absent words for a word $y$ is denoted by $\operatorname{\mathrm{M}}_{y}$. The set of all minimal absent words of length at most $\ell$ of a word $y$ is denoted by $\operatorname{\mathrm{M}}^{\ell}_{y}$. For example, if $y=\texttt{abaab}$, then $\operatorname{\mathrm{M}}_{y}=\{\texttt{aaa},\texttt{aaba},\texttt{bab},\texttt{bb}\}$ and $\operatorname{\mathrm{M}}^{3}_{y}=\{\texttt{aaa},\texttt{bab},\texttt{bb}\}$. The upper bound on the number of minimal absent words is $\mathcal{O}(\sigma n)$ [10], where $\sigma$ is the size of the alphabet and $n$ is the length of $y$, and this is tight for integer alphabets [6]; in fact, for large alphabets, such as when $\sigma\geq\sqrt{n}$, this bound is also tight even for minimal absent words having the same length [1].

State-of-the-art algorithms compute all minimal absent words of $y$ in $\mathcal{O}(\sigma n)$ time [10, 2] or in $\mathcal{O}(n+|\operatorname{\mathrm{M}}_{y}|)$ time [16, 7] for integer alphabets. There also exist space-efficient data structures based on the Burrows-Wheeler transform of $y$ that can be applied for this computation [5, 4]. In many real-world applications of minimal absent words, such as in data compression [11, 12, 14, 22], in sequence comparison [6, 7], in on-line pattern matching [9], or in identifying pathogen-specific signatures [23], only a subset of minimal absent words may be considered, and, in particular, the minimal absent words of length (at most) $\ell$. Since, in the worst case, the number of minimal absent words of $y$ is $\Theta(\sigma n)$, $\Omega(\sigma n)$ space is required to represent them explicitly. In [7], the authors presented an $\mathcal{O}(n)$-sized data structure for outputting minimal absent words of a specific length in optimal time for integer alphabets.

The problem with existing algorithms for computing minimal absent words is that they make use of $\Omega(n)$ space; and the same amount is required even if one is merely interested in the minimal absent words of length at most $\ell$. This is because all of these algorithms construct global data structures, such as the suffix array [2]. In theory, this problem can be addressed by using the external memory algorithm for computing minimal absent words presented in [19]. The I/O-optimal version of this algorithm, however, requires a lot of external memory to build the global data structures for the input [20]. One could also use the algorithm of [15] that computes $\operatorname{\mathrm{M}}^{\ell}_{y}$ in $\mathcal{O}(n+|\operatorname{\mathrm{M}}^{\ell}_{y}|)$ time using $\mathcal{O}(\min\{n,\ell z\})$ space, where $z$ is the size of the LZ77 factorisation of $y$. This algorithm also requires constructing the truncated DAWG, a type of global data structure which could take space $\Omega(n)$. Thus, in this paper, we investigate whether $\operatorname{\mathrm{M}}^{\ell}_{y}$ can be computed efficiently in output-sensitive space. As $y$ can be “decomposed” into a collection of $k$ words—with a suitable overlap of length $\ell$ so as not to lose information—we consider the following, general, computational problem.

Problem

Given $k$ words $y_{1},y_{2},\ldots,y_{k}$ over an alphabet $\Sigma$ and an integer $\ell>0$, compute the set $\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{k}}$ of minimal absent words of length at most $\ell$ of $y=y_{1}\#y_{2}\#\ldots\#y_{k}$, $\#\notin\Sigma$.

In data compression, this scenario corresponds to computing the antidictionary of $k$ documents [11, 12]. In bioinformatics, it corresponds to computing words that are absent from a genome of $k$ chromosomes. As discussed above, this computation generally requires $\Omega(n)$ space for $n=|y|$. We do the identical computation incrementally using output-sensitive space. This goal is reasonable when $||\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N}}||=o(n)$, for all $N\in[1,k]$. In the human genome, $n\approx 3\times 10^{9}$ but $||\operatorname{\mathrm{M}}^{12}_{y_{1}\#\ldots\#y_{k}}||\approx 10^{6}$, where $k$ is the total number of chromosomes.

Our Results

Antidictionary-based compressors work on $\Sigma=\{\texttt{0},\texttt{1}\}$ and in bioinformatics we have $\Sigma=\{\texttt{A},\texttt{C},\texttt{G},\texttt{T}\}$; we thus consider a constant-sized alphabet for stating our results. We show that all $\operatorname{\mathrm{M}}^{\ell}_{y_{1}},\ldots,\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{k}}$ can be computed in $\mathcal{O}(kn+\sum^{k}_{N=1}||\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N}}||)$ total time using $\mathcal{O}(\textsc{MaxIn}+\textsc{MaxOut})$ space, where MaxIn is the length of the longest word in $\{y_{1},\ldots,y_{k}\}$ and $\textsc{MaxOut}=\max\{||\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N}}||:N\in[1,k]\}$. Proof-of-concept experimental results are provided confirming our theoretical findings and justifying our contribution.

2 Preliminaries

We generally follow [8]. An alphabet $\Sigma$ is a finite ordered non-empty set of elements called letters. A word is a sequence of elements of $\Sigma$. The set of all words over $\Sigma$ of length at most $\ell$ is denoted by $\Sigma^{\leq\ell}$. We fix a constant-sized alphabet $\Sigma$, i.e., $|\Sigma|=\mathcal{O}(1)$. Given a word $y=uxv$ over $\Sigma$, we say that $u$ is a prefix of $y$, $x$ is a factor (or subword) of $y$, and $v$ is a suffix of $y$. We also say that $y$ is a superword of $x$. A factor $x$ of $y$ is called proper if $x\neq y$.

Given a word $y$ over $\Sigma$, the set of minimal absent words (MAWs) of $y$ is defined as

[TABLE]

For instance, over $\Sigma=\{\texttt{a,b,c}\}$, for $y=\texttt{ab}$ we have $\operatorname{\mathrm{M}}_{y}=\{\texttt{aa,bb,ba,c}\}$. MAWs of length 1 for $y$ can be found in $\mathcal{O}(|y|+|\Sigma|)=\mathcal{O}(|y|)$ time using $\mathcal{O}(|\Sigma|)=\mathcal{O}(1)$ working space, and so, in what follows, we focus on the computation of MAWs of length at least 2.

The suffix tree $\mathcal{T}(y)$ of a non-empty word $y$ of length $n$ is the compact trie representing all suffixes of $y$ [8]. The branching nodes of the trie as well as the terminal nodes, that correspond to non-empty suffixes of $y$, become explicit nodes of the suffix tree, while the other nodes are implicit. We let $\mathcal{L}(v)$ denote the path-label from the root node to node $v$. We say that node $v$ is path-labeled $\mathcal{L}(v)$; i.e., the concatenation of the edge labels along the path from the root node to $v$. Additionally, $\mathcal{D}(v)=|\mathcal{L}(v)|$ is used to denote the word-depth of node $v$. A node $v$ such that the path-label $\mathcal{L}(v)=y[i\mathinner{.\,.}n-1]$, for some $0\leq i\leq n-1$, is terminal and is also labeled with index $i$. Each factor of $y$ is uniquely represented by either an explicit or an implicit node of $\mathcal{T}(y)$ called its locus. The suffix-link of a node $v$ with path-label $\mathcal{L}(v)=aw$ is a pointer to the node path-labeled $w$, where $a\in\Sigma$ is a single letter and $w$ is a word. The suffix-link of $v$ exists by construction if $v$ is a non-root branching node of $\mathcal{T}(y)$. The matching statistics of a word $x[0\mathinner{.\,.}|x|-1]$ with respect to word $y$ is an array $\text{MS}_{x}[0\mathinner{.\,.}|x|-1]$, where $\text{MS}_{x}[i]$ is a pair $(f_{i},p_{i})$ such that (i) $x[i\mathinner{.\,.}i+f_{i}-1]$ is the longest prefix of $x[i\mathinner{.\,.}|x|-1]$ that is a factor of $y$; and (ii) $y[p_{i}\mathinner{.\,.}p_{i}+f_{i}-1]=x[i\mathinner{.\,.}i+f_{i}-1]$ [18]. $\mathcal{T}(y)$ is constructible in time $\mathcal{O}(n)$, and, given $\mathcal{T}(y)$, we can compute $\text{MS}_{x}$ in time $\mathcal{O}(|x|)$ [18].

3 Combinatorial Properties

For convenience, we consider the following setting. Let $y_{1},y_{2}$ be words over the alphabet $\Sigma$ and let $y_{3}=y_{1}\#y_{2}$, with $\#\notin\Sigma$. Let $\ell$ be a positive integer and set $\operatorname{\mathrm{M}}^{\ell}_{y_{1}}=\operatorname{\mathrm{M}}_{y_{1}}\cap\Sigma^{\leq\ell}$ and $\operatorname{\mathrm{M}}^{\ell}_{y_{2}}=\operatorname{\mathrm{M}}_{y_{2}}\cap\Sigma^{\leq\ell}$. We want to construct $\operatorname{\mathrm{M}}^{\ell}_{y_{3}}=\operatorname{\mathrm{M}}_{y_{3}}\cap\Sigma^{\leq\ell}$. Let $x\in\operatorname{\mathrm{M}}^{\ell}_{y_{3}}$. We have two cases:

Case 1

: $x\in\operatorname{\mathrm{M}}^{\ell}_{y_{1}}\cup\operatorname{\mathrm{M}}^{\ell}_{y_{2}}$;

Case 2

: $x\notin\operatorname{\mathrm{M}}^{\ell}_{y_{1}}\cup\operatorname{\mathrm{M}}^{\ell}_{y_{2}}$.

The following auxiliary fact follows directly from the minimality property.

Fact 1**.**

Word $x$ is absent from word $y$ if and only if $x$ is a superword of a MAW of $y$.

For Case 1, we prove the following lemma.

Lemma 1** (Case 1).**

A word $x\in\operatorname{\mathrm{M}}^{\ell}_{y_{1}}$ (resp. $x\in\operatorname{\mathrm{M}}^{\ell}_{y_{2}}$) belongs to $\operatorname{\mathrm{M}}^{\ell}_{y_{3}}$ if and only if $x$ is a superword of a word in $\operatorname{\mathrm{M}}^{\ell}_{y_{2}}$ (resp. in $\operatorname{\mathrm{M}}^{\ell}_{y_{1}}$).

Proof.

Let $x\in\operatorname{\mathrm{M}}^{\ell}_{y_{1}}$ (the case $x\in\operatorname{\mathrm{M}}^{\ell}_{y_{2}}$ is symmetric). Suppose first that $x$ is a superword of a word in $\operatorname{\mathrm{M}}^{\ell}_{y_{2}}$, that is, there exists $v\in\operatorname{\mathrm{M}}^{\ell}_{y_{2}}$ such that $v$ is a factor of $x$. If $v=x$, then $x\in\operatorname{\mathrm{M}}^{\ell}_{y_{1}}\cap\operatorname{\mathrm{M}}^{\ell}_{y_{2}}$ and therefore, using the definition of MAW, $x\in\operatorname{\mathrm{M}}^{\ell}_{y_{3}}$. If $v$ is a proper factor of $x$, then $x$ is an absent word of $y_{2}$ and again, by definition of MAW, $x\in\operatorname{\mathrm{M}}^{\ell}_{y_{3}}$.

Suppose now that $x$ is not a superword of any word in $\operatorname{\mathrm{M}}^{\ell}_{y_{2}}$. Then $x$ is not absent in $y_{2}$ by Fact 1, and hence in $y_{3}$, thus $x$ cannot belong to $\operatorname{\mathrm{M}}^{\ell}_{y_{3}}$. ∎

It should be clear that the statement of Lemma 1 implies, in particular, that all words in $\operatorname{\mathrm{M}}^{\ell}_{y_{1}}\cap\operatorname{\mathrm{M}}^{\ell}_{y_{2}}$ belong to $\operatorname{\mathrm{M}}^{\ell}_{y_{3}}$. Furthermore, Lemma 1 motivates us to introduce the reduced set of MAWs of $y_{1}$ with respect to $y_{2}$ as the set $\operatorname{\mathrm{R}}^{\ell}_{y_{1}}$ obtained from $\operatorname{\mathrm{M}}^{\ell}_{y_{1}}$ after removing those words that are superwords of words in $\operatorname{\mathrm{M}}^{\ell}_{y_{2}}$. The set $\operatorname{\mathrm{R}}^{\ell}_{y_{2}}$ is defined analogously.

Example 1**.**

Let $y_{1}=\texttt{abaab}$, $y_{2}=\texttt{bbaaab}$ and $\ell=5$. We have $\operatorname{\mathrm{M}}^{\ell}_{y_{1}}=\{\texttt{bb,aaa,bab,aaba}\}$ and $\operatorname{\mathrm{M}}^{\ell}_{y_{2}}=\{\texttt{bbb,aaaa,baab,aba,bab,abb}\}.$ The word bab is contained in $\operatorname{\mathrm{M}}^{\ell}_{y_{1}}\cap\operatorname{\mathrm{M}}^{\ell}_{y_{2}}$ so it belongs to $\operatorname{\mathrm{M}}^{\ell}_{y_{3}}$. The word $\texttt{aaba}\in\operatorname{\mathrm{M}}^{\ell}_{y_{1}}$ is a superword of $\texttt{aba}\in\operatorname{\mathrm{M}}^{\ell}_{y_{2}}$ hence $\texttt{aaba}\in\operatorname{\mathrm{M}}^{\ell}_{y_{3}}$. On the other hand, the words bbb, aaaa and abb are superwords of words in $\operatorname{\mathrm{M}}^{\ell}_{y_{1}}$, hence they belong to $\operatorname{\mathrm{M}}^{\ell}_{y_{3}}$. The remaining MAWs are not superwords of MAWs of the other word. The reduced sets are therefore $\operatorname{\mathrm{R}}^{\ell}_{y_{1}}=\{\texttt{bb},\texttt{aaa}\}$ and $\operatorname{\mathrm{R}}^{\ell}_{y_{2}}=\{\texttt{baab},\texttt{aba}\}$. In conclusion, we have for Case 1 that $\operatorname{\mathrm{M}}^{\ell}_{y_{3}}\cap(\operatorname{\mathrm{M}}^{\ell}_{y_{1}}\cup\operatorname{\mathrm{M}}^{\ell}_{y_{2}})=\{\texttt{aaaa,bab,aaba,abb,bbb}\}$.∎

We now investigate the set $\operatorname{\mathrm{M}}^{\ell}_{y_{3}}\setminus(\operatorname{\mathrm{M}}^{\ell}_{y_{1}}\cup\operatorname{\mathrm{M}}^{\ell}_{y_{2}})$ (Case 2).

Fact 2**.**

Let $x=aub$, $a,b\in\Sigma$, be such that $x\in\operatorname{\mathrm{M}}^{\ell}_{y_{3}}$ and $x\notin\operatorname{\mathrm{M}}^{\ell}_{y_{1}}\cup\operatorname{\mathrm{M}}^{\ell}_{y_{2}}$. Then $au$ occurs in $y_{1}$ but not in $y_{2}$ and $ub$ occurs in $y_{2}$ but not in $y_{1}$, or vice versa.

The rationale for generating the reduced sets should become clear with the next lemma.

Lemma 2** (Case 2).**

Let $x\in\operatorname{\mathrm{M}}^{\ell}_{y_{3}}\setminus(\operatorname{\mathrm{M}}^{\ell}_{y_{1}}\cup\operatorname{\mathrm{M}}^{\ell}_{y_{2}})$. Then $x$ has a prefix $x_{i}$ in $\operatorname{\mathrm{R}}^{\ell}_{y_{i}}$ and a suffix $x_{j}$ in $\operatorname{\mathrm{R}}^{\ell}_{y_{j}}$, for $i,j$ such that $\{i,j\}=\{1,2\}$.

Proof.

Let $x=aub$, $a,b\in\Sigma$, be a word in $\operatorname{\mathrm{M}}^{\ell}_{y_{3}}\setminus(\operatorname{\mathrm{M}}^{\ell}_{y_{1}}\cup\operatorname{\mathrm{M}}^{\ell}_{y_{2}})$. By Fact 2, $au$ occurs in $y_{1}$ but not in $y_{2}$ and $ub$ occurs in $y_{2}$ but not in $y_{1}$, or vice versa. Let us assume the first case holds (the other case is symmetric). Since $au$ does not occur in $y_{2}$, there is a MAW $x_{2}\in\operatorname{\mathrm{M}}^{\ell}_{y_{2}}$ that is a factor of $au$. Since $ub$ occurs in $y_{2}$, $x_{2}$ is not a factor of $ub$. Consequently, $x_{2}$ is a prefix of $au$.

Analogously, there is an $x_{1}\in\operatorname{\mathrm{M}}^{\ell}_{y_{1}}$ that is a suffix of $ub$. Furthermore, $x_{1}$ and $x_{2}$ cannot be factors one of another. Inspect Figure 1 in this regard. ∎

Example 2**.**

Let $y_{1}=\texttt{abaab}$, $y_{2}=\texttt{bbaaab}$ and $\ell=5$. Consider $x=\texttt{abaaa}\in\operatorname{\mathrm{M}}^{\ell}_{y_{3}}\setminus(\operatorname{\mathrm{M}}^{\ell}_{y_{1}}\cup\operatorname{\mathrm{M}}^{\ell}_{y_{2}})$ (Case 2 MAW). We have that abaa occurs in $y_{1}$ but not in $y_{2}$ and baaa occurs in $y_{2}$ but not in $y_{1}$. Since abaa does not occur in $y_{2}$, there is a MAW $x_{2}\in\operatorname{\mathrm{R}}^{\ell}_{y_{2}}$ that is a factor of abaa. Since baaa occurs in $y_{2}$, $x_{2}$ is not a factor of baaa. So $x_{2}$ is a prefix of abaa and this is aba. Analogously, there is MAW $x_{1}\in\operatorname{\mathrm{R}}^{\ell}_{y_{1}}$ that is a suffix of abaaa and this is aaa. ∎

As a consequence of Lemma 2, in order to construct the set $\operatorname{\mathrm{M}}^{\ell}_{y_{3}}\setminus(\operatorname{\mathrm{M}}^{\ell}_{y_{1}}\cup\operatorname{\mathrm{M}}^{\ell}_{y_{2}})$, we should consider all pairs $(x_{i},x_{j})$ with $x_{i}$ in $\operatorname{\mathrm{R}}^{\ell}_{y_{i}}$ and $x_{j}$ in $\operatorname{\mathrm{R}}^{\ell}_{y_{j}}$, $\{i,j\}=\{1,2\}$. In order to construct the final set $\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N}}$, we use incrementally Lemmas 1 and 2. We summarise the whole approach in the following general theorem, which forms the theoretical basis of our technique.

Theorem 1**.**

Let $N>1$, and let $x\in\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N}}$. Then, either $x\in\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}\cup\operatorname{\mathrm{M}}^{\ell}_{y_{N}}$ (Case 1 MAWs) or, otherwise, $x\in\operatorname{\mathrm{M}}^{\ell}_{y_{i}\#y_{N}}\setminus(\operatorname{\mathrm{M}}^{\ell}_{y_{i}}\cup\operatorname{\mathrm{M}}^{\ell}_{y_{N}})$ for some $i$. Moreover, in this latter case, $x$ has a prefix in $\operatorname{\mathrm{R}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$ and a suffix in $\operatorname{\mathrm{R}}^{\ell}_{y_{N}}$, or the converse, i.e., $x$ has a prefix in $\operatorname{\mathrm{R}}^{\ell}_{y_{N}}$ and a suffix in $\operatorname{\mathrm{R}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$ (Case 2 MAWs).

Proof.

Let $x\in\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N}}$ and $x\notin\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}\cup\operatorname{\mathrm{M}}^{\ell}_{y_{N}}$. Then, $x\notin\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$ and $x\notin\operatorname{\mathrm{M}}^{\ell}_{y_{N}}$.

Let $x$ be a word of length $m$. By the definition of MAW, $x[0\mathinner{.\,.}m-2]$ and $x[1\mathinner{.\,.}m-1]$ must both be factors of $y_{1}\#\ldots\#y_{N}$. However, both cannot be factors of $y_{1}\#\ldots\#y_{N-1}$ and both cannot be factors of $y_{N}$. Therefore, we have one of the two cases:

Case 1

: $x[0\mathinner{.\,.}m-2]$ is factor of $y_{1}\#\ldots\#y_{N-1}$ but not of $y_{N}$ and $x[1\mathinner{.\,.}m-1]$ is a factor of $y_{N}$ but not of $y_{1}\#\ldots\#y_{N-1}$.

Case 2

: $x[0\mathinner{.\,.}m-2]$ is factor of $y_{N}$ but not of $y_{1}\#\ldots\#y_{N-1}$ and $x[1\mathinner{.\,.}m-1]$ is a factor of $y_{1}\#\ldots\#y_{N-1}$ but not of $y_{N}$.

These two cases are symmetric, thus only proof of Case 1 will be presented here. If $x[0]$ does not occur in $y_{N}$ then $x[0]\in\operatorname{\mathrm{R}}^{\ell}_{y_{N}}$. Otherwise, let $x[0\mathinner{.\,.}t]$ be the longest prefix of $x[0\mathinner{.\,.}m-2]$ that is a factor of $y_{N}$.

Because $0\leq t<m-1$ then $x[1\mathinner{.\,.}t+1]$ is a factor of $y_{N}$. Therefore, $x[0\mathinner{.\,.}t+1]\in\operatorname{\mathrm{M}}^{\ell}_{y_{N}}$. In addition, all factors of $x[0\mathinner{.\,.}t+1]$ occur in $y_{1}\#\ldots\#y_{N-1}$, so $x[0\mathinner{.\,.}t+1]\in\operatorname{\mathrm{R}}^{\ell}_{y_{N}}$.

Now, $x[1\mathinner{.\,.}m-1]$ does not occur in $y_{1}\#\ldots\#y_{N-1}$, so either $x[m-1]$ does not occur in $y_{1}\#\ldots\#y_{N-1}$ which means that $x[m-1]\in\operatorname{\mathrm{R}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$, or let $x[p\mathinner{.\,.}m-1]$ be the longest suffix of $x[1\mathinner{.\,.}m-1]$ that occurs in $y_{1}\#\ldots\#y_{N-1}$.

Because $0<p\leq m-1$ then $x[p-1\mathinner{.\,.}m-2]$ occurs in $y_{1}\#\ldots\#y_{N-1}$, therefore $x[p-1\mathinner{.\,.}m-1]\in\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$. Since all factors of $x[p-1\mathinner{.\,.}m-1]$ occur in $y_{N}$, we have $x[p-1\mathinner{.\,.}m-1]\in\operatorname{\mathrm{R}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$. ∎

4 Algorithm

Let us first introduce an algorithmic tool. In the weighted ancestor problem, introduced in [13], we consider a rooted tree $T$ with an integer weight function $\mu$ defined on the nodes. We require that the weight of the root is zero and the weight of any other node is strictly larger than the weight of its parent. A weighted ancestor query, given a node $v$ and an integer value $w\leq\mu(v)$, asks for the highest ancestor $u$ of $v$ such that $\mu(u)\geq w$, i.e., such an ancestor $u$ that $\mu(u)\geq w$ and $\mu(u)$ is the smallest possible. When $T$ is the suffix tree of a word $y$ of length $n$, we can locate the locus of any factor $y[i\mathinner{.\,.}j]$ using a weighted ancestor query. We define the weight of a node of the suffix tree as the length of the word it represents. Thus a weighted ancestor query can be used for the terminal node decorated with $i$ to create (if necessary) and mark the node that corresponds to $y[i\mathinner{.\,.}j]$.

Theorem 2** ([3]).**

Given a collection $Q$ of weighted ancestor queries on a weighted tree $T$ on $n$ nodes with integer weights up to $n^{\mathcal{O}(1)}$, all the queries in $Q$ can be answered off-line in $\mathcal{O}(n+|Q|)$ time.

4.1 The Algorithm

At the $N$th step, we have in memory the set $\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$. Our algorithm works as follows:

We read word $y_{N}$ from the disk and compute $\operatorname{\mathrm{M}}^{\ell}_{y_{N}}$ in time $\mathcal{O}(|y_{N}|)$. We output the words in the following constant-space form: $<i_{1},i_{2},\alpha>$ per word [2]; such that $y_{N}[i_{1}\mathinner{.\,.}i_{2}]\cdot\alpha\in\operatorname{\mathrm{M}}^{\ell}_{y_{N}}$. 2. 2.

Here we compute Case 1 MAWs. We apply Lemma 1 to construct set $M=\{w:w\in\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N}},w\in\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}\cup\operatorname{\mathrm{M}}^{\ell}_{y_{N}}\}$ and the sets $\operatorname{\mathrm{R}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}},\operatorname{\mathrm{R}}^{\ell}_{y_{N}}$ as follows.

1. (a)

We first want to find the elements of $\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$ that are superwords of any word $y_{N}[i_{1}\mathinner{.\,.}i_{2}]\cdot\alpha$. We build the generalised suffix tree $T_{1}=\mathcal{T}(\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}\cup\{y_{N}\})$ [18]. We find the locus of the longest proper prefix $y_{N}[i_{1}\mathinner{.\,.}i_{2}]$ of each element of $\operatorname{\mathrm{M}}^{\ell}_{y_{N}}$ in $T_{1}$ via answering off-line a batch of weighted ancestor queries (Theorem 2). From there on, we spell $\alpha$ and mark the corresponding node on $T_{1}$, if any. After processing all $<i_{1},i_{2},\alpha>$ in the same manner, we traverse $T_{1}$ to gather all occurrences (starting positions) of words $y_{N}[i_{1}\mathinner{.\,.}i_{2}]\cdot\alpha$ in the elements of $\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$, thus finding the elements of $\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$ that are superwords of any $y_{N}[i_{1}\mathinner{.\,.}i_{2}]\cdot\alpha$. By definition, no MAW $y_{N}[i_{1}\mathinner{.\,.}i_{2}]\cdot\alpha$ is a prefix of another MAW $y_{N}[i_{1}^{\prime}\mathinner{.\,.}i_{2}^{\prime}]\cdot\alpha^{\prime}$, thus the marked nodes form pairwise disjoint subtrees, and the whole process takes time $\mathcal{O}(|y_{N}|+||\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}||)$, the size of $T_{1}$. 2. (b)

We next want to check if the words $y_{N}[i_{1}\mathinner{.\,.}i_{2}]\cdot\alpha$ are superwords of any element of $\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$. We first sort all tuples $<i_{1},i_{2},\alpha>$ using radixsort and then check this using the matching statistics algorithm for $y_{N}$ with respect to $\mathcal{T}(\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}})$ considering the tuples in ascending order (from left to right) at the same time. By definition, no element in $\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$ is a factor of another element in the same set. Thus if a factor of $y_{N}[i_{1}\mathinner{.\,.}i_{2}]\cdot\alpha$ corresponds to an element in $\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$ this is easily located in $\mathcal{T}(\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}})$ while running the matching statistics algorithm. The whole process takes $\mathcal{O}(|y_{N}|+||\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}||)$ time: $\mathcal{O}(||\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}||)$ time to construct the suffix tree and a further $\mathcal{O}(|y_{N}|)$ time for the matching statistics algorithm and for processing the $\mathcal{O}(|y_{N}|)$ tuples.

We create set $\operatorname{\mathrm{R}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$ explicitly since it is a subset of $\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$. We create set $\operatorname{\mathrm{R}}^{\ell}_{y_{N}}$ implicitly: every element $x\in\operatorname{\mathrm{R}}^{\ell}_{y_{N}}$ is stored as a tuple $<i_{1},i_{2},\alpha>$ such that $x=y_{N}[i_{1}\mathinner{.\,.}i_{2}]\cdot\alpha$. We store every element of $\{x_{2}:x_{2}\in M\cap\operatorname{\mathrm{M}}^{\ell}_{y_{N}}\}$ with the same representation. All other elements of $M$ are stored explicitly. 3. 3.

Construct the suffix tree of $y_{N}$ and use it to locate all occurrences of words in $\operatorname{\mathrm{R}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$ in $y_{N}$ and store the occurrences as pairs (starting position, ending position). This step can be done in time $\mathcal{O}(|y_{N}|+||\operatorname{\mathrm{R}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}||)$. By definition, no element in $\operatorname{\mathrm{R}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$ is a prefix of another element in $\operatorname{\mathrm{R}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$, and thus this can be done within the claimed time complexity. 4. 4.

For every $i\in[1,N-1]$, we perform the following to compute Case 2 MAWs:

1. (a)

Read word $y_{i}$ from the disk. Construct the suffix tree $T_{x}$ of word $x=y_{i}\#y_{N}$ in time $\mathcal{O}(|y_{i}|+|y_{N}|)$. Use $T_{x}$ to locate all occurrences of elements of $\operatorname{\mathrm{R}}^{\ell}_{y_{N}}$ in $y_{i}$ and store the occurrences as pairs (starting position, ending position). This step can be done in time $\mathcal{O}(|y_{i}|+|y_{N}|)$ similar to step 2. By definition, no element in $\operatorname{\mathrm{R}}^{\ell}_{y_{N}}$ is a prefix of another element in $\operatorname{\mathrm{R}}^{\ell}_{y_{N}}$, and thus this can be done within the claimed time complexity. 2. (b)

During a bottom-up traversal of $T_{x}$ mark, at each explicit node of $T_{x}$, the smallest starting position of the subword represented by that node, and the largest starting position of the same subword. This can be done in time $\mathcal{O}(|y_{i}|+|y_{N}|)$ by propagating upwards the labels of the terminal nodes (starting positions of suffixes) and updating the smallest and largest positions analogously. 3. (c)

Compute the set $\operatorname{\mathrm{M}}^{\ell}_{y_{i}\#y_{N}}$ and output the words in the following constant-space form: $<a,i_{1},i_{2},b>$ per word; such that $a\cdot x[i_{1}\mathinner{.\,.}i_{2}]\cdot b$ is a MAW. This can be done in time $\mathcal{O}(|y_{i}|+|y_{N}|)$. 4. (d)

For each element of $\operatorname{\mathrm{M}}^{\ell}_{y_{i}\#y_{N}}$, we need to locate the node representing word $ax[i_{1}\mathinner{.\,.}i_{2}]=au$ and the node representing word $x[i_{1}\mathinner{.\,.}i_{2}]b=ub$. This can be done in time $\mathcal{O}(|y_{i}|+|y_{N}|)$ via answering off-line a batch of weighted ancestor queries (Theorem 2). At this point, we have located the two nodes on $T_{x}$. We assign a pointer from the stored starting position $g$ of $au$ to the ending position $f$ of $ub$, only if $g$ is before $\#$ and $f$ is after $\#$ ($f$ can be trivially computed using the stored starting position of $ub$ and the length of $ub$). Conversely, we assign a pointer from the ending position $f$ of $ub$ to the stored starting position $g$ of $au$, only if $f$ is before $\#$ and $g$ is after $\#$. 5. (e)

Suppose $au$ occurs in $y_{i}$ and $ub$ in $y_{N}$. We make use of the pointers as follows. Recall steps 3 and 4(a) and check whether $au$ starts where a word $r_{1}$ of $\operatorname{\mathrm{R}}^{\ell}_{y_{N}}$ starts and $ub$ ends where a word $r_{2}$ of $\operatorname{\mathrm{R}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$ ends. If this is the case and $|u|\geq\max\{|r_{1}|,|r_{2}|\}-1$, then by Theorem 1 $aub$ is added to our output set $M$, otherwise discard it. Inspect Figure 2 in this regard. Conversely, if $au$ occurs in $y_{N}$ and $ub$ in $y_{i}$ check whether $au$ starts where a word $r_{2}$ of $\operatorname{\mathrm{R}}^{\ell}_{y_{1}\#\ldots\#y_{N-1}}$ starts and whether $ub$ ends where a word $r_{1}$ of $\operatorname{\mathrm{R}}^{\ell}_{y_{N}}$ ends. If this is the case and $|u|\geq\max\{|r_{1}|,|r_{2}|\}-1$, then $aub$ is added to $M$, otherwise discard it.

Finally, we set $\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N}}=M$ as the output of the $N$th step. Let MaxIn be the length of the longest word in $\{y_{1},\ldots,y_{k}\}$ and $\textsc{MaxOut}=\max\{||\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N}}||:N\in[1,k]\}$.

Theorem 3**.**

Given $k$ words $y_{1},y_{2},\ldots,y_{k}$ and an integer $\ell>0$, all $\operatorname{\mathrm{M}}^{\ell}_{y_{1}},\ldots,\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{k}}$ can be computed in $\mathcal{O}(kn+\sum^{k}_{N=1}||\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N}}||)$ total time using $\mathcal{O}(\textsc{MaxIn}+\textsc{MaxOut})$ space, where $n=|y_{1}\#\ldots\#y_{k}|$.

Proof.

From the above discussion, the time is bounded by $\mathcal{O}(\sum^{k}_{N=1}\sum^{N-1}_{i=1}(|y_{N}|+|y_{i}|)+\sum^{k}_{N=1}||\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N}}||)$. We can bound the first term as follows.

[TABLE]

Therefore the time is bounded by $\mathcal{O}(kn+\sum^{k}_{N=1}||\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N}}||)$.

The space is bounded by the maximum time spent at a single step; namely, the length of the longest word in the collection plus the maximum total size of set elements across all output sets. Note that the total output size of the algorithm is the sum of all its output sets, that is $\sum^{k}_{N=1}||\operatorname{\mathrm{M}}^{\ell}_{y_{1}\#\ldots\#y_{N}}||$, and MaxOut could come from any intermediate set.

The correctness of the algorithm follows from Lemma 1 and Theorem 1. ∎

5 Proof-of-Concept Experiments

In this section, we do not directly compare against the fastest internal [2] or external [19] memory implementations because the former assumes that we have the required amount of internal memory, and the latter assumes that we have the required amount of external memory to construct and store the global data structures for a given input dataset. If the memory for constructing and storing the data structures is available, these linear-time algorithms are surely faster than the method proposed here. In what follows, we rather show that our output-sensitive technique offers a space-time tradeoff, which can be usefully exploited for specific values of $\ell$, the maximal length of MAWs we wish to compute.

The algorithm discussed in Section 4 (with the exception of storing and searching the reduced set words explicitly rather than in the constant-space form previously described) has been implemented in the C++ programming language111The implementation can be made available upon request.. The correctness of our implementation has been confirmed against that of [2]. As input dataset here we used the entire human genome (version hg38) [21], which has an approximate size of 3.1GB. The following experiment was conducted on a machine with an Intel Core i5-4690 CPU at 3.50 GHz and 128GB of memory running GNU/Linux. We ran the program by splitting the genome into $k={2,4,6,8,10}$ blocks and setting $\ell=10,11,12$. Figure 3 depicts the change in elapsed time and peak memory usage as $k$ and $\ell$ increase (space-time tradeoff).

Graph (a) shows an increase of time as $k$ and $\ell$ increase; and graph (b) shows a decrease in memory as $k$ increases (as proved in Theorem 3). Notice that the space to construct the block-wise data structures bounds the total space used for the specific $\ell$ values and that is why the memory peak is essentially the same for the $\ell$ values used. This can specifically be seen for $\ell=10$ where all words of length $10$ are present in the genome. The same dataset was used to run the fastest internal memory implementation for computing MAWs [2] on the same machine. It took only $2242$ seconds to compute all MAWs but with a peak memory usage of $60.80$GB. The results confirm our theoretical findings and justify our contribution.