The Alternating BWT: an algorithmic perspective

TL;DR

This paper provides a comprehensive combinatorial and algorithmic analysis of the Alternating Burrows-Wheeler Transform (ABWT), demonstrating its invertibility, efficient computation, and applicability in compressed full-text indexing, analogous to the classical BWT.

Contribution

It establishes the invertibility and efficient computation of ABWT, extending BWT techniques to an alternating lexicographical order and enabling its use in compressed indexing.

Findings

ABWT is rank-invertible, like BWT.

Efficient algorithms for ABWT computation are developed.

ABWT can be used for compressed full-text indices.

Abstract

The Burrows-Wheeler Transform (BWT) is a word transformation introduced in 1994 for Data Compression. It has become a fundamental tool for designing self-indexing data structures, with important applications in several area in science and engineering. The Alternating Burrows-Wheeler Transform (ABWT) is another transformation recently introduced in [Gessel et al. 2012] and studied in the field of Combinatorics on Words. It is analogous to the BWT, except that it uses an alternating lexicographical order instead of the usual one. Building on results in [Giancarlo et al. 2018], where we have shown that BWT and ABWT are part of a larger class of reversible transformations, here we provide a combinatorial and algorithmic study of the novel transform ABWT. We establish a deep analogy between BWT and ABWT by proving they are the only ones in the above mentioned class to be rank-invertible, a novel notion guaranteeing efficient invertibility. In addition, we show that the backward-search procedure can be efficiently generalized to the ABWT; this result implies that also the ABWT can be used as a basis for efficient compressed full text indices. Finally, we prove that the ABWT can be efficiently computed by using a combination of the Difference Cover suffix sorting algorithm [K\"{a}rkk\"{a}inen et al., 2006] with a linear time algorithm for finding the minimal cyclic rotation of a word with respect to the alternating lexicographical order.

Equations

$$H_0(w)=-\sum _{i=0}^{\sigma -1}\frac{|w{|}_{c_i}}{|w|}\log \frac{|w{|}_{c_i}}{|w|}$$

$$H_k(w)=\frac{1}{|w|}\sum _{x\in {\Sigma }^k}|x_w|H_0(x_w).$$

$$\begin{array}[]{cccccc}&&&&&\\ &&&&&\\ a&b&r&a&c&a\\ b&r&a&c&a&a\\ r&a&c&a&a&b\\ a&c&a&a&b&r\\ c&a&a&b&r&a\\ a&a&b&r&a&c\\ &&&&&\\ \lx@intercol\hfil M(w)\hfil\lx@intercol\\ \end{array}\qquad\qquad\begin{array}[]{ccccccccc}&&F&&&&&L\\ &&\downarrow&&&&&\downarrow\\ &&a&a&b&r&a&c\\ &&a&b&r&a&c&a\\ I&\rightarrow&a&c&a&a&b&r\\ &&b&r&a&c&a&a\\ &&c&a&a&b&r&a\\ &&r&a&c&a&a&b\\ &&&&&&&\\ &&\lx@intercol\hfil M_{{lex}}(w)\hfil\lx@intercol\end{array}\qquad\qquad\begin{array}[]{ccccccccc}&&F&&&&&L\\ &&\downarrow&&&&&\downarrow\\ I&\rightarrow&a&c&a&a&b&r\\ &&a&b&r&a&c&a\\ &&a&a&b&r&a&c\\ &&b&r&a&c&a&a\\ &&c&a&a&b&r&a\\ &&r&a&c&a&a&b\\ &&&&&&&\\ &&\lx@intercol\hfil M_{{alt}}(w)\hfil\lx@intercol\end{array}$$

$$au{⪯}_{alt}av⟹va{⪯}_{alt}ua.$$

$$\rho (bw{t}_K(w))\le 2\rho (w).$$

$$H_r(w)=\frac{1}{|u|}\sum _{i=1}^m|u_i|H_0(u_i).$$

$$w_{n-1-j}=L[L{F}^j(I)]$$

$$LF(i)=\sum _{c\in \Sigma }^{c<L(i)}{P}_L[c]+{\mathrm{rank}}_{L[i]}(L,i-1)$$

$$LF(i)=\sum _{c\in \Sigma }^{c\le L(i)}{P}_L[c]-{\mathrm{rank}}_{L[i]}(L,i-1)-1$$

$$b^{\prime }=\sum _{c\in \Sigma }^{c\le x}{P}_L[c]-{\mathrm{rank}}_x(L,e-1)-1e^{\prime }=\sum _{c\in \Sigma }^{c\le x}{P}_L[c]-{\mathrm{rank}}_x(L,b).$$

$$LF(i)=f_K(P_L,L[i],{\mathrm{rank}}_{L[i]}(L,i)).$$

$$x{<}_{Id}y,z{<}_{Id}w\text{mbox}andx{<}_{\pi }y,z{>}_{\pi }w,$$

$$a{<}_{\pi }b\text{mbox}andb{>}_{\pi }c.$$

$$f_K(⟨2,1,2⟩,a,1)=1,f_K(⟨2,1,2⟩,a,2)=2,$$

$$f_K(⟨2,1,2⟩,a,1)=2,f_K(⟨2,1,2⟩,a,2)=1.$$

$$\begin{array}[]{llllll}&F_{1}&&&&L_{1}\\ &\downarrow&&&&\downarrow\\ s_{1}\rightarrow&a_{1}&a_{2}&b&c&c\\ &a_{2}&b&c&c&a_{1}\\ &b&c&c&a_{1}&a_{2}\\ &c&c&a_{1}&a_{2}&b\\ &c&a_{1}&a_{2}&b&c\end{array}\qquad\qquad\begin{array}[]{llllll}&F_{2}&&&&L_{2}\\ &\downarrow&&&&\downarrow\\ &a_{2}&c&c&a_{1}&b\\ s_{2}\rightarrow&a_{1}&b&a_{2}&c&c\\ &b&a_{2}&c&c&a_{1}\\ &c&c&a_{1}&b&a_{2}\\ &c&a_{1}&b&a_{2}&c\end{array}$$

$$f_K(⟨2,2i+3⟩,a,1)=1,f_K(⟨2,2i+3⟩,a,2)=2.$$

$$f_K(⟨2,2i+3⟩,a,1)=2,f_K(⟨2,2i+3⟩,a,2)=1.$$

$$f_K(⟨2,1,2i+3⟩,a,1)=1,f_K(⟨2,1,2i+3⟩,a,2)=2.$$

$$f_K(⟨2,1,2i+3⟩,a,1)=2,f_K(⟨2,1,2i+3⟩,a,2)=1$$

$$\{(i-j)\mathrm{mod}v\mid i,j,\in D\}=[0,v).$$

$$R[0,6]=\stackrel{w[0,5]}{\mathrm{abaaca}}\stackrel{w[6,11]}{\mathrm{baacab}}\stackrel{w[12,18]}{\$\$\$\$\$\$}\stackrel{w[1,6]}{\mathrm{baacab}}\stackrel{w[7,12]}{aacab\$}\stackrel{w[3,8]}{\mathrm{acabaa}}\stackrel{w[9,14]}{cab\$\$\$}$$

$$\$\$\$\$\$\$\preceq_{lex}aacab\$\preceq_{lex}abaaca\preceq_{lex}acabaa\preceq_{lex}baacab\preceq_{lex}cab\$\$\$$

$

$$\$\$\$\$\$\$\preceq_{alt}acabaa\preceq_{alt}abaaca\preceq_{alt}aacab\$\preceq_{alt}baacab\preceq_{alt}cab\$\$\$$

$

$$w[i,n-1]{\prec }_{alt}w[j,n-1]$$

$$⟺R_{abwt}[t_i+\lfloor \ell /v\rfloor ]<R_{abwt}[t_j+\lfloor \ell /v\rfloor ]$$

$$⟺R_{abwt}[t_i,r]{⪯}_{lex}{R}_{abwt}[t_j,r]$$

$$c\alpha {\prec }_{alt}c\beta ⟺\beta {\prec }_{alt}\alpha .$$

$$\begin{array}[]{cccccc}a&b&a&b&b&a\\ a&b&b&a&a&b\\ a&a&b&a&b&b\\ b&b&a&a&b&a\\ b&a&a&b&a&b\\ b&a&b&b&a&a\\ &&&&&\\ \lx@intercol\hfil M_{{alt}}(ababba)\hfil\lx@intercol\end{array}\qquad\qquad\begin{array}[]{cccccc}a&b&a&b&b&a\\ a&b&b&a&&\\ a&&&&&\\ b&b&a&&&\\ b&a&&&&\\ b&a&b&b&a&\\ &&&&&\\ \lx@intercol\hfil Suf(ababba)\hfil\lx@intercol\end{array}\qquad\qquad\begin{array}[]{cccccc}a&b&a&b&b&\\ a&b&b&&&\\ a&a&b&a&b&b\\ b&b&&&&\\ b&a&b&b&&\\ b&&&&&\\ &&&&&\\ \lx@intercol\hfil Suf(aababb)\hfil\lx@intercol\end{array}$$

$$\begin{array}[]{cccccc}a&n&a&n&a&b\\ a&n&a&b&a&n\\ a&b&a&n&a&n\\ b&a&n&a&n&a\\ n&a&b&a&n&a\\ n&a&n&a&b&a\\ &&&&&\\ \lx@intercol\hfil M_{{alt}}(banana)\hfil\lx@intercol\\ \end{array}\qquad\qquad\begin{array}[]{ccccccc}\$&b&a&n&a&n&a\\ a&n&a&n&a&\$&b\\ a&n&a&\$&b&a&n\\ a&\$&b&a&n&a&n\\ b&a&n&a&n&a&\$\\ n&a&\$&b&a&n&a\\ n&a&n&a&\$&b&a\\ &&&&&\\ \lx@intercol\hfil M_{{alt}}(banana\$)\hfil\lx@intercol\\ \end{array}\qquad\qquad\begin{array}[]{ccccccc}\$&a&n&a&n&a&b\\ a&n&a&n&a&b&\$\\ a&n&a&b&\$&a&n\\ a&b&\$&a&n&a&n\\ b&\$&a&n&a&n&a\\ n&a&b&\$&a&n&a\\ n&a&n&a&b&\$&a\\ &&&&&\\ \lx@intercol\hfil M_{{alt}}(ananab\$)\hfil\lx@intercol\\ \end{array}$$

Full text

1 Introduction

Michael Burrows and David Wheeler introduced in 1994 a reversible word transformation [4], denoted by $BWT$, that turned out to have “myriad virtues”. At the time of its introduction in the field of text compression, the Burrows-Wheeler Transform was perceived as a magic box: when used as a preprocessing step it would bring rather weak compressors to be competitive in terms of compression ratio with the best ones available [12]. In the years that followed, many studies have shown the effectiveness of $BWT$ and its central role in the field of Data Compression due to the fact that it can be seen as a “booster” of the performance of memoryless compressors [14, 22, 36]. Moreover, it was shown in [15] that the $BWT$ can be used to efficiently search for occurrences of patterns inside the original text. Such capabilities of the $BWT$ have originated the field of Compressed Full-text Self-indices [29, 38]. The remarkable properties of the $BWT$ have aroused great interest both from the theoretical and applicative points of view [30, 31, 35, 48, 27, 7, 26, 44, 17, 40].

In the context of Combinatorics on Words, many studies have addressed the characterization of the words that become the most compressible after the application of the $BWT$ [34, 47, 39, 41, 42, 13]. Recent studies have focused on measuring the “clustering effect” of $BWT$, which is a property related to its boosting role as preprocessing of a text compressor [33, 32].

In [8], the authors characterize the $BWT$ as the inverse of a known bijection between words and multisets of primitive necklaces [19]. From this result, in [18] the authors introduce and study the basic properties of the Alternating $BWT$, $ABWT$ from now on. It is a transformation on words analogous to the $BWT$ but the cyclic rotations of the input word are sorted by using the alternating lexicographic order instead of the usual lexicographic order. The alternating lexicographic order is defined for infinite words as follows: the first letters are compared with the given alphabetic order, in case of equality the second letters are compared with the opposite order, and so on alternating the two orders for even/odd positions.

In this paper we show that the $ABWT$ satisfies most of the properties that make the $BWT$ such a useful transformation. Not only the $ABWT$ can be computed and inverted in linear time, but also the backward-search procedure, which is the basis for indexed exact string matching on the $BWT$, can be efficiently generalized to the $ABWT$. This implies that the $ABWT$ can be used to build an efficient compressed full text index for the transformed string, similarly to the $BWT$. Note that the variants of the original $BWT$ which have been introduced so far in the literature [45, 5, 9], were either simple modifications that do not bring new theoretical insight or they were significantly different but without the remarkable compression and search properties of the original $BWT$ (see [21, Section 2.1] for a more detailed discussion of these variants).

The existence of the $ABWT$ shows that the classical lexicographic order is not the only order relation that one can use to obtain a reversible transformation. Indeed, lexicographic and alternating lexicographic order are two particular cases of a more general class of order relations considered in [10, 43]. In a preliminary version of this paper [20] we introduce therefore a class of reversible word transformations based on the above order relations that includes both the original $BWT$ and the Alternating $BWT$. Within this class, we introduce the notion of rank-invertibility, a property that guarantees that the transformation can be efficiently inverted using rank operations, and we prove that $BWT$ and $ABWT$ are the only transformations within this class that are rank-invertible.

We consider also the problem of efficiently computing the $ABWT$. We first show how to generalize to the alternating lexicographic order the Difference Cover technique introduced in [25]. This result leads to the design of time optimal and space efficient algorithms for the construction of the $ABWT$ in different models of computation when the input string ends with a unique end-of-string symbol. Finally, we explore some combinatorial properties of the Galois words, which are minimal cyclic rotations within a conjugacy class, with respect to the alternating lexicographical order. We provide a linear time and space algorithm to find the Galois rotation of a given word and we show that, combining this algorithm with the Difference Cover technique, the $ABWT$ can be computed in linear time even when the input string does not end with a unique end-of-string symbol.

Motivated by the discovering of the $ABWT$, in [21] the authors explore a class of string transformations that includes the one considered in this paper. In this larger class, the cyclic rotations of the input string are sorted using an alphabet ordering that depends on the longest common prefix of the rotations being compared. Somewhat surprisingly some of the transformations in this class do have the same properties of the $BWT$ and $ABWT$, thus showing that our understanding of these transformations is still incomplete.

2 Preliminaries

Let $\Sigma=\{c_{0},c_{1},\ldots,c_{\sigma-1}\}$ be an ordered constant size alphabet with $c_{0}<c_{1}<\ldots<c_{\sigma-1}$, where $<$ denotes the standard lexicographic order. We denote by $\Sigma^{*}$ the set of words over $\Sigma$. Let $w=w_{0}w_{1}\cdots w_{n-1}\in\Sigma^{*}$ be a finite word, we denote by $|w|$ its length $n$. We use $\epsilon$ to denote the empty word. We denote by $|w|_{c}$ the number of occurrences of a letter $c$ in $w$. The Parikh vector $P_{w}$ of a word $w$ is a $\sigma$-length array of integers such that for each $c\in\Sigma$, $P_{w}[c]=|w|_{c}$. Given a word $x$ and $c\in\Sigma$, we write $\mathop{\mathrm{rank}}\nolimits_{c}(x,i)$ to denote the number of occurrences of $c$ in $x[0,i]$.

Given a finite word $w$, a factor of $w$ is written as $w[i,j]=w_{i}\cdots w_{j}$, with $0\leq i\leq j\leq n-1$. A factor of type $w[0,j]$ is called a prefix, while a factor of type $w[i,n-1]$ is called a suffix. The longest proper factor of $w$ that is both prefix and suffix is called border. The $i$-th symbol in $w$ is denoted by $w[i]$. Two words $x,y\in\Sigma^{*}$ are conjugate, if $x=uv$ and $y=vu$, where $u,v\in\Sigma^{*}$. We also say that $x$ is a cyclic rotation of $y$. A word $x$ is primitive if all its cyclic rotations are distinct. A primitive word is a Lyndon word if it is smaller than all of its conjugates. Conjugacy between words is an equivalence relation over $\Sigma^{*}$. A word $z$ is called a circular factor of $x$ if it is a factor of some conjugate of $x$.

Given two words of the same length $x=x_{0}x_{1}\ldots x_{s-1}$ and $y=y_{0}y_{1}\ldots y_{s-1}$, we write $x\preceq_{lex}y$ if and only if $x=y$ or $x_{i}<y_{i}$, where $i$ is the smallest index in which the corresponding characters of the two words differ. Analogously, and with the same notation as before, we write $x\preceq_{alt}y$ if and only if $x=y$ or (a) $i$ is even and $x_{i}<y_{i}$ or (b) $i$ is odd and $x_{i}>y_{i}$. Notice that $\preceq_{lex}$ is the standard lexicographic order relation on words while $\preceq_{alt}$ is the alternating lexicographic order relation. Such orders are used in Section 3 to define two different transformations on words.

The run-length encoding of a word $w$, denoted by $\mathop{\mathrm{rle}}\nolimits(w)$, is a sequence of pairs $(w_{i},l_{i}$) such that $w_{i}w_{i+1}\cdots w_{i+l_{i}-1}$ is a maximal run of a letter $w_{i}$ (i.e., $w_{i}=w_{i+1}=\cdots=w_{i+l_{i}-1}$, $w_{i-1}\neq w_{i}$ and $w_{i+l_{i}}\neq w_{i}$), and all such maximal runs are listed in $\mathop{\mathrm{rle}}\nolimits(w)$ in the order they appear in $w$. We denote by $\rho(w)=|\mathop{\mathrm{rle}}\nolimits(w)|$ i.e., is the number of pairs in $w$, or equivalently the number of equal-letter runs in $w$. Moreover we denote by $\rho(w)_{c_{i}}$ the number of pairs $(w_{j},l_{j})$ in $\mathop{\mathrm{rle}}\nolimits(w)$ where $w_{j}=c_{i}$. Notice that $\rho(w)\leq\rho(w_{1})+\rho(w_{2})+\cdots+\rho(w_{p})$, where $w_{1}w_{2}\cdots w_{p}=w$ is any partition of $w$.

The zero-th order empirical entropy of the word $w$ is defined as

[TABLE]

(all logarithms are taken to the base $2$ and we assume $0\log 0=0$). The value $|w|H_{0}(w)$ is the output size of an ideal compressor that uses $-\log({|w|_{c_{i}}}/{|w|})$ bits to encode each occurrence of symbol $c_{i}$. This is the minimum size we can achieve using a uniquely decodable code in which a fixed codeword is assigned to each symbol.

For any length-$k$ factor $x$ of $w$, we denote by $x_{w}$ the sequence of characters preceding the occurrences of $x$ in $w$, taken from left to right. If $x$ is not a factor of $w$ the word $x_{w}$ is empty. The $k$-th order empirical entropy of $w$ is defined as

[TABLE]

The value $|w|H_{k}(w)$ is a lower bound to the output size of any compressor that encodes each symbol with a code that only depends on the symbol itself and on the $k$ preceding symbols. Since the use of a longer context helps compression, it is not surprising that for any $k\geq 0$ it is $H_{k+1}(w)\leq H_{k}(w)$.

3 BWT and Alternating BWT

In this section we describe two different invertible transformations on words based on the lexicographic and alternating lexicographic order, respectively. Given a primitive word $w$ of length $n$ in $\Sigma^{*}$, the Burrows-Wheeler transform, denoted by $BWT$ [4] and the Alternating Burrows-Wheeler transform, denoted by $ABWT$ [18] for $w$ are defined constructively as follows:

Create the matrix $M(w)$ of the cyclic rotations of $w$; 2. 2.

Create the matrix

1. (a)

for $BWT$, $M_{{lex}}(w)$ by sorting the rows of $M(w)$ according to $\preceq_{lex}$; 2. (b)

for $ABWT$, $M_{{alt}}(w)$ by sorting the rows of $M(w)$ according to $\preceq_{alt}$; 3. 3.

Return as output the pair

1. (a)

for $BWT$, $(bwt(w),I)$, where $bwt(w)$ is the last column $L$ in the matrix $M_{{lex}}(w)$ 2. (b)

for $ABWT$, $(abwt(w),I)$ where $abwt(w)$ is the last column $L$ in the matrix $M_{{alt}}(w)$

and, in both case, the integer $I$ giving the position of $w$ in that matrix.

An example of the above process, together with the corresponding output, is provided in Fig. 1.

Remark 3.1

If two words are conjugate the $BWT$ (resp. $ABWT$) will have the same column $L$ and differ only in $I$, whose purpose is only to distinguish between the different members of the conjugacy class. However, $I$ is not necessary in order to recover the matrix $M$ from the last column $L$.

The following proposition, proved in [20], states that three well known properties of the $BWT$ hold, in a slightly modified form, for the $ABWT$ as well. Here we report the proof for the sake of completeness.

Proposition 3.2

Let $w$ be a word and let $(L,I)$ be the output of $BWT$ or $ABWT$ applied to $w$. The following properties hold:

Let $F$ denote the first column of $M_{{lex}}(w)$ (resp. $M_{{alt}}(w))$, then $F$ is obtained by lexicographically sorting the symbols of $L$. 2. 2.

For every $i$, $0\leq i<n$, $L[i]$ circularly precedes $F[i]$ in the original word, for both $BWT$ and $ABWT$. 3. 3.

For each symbol $a$, and $1\leq j\leq|w|_{a}$, the $j$-th occurrence of $a$ in $F$ corresponds

1. (a)

for $BWT$, to its $j$-th occurrence in $L$ 2. (b)

for $ABWT$, to its $(|w|_{a}-j+1)$-th occurrence in $L$.

Proof 1

Properties 1, 2 and 3a for the $BWT$ have been established in [4]. Properties 1 and 2 for the $ABWT$ are straightforward. To prove property 3b, consider two rows $i$ and $j$ in $M_{{alt}}(w)$ with $i<j$ starting with the symbol $a$. Let $w_{i}$ and $w_{j}$ be the two conjugates of $w$ in rows $i$ and $j$ of $M_{{alt}}(w)$. By construction we have $w_{i}=au$, $w_{j}=av$ and $w_{i}\preceq_{alt}w_{j}$. To establish Property 3b, we need to show that row $w_{j}$ cyclically rotated precedes in the $\preceq_{alt}$ order row $w_{i}$ cyclically rotated. In other words, we need to show that

[TABLE]

To prove the above implication, we notice that if the first position in which $au$ and $av$ differ is odd (resp. even) then the first position in which $va$ and $ua$ differ will be in an even (resp. odd) position. The thesis follow by the alternate use of the standard and reverse order in $\preceq_{alt}$ (see [18] for a different proof of the same property).\qed

It is well known that in the $BWT$ the occurrences of the same symbol appear in columns $F$ and $L$ in the same relative order; according to Property 3b. In the $ABWT$, the occurrences in $L$ appear in the reverse order than in $F$. For example, in Fig. 1 (right) we see that the $a$’s of $acaabr$ in the columns $F$ appear in the order 1st, 3rd, and 2nd, while in column $L$ they are in the reverse order: 2nd, 3rd, and 1st.

Proposition 3.2 is the key motivations to efficiently recover the original string from the output of $BWT$ or $ABWT$, as we will see in Section 5.

Note that, although $BWT$ and $ABWT$ are very similarly defined, they are very different combinatorial tools. Combinatorial aspects that distinguish $ABWT$ and $BWT$ can be found in [18, 20], which makes it interesting to study $ABWT$ in terms of tool characterizing families of words.

However, in [20] we experimentally tested $ABWT$ as pre-processing of a compression tool, by comparing its performance with a $BWT$-based compressor. We have shown that the behaviour of the two transformations is essentially equivalent in terms of compression. Actually, such experiments confirm a theoretical result we proved in [20] for a larger class of transformations that can be seen as a generalization of the $BWT$ and that includes the $ABWT$ as a special case. In next section, we give a brief description of the properties we proved in [20] for such a class of transformations, all of which also hold for the $ABWT$.

4 Generalized BWTs: a synopsis

In this section we describe the class of Generalized BWTs, introduced in [20], by reporting their main properties.

Given the alphabet $\Sigma$ of size $\sigma$, in the following, we denote by $\Pi_{\Sigma}$ the set of $\sigma!$ permutations of the alphabet symbols. Two important permutations are distinguished in $\Pi_{\Sigma}$: the identity permutation $Id$ corresponding to the lexicographic order, and the reverse permutation $Rev$ corresponding to the reverse lexicographic order. We consider generalized lexicographic orders introduced in [43] (cf. also [10]) that, for the purposes of this paper, can be formalized as follows.

Definition 4.1

Given a $k$-tuple $K=(\pi_{0},\pi_{1},\ldots,\pi_{k-1})$ of elements of $\Pi_{\Sigma}$, we denote by $\preceq_{K}$ the lexicographic order such that given two words of the same length $x=x_{0}x_{1}\cdots x_{s-1}$ and $y=y_{0}y_{1}\cdots y_{s-1}$ it is $x\preceq_{K}y$ if and only if $x=y$ or $x_{i}<_{i}y_{i}$ where $i$ is the smallest index such that $x_{i}\neq y_{i}$, and $<_{i}$ is the lexicographic order induced by the permutation $\pi_{i\bmod k}$. Without loss of generality, we can assume $\pi_{0}=Id$.

Using the above definition, a class of generalized $BWT$s can be defined as follows:

Definition 4.2

Given a $k$-tuple $K=(Id,\pi_{1},\ldots,\pi_{k-1})$ of elements of $\Pi_{\Sigma}$, we denote by $BWT_{K}$ the transformation mapping a primitive word $w$ to the last column $L$ of the matrix $M_{K}(w)$ containing the cyclic rotations of $w$ sorted according to the lexicographic order $\preceq_{K}$. The output of $BWT_{K}$ applied to $w$ is the pair $(bwt_{K}(w),I)$, where $bwt_{K}(w)$ is the last column $L$ of the matrix and $I$ is the row of $M_{K}(w)$ containing the word $w$.

Note that for $K=(Id)$, $BWT_{K}$ is the usual $BWT$, while for $K=(Id,Rev)$, $BWT_{K}$ coincides with the $ABWT$ defined in Section 3.

Remark 4.3

For most applications, it is assumed that the last symbol of $w$ is a unique end-of-string marker smaller than each symbol of the alphabet $\Sigma$. Under this assumption, lexicographically sorting $w$’s cyclic rotations is equivalent to building the suffix tree [28, 23] for $w$, which can be done in linear time. In this setting, we can compute $bwt_{K}(w)$ in linear time: we do a depth-first visit of the suffix tree in which the children of each node are visited in the order induced by $K$. In other words, the children of each node $v$ are visited according to the order $\pi_{|v|\bmod k}$ where $|v|$ is the string-depth111The number of letters in the word obtained by concatenating the labels of the edges in the path from the root of the suffix tree to the node $v$ of node $v$. Since the suffix tree has $O(|w|)$ nodes, for a constant alphabet the whole procedure takes linear time.

The next result proved in [20] guarantees that the transformations $BWT_{K}$ are invertible. As we specify in next section, the inversion procedure for $ABWT$ is more efficient.

Theorem 4.4

For every $k$-tuple $K=(Id,\pi_{1},\ldots,\pi_{k-1})$ the transformation $BWT_{K}$ is invertible in $O(n^{3})$ time, where $n=|w|$.\qed

Note that recently [21], the complexity for the inversion of a generic transformation in $BWT_{K}$ has been improved to ${\cal O}(n^{2})$ time.

The following theorem proved in [20] shows that each transformation $BWT_{K}$ produces a number of equal-letter runs that is at most the double of the number of equal-letter runs of the input word. This fact generalizes a result proved for $BWT$ [33].

Theorem 4.5

Given a $k$-tuple $K=(Id,\pi_{1},\ldots,\pi_{k-1})$ and a word $w$ over a finite alphabet $\Sigma$, then

[TABLE]

\qed

A key property of $BWT$ is that it allows to reduce the problem of compressing a string $w$ up to its $r$-th order entropy to the problem of compressing a collection of factors of $bwt(w^{R})$ up to their [math]-th order entropy, where $w^{R}$ is the reverse of the word $w$. This means that a $BWT$-based compressor combining $BWT$ with a zero order (memoryless) compressor, is able to achieve the same high order compression typical of more complex tools such as Lempel-Ziv encoders. In [20], we prove that a similar result also holds for the transformation $BWT_{K}$.

Theorem 4.6

Let $K$ be a $k$-tuple and $u=bwt_{K}(w^{R})$, where $w^{R}$ is the reverse of the word $w$. For each positive integer $r$, there exists a factorization of $u=u_{1}u_{2}\ldots u_{m}$ such that

[TABLE]

5 Rank-invertible transformations

It is well known that the key to efficiently compute the inverse of original $BWT$ is the existence of a easy-to-compute permutation mapping, in the matrix $M_{lex}(w)$, a row index $i$ to the row index $LF(i)$ containing row $i$ right-shifted by one position. This permutation is called $LF$-mapping since, by Proposition 3.2, $LF(i)$ is the position in the first column $F$ of $M_{lex}(w)$ corresponding to the $i$-th entry in column $L$: in other words, $F[LF(i)]$ is the same symbol in $w$ as $L[i]$. Again, by Proposition 3.2 we have that $L[LF(i)]$ is the symbol preceding $L[i]$ in the input word $w$. Define $LF^{0}(x)=x$ and $LF^{j+1}(x)=LF(LF^{j}(x))$. If $bwt(w)=(L,I)$ with $|w|=n$, then by construction $L[I]=w_{n-1}$ and we can recover $w$ with the formula:

[TABLE]

Note that the inversion formula (1) only depends on Properties 1 and 2 of Proposition 3.2. Since such properties hold for every generalized transformation $BWT_{K}$, (1) provides an inversion formula for every transformation in that class. In other words, inverting a generalized $BWT$ amounts to computing $n$ iterations of the $LF$-mapping.

By Property 3a in Proposition 3.2, the $LF$-mapping for the original $BWT$ can be expressed using the Parikh vector $P_{L}$ of $L$ and a rank operation over $L$:

[TABLE]

Note that $\sum_{c\in\Sigma}^{c<L(i)}P_{L}[c]$ is simply the total number of occurrences of symbols smaller than $L[i]$ in $L$, and $\mathop{\mathrm{rank}}\nolimits_{L[i]}(L,i-1)$ is the number of occurrences of symbol $L[i]$ in among the first $i$ symbols of $L$.

By Property 3b in Proposition 3.2, for the $ABWT$, the corresponding formula is:

[TABLE]

Since the rank operation on (compressed) arrays over finite alphabet can be computed in constant time [1] and the partial sums $\sum_{c<i}P_{L}[c]$ can be precomputed, the computation of the $LF$ map for both the $BWT$ and $ABWT$ takes $O(1)$ time. This implies that, thanks to the simple structure of its $LF$-mapping, also the $ABWT$ can be inverted in linear time.

The computation of the $LF$ map is the main operation also for the so-called backward-search procedure which makes it possible to use (a compressed version of) $bwt(w)$ as a full text index for $w$ [16]. The following proposition is the key to generalize the backward search procedure to the $ABWT$.

Proposition 5.1

Given a string $p\in\Sigma^{*}$, let $[b,e]$ denote the range of rows of $M_{alt}(w)$ which are prefixed by $p$. For any $x\in\Sigma$, let

[TABLE]

If $b^{\prime}\leq e^{\prime}$, then $[b^{\prime},e^{\prime}]$ is the range of rows of $M_{alt}(w)$ which are prefixed by $xp$ if $b^{\prime}>e$ then no rows of $M_{alt}(w)$ are prefixed by $xp$ and therefore $xp$ is not a (circular) substring of $w$.

Proof 2

Assume first $b^{\prime}\leq e^{\prime}$. It is immediate that if $i$, $j$ are the positions of the first and last $x$ in $L[b,e]$, then $b^{\prime}=LF(j)$ and $e^{\prime}=LF(i)$ and every other $x$ in $L[b,e]$ is mapped to a position between $b^{\prime}$ and $e^{\prime}$. The thesis follows since all rows in $M_{alt}(w)$ are rotations of $w$. If $b^{\prime}>e$, then $\mathop{\mathrm{rank}}\nolimits_{x}(L,e-1)=\mathop{\mathrm{rank}}\nolimits_{x}(L,b)$ and there are no $x$’s in $L[e,b]$ and $xp$ is not a circular substring of $w$.\qed

Proposition 5.1 implies that if we use a compressed representation of the last columns $L$ of $M_{alt}(w)$ supporting constant time rank operations, then, for any pattern $p$, we can compute in $O(|p|)$ time the range of rows of the matrix $M_{lex}(w)$ which are prefixed by $p$. Hence, the $ABWT$ can be used as a compressed index in the same way as the $BWT$.

The above results suggest that it is worthwhile to search for other transformations in the class $BWT_{K}$ which share the same properties of $BWT$ and $ABWT$. Because of the important role played by the rank operation, we introduce the notion of rank-invertibility for the class of $BWT_{K}$ transformations.

Definition 5.2

The transformation $BWT_{K}$ is rank-invertible if there exists a function $f_{K}$ such that, for any word $w$, setting $L=bwt_{K}(w)$ we have

[TABLE]

In other words, $LF(i)$ only depends on the Parikh vector $P_{L}$ of $L$, the symbol $L[i]$, and the number of occurrences of $L[i]$ in $L$ up to position $i$.\qed

Note that we pose no limit to the complexity of the function $f_{K}$, we only ask that it can be computed using only $P_{L}$ and the number of occurrences of $L[i]$ in $L[0,i]$.

We observed that, for $K=(Id,Rev)$, $BWT_{K}$ coincides with $ABWT$ and it is therefore rank-invertible. The main result of this section is Theorem 5.9 establishing that $BWT$ and $ABWT$ are the only rank-invertible transformations in the class $BWT_{K}$. We start our analysis considering the case $|K|=2$.

Lemma 5.3

Let $\Sigma=\{a,b,c\}$, and $K=(Id,\pi)$, where $\pi$ is a permutation of $\Sigma$. If there exist two pairs $t_{1}=(x,y)$ and $t_{2}=(z,w)$ of symbols of $\Sigma$ such that

[TABLE]

then $BWT_{K}$ is not rank-invertible.

Proof 3

Consider for example the case $\pi=(c,a,b)$. Two pairs satisfying the hypothesis are $t_{1}=(a,b)$ and $t_{2}=(b,c)$ since according to the ordering $<_{\pi}$ it is

[TABLE]

Consider now the two words $s_{1}=aabcc$ and $s_{2}=abacc$. Both words contain two $a$’s. In the first word the $a$’s are followed respectively by $a,b$ (the symbols in $t_{1}$), and in $s_{2}$ the $a$’s are followed by $b,c$ (the symbols in $t_{2}$).

Let $F_{1}$, $L_{1}$ (resp. $F_{2}$, $L_{2}$) denote the first and last column of the matrix $M_{K}$ associated to $bwt_{K}(s_{1})$ (resp. $bwt_{K}(s_{2})$). By definition, each matrix is obtained sorting the cyclic rotations of $s_{1}$ and $s_{2}$ according to the lexicographic order $\prec_{K}$ where symbols in odd positions are sorted according to the usual alphabetic order, while symbols in even positions are sorted according to the ordering $\pi$. We show the two matrices in Fig. 2, where we use subscripts to distinguish the two $a$’s occurrences in $s_{1}$ and $s_{2}$.

The relative position of the two $a$’s in $L_{1}$ is determined by the symbols following them in $s_{1}$, namely those in $t_{1}=(a,b)$. Since these symbols are in the first column of the cyclic rotations matrix, which is sorted according to the usual alphabetic order, the two $a$’s appear in $L_{1}$ in the order $a_{1},a_{2}$. The same is true for $L_{2}$: since the pair $t_{2}$ is also sorted, the two $a$’s appear in $L_{2}$ in the order $a_{1},a_{2}$.

The position of the two $a$’s in $F_{1}$ is also determined by the symbols following them in $s_{1}$; but since these symbols are now in the second column, their relative order is determined by the ordering $\pi$. Hence the two $a$’s appear in $F_{1}$ in the order $a_{1},a_{2}$. In $F_{2}$ the ordering of the $a$’s is $a_{2},a_{1}$ since it depends on the $\pi$-ordering of the symbols of $t_{2}$ which by construction is different than their $Id$-ordering.

Note that $s_{1}$ and $s_{2}$ have the same Parikh vector $\langle 2,1,2\rangle$. If, by contradiction, $BWT_{K}$ were rank invertible, the function $f_{K}$ should give the correct LF-mapping for both $s_{1}$ and $s_{2}$. This is impossible since for $s_{1}$ we should have

[TABLE]

while for $s_{2}$ we should have

[TABLE]

In the general case of an arbitrary permutation $\pi$ satisfying the hypothesis of the lemma the reasoning is the same. Note that such permutations are $(a,c,b)$, $(b,a,c)$, $(b,c,a)$ and $(c,a,b)$. Given the two pairs $t_{1}$ and $t_{2}$ we build two words $s_{1}$ and $s_{2}$ with Parikh vector $\langle 2,1,2\rangle$ such that in $s_{1}$ (resp. $s_{2}$) the two occurrences of $a$ are followed by the symbols in $t_{1}$ (resp. $t_{2}$). We then build the rotation matrices as before, and we find that in both $L_{1}$ and $L_{2}$ the two $a$’s are in the order $a_{1},a_{2}$. However, in columns $F_{1}$ and $F_{2}$ the two $a$’s are not in the same relative order since it depends on the ordering $\pi$, and, by construction, such an order is not the same. Reasoning as before, we get that there cannot exist a function $f_{K}$ giving the correct LF-mapping for both $s_{1}$ and $s_{2}$.\qed

Lemma 5.4

Let $|\Sigma|\geq 2$ and $K=(Id,\pi)$. Then $BWT_{K}$ is rank-invertible if and only if $\pi=Id$ or $\pi=Rev$.

Proof 4

If $|\Sigma|=2$ the result is trivial since the only possible permutations on binary alphabet are the identity and reverse permutation. Let us assume $|\Sigma|\geq 3$. We need to prove that if $\pi\neq Id$ and $\pi\neq Rev$ then $BWT_{K}$ is not rank-invertible.

Note that any permutation $\pi$ over the alphabet $\Sigma$ induces a new ordering on any triplet of symbols in $\Sigma$. For example, if $\Sigma=\{a,b,c,d,e,f\}$ the permutation $\pi=(d,e,c,f,a,b)$ induces on the triplet $\{a,b,c\}$ the ordering $\pi_{abc}=(c,a,b)$. It is easy to prove by induction on the alphabet size that, if $\pi\neq Id$ and $\pi\neq Rev$, then there exists a triplet $\{x,y,z\}$, with $x<y<z$, such that $\pi_{xyz}\neq(x,y,z)$ and $\pi_{xyz}\neq(z,y,x)$. That is, $\pi$ restricted to $\{x,y,z\}$ is different from the identity and reverse permutation. Without loss of generality we can assume that the triplet is $\{a,b,c\}$.

For any permutation $\pi_{abc}$, different from $(a,b,c)$ and $(c,b,a)$, there exist two pairs of symbols satisfying the hypothesis of Lemma 5.3. Hence, we can build two words $s_{1}$ and $s_{2}$ which show that $BWT_{K}$ is not rank-invertible. Note that the argument in the proof of Lemma 5.3 is still valid if we add to $s_{1}$ and $s_{2}$ the same number of occurrences of symbols in $\Sigma$ different from $a,b,c$ so that $s_{1}$ and $s_{2}$ are effectively over an alphabet of size $|\Sigma|$.\qed

Lemma 5.4 establishes which $BWT_{K}$ transformations are rank-invertible when $|K|=2$. To study the general case $|K|>2$, we start by establishing a simple corollary.

Corollary 5.5

Let $|\Sigma|\geq 3$ and $K=(Id,\pi,\pi_{2},\ldots,\pi_{k-1})$. If $\pi\neq Id$ and $\pi\neq Rev$ then $BWT_{K}$ is not rank-invertible.

Proof 5

We reason as in the proof of Lemma 5.4, observing that the presence of the permutations $\pi_{2},\ldots,\pi_{k-1}$ has no influence on the proof since the row ordering is determined by the first two symbols of each rotation.\qed

The following three lemmas establish necessary conditions on the structure of the tuple $K$ for $BWT_{K}$ to be rank-invertible. In particular, the following lemma shows that $BWT_{K}$ is not rank-invertible if $K$ contains anywhere a triplet $(Id,Id,\pi)$ with $\pi\neq Id$.

Lemma 5.6

Let $|\Sigma|\geq 2$ and $K=(Id,\pi_{1},\ldots,\pi_{i-1},Id,Id,\pi,\pi_{i+3},\ldots,\pi_{k-1})$, $i\geq 0$, with $\pi\neq Id$. Then $BWT_{K}$ is not rank-invertible.

Proof 6

Note that when $i=0$, the $k$-tuple $K$ starts with the triplet $(Id,Id,\pi)$. We first analyze the case $|\Sigma|=2$ implying that $\pi=Rev$. Let us consider the words $s_{1}=a_{1}b^{i}a_{2}b^{i+1}bb$ and $s_{2}=a_{1}b^{i+1}a_{2}b^{i+1}b$, where we use subscripts to distinguish the two different occurrences of the symbol $a$. It is easy to see that, in the cyclic rotations matrix for $s_{1}$, $a_{1}$ precedes $a_{2}$ in both the first and the last column. Hence if $BWT_{K}$ were rank-invertible we should have

[TABLE]

At the same time, in the cyclic rotations matrix for $s_{2}$, $a_{1}$ precedes $a_{2}$ in the last columns, but in the first column $a_{2}$ precedes $a_{1}$ since the two rotations prefixed by $a$ differ in the third column and $b<_{Rev}a$. Therefore we should have

[TABLE]

Hence $BWT_{K}$ cannot be rank-invertible.

Let us consider the case $|\Sigma|\geq 3$. Since $\pi\neq Id$ there are two symbols, say $b$ and $c$, such that their relative order according to $\pi$ is reversed, that is, $b<c$ and $c<_{\pi}b$. Consider now the words $s_{1}=a_{1}c^{i}ba_{2}c^{i}ccc$ and $s_{2}=a_{1}c^{i+1}ba_{2}c^{i+1}c$ where we use subscripts to distinguish the two different occurrences of the symbol $a$. It is immediate to see that, in the cyclic rotations matrix for $s_{1}$, $a_{1}$ precedes $a_{2}$ in both the first and the last column. Hence if $BWT_{K}$ were rank-invertible we should have

[TABLE]

At the same time, in the cyclic rotations matrix for $s_{2}$, $a_{1}$ precedes $a_{2}$ in the last columns, but in the first column $a_{2}$ precedes $a_{1}$ since the two rotations prefixed by $a$ differ in the $(i+3)$-th column and $c<_{\pi}b$. Hence we should have

[TABLE]

hence $BWT_{K}$ cannot be rank-invertible.\qed

The following lemma shows that $BWT_{K}$ is not rank-invertible if $K$ contains anywhere a triplet $(Id,Rev,\pi)$, with $\pi\neq Id$.

Lemma 5.7

Let $|\Sigma|\geq 2$ and $K=(Id,\pi_{1},\ldots,\pi_{i-1},Id,Rev,\pi,\pi_{i+3},\ldots,\pi_{k-1})$, $i\geq 0$, with $\pi\neq Id$. Then $BWT_{K}$ is not rank-invertible.

Proof 7

As in the proof of Lemma 5.6, we can consider the words $s_{1}=a_{1}b^{i}a_{2}b^{i+1}bb$ and $s_{2}=a_{1}b^{i+1}a_{2}b^{i+1}b$ in case of binary alphabet, and the words $s_{1}=a_{1}c^{i}ba_{2}c^{i}ccc$ and $s_{2}=a_{1}c^{i+1}ba_{2}c^{i+1}c$ in the general case by assuming that there are two symbols, say $b$ and $c$, such that their relative order according to $\pi$ is reversed, that is, $b<c$ and $c<_{\pi}b$. Recall that we use subscripts to distinguish the two different occurrences of the symbol $a$. In the cyclic rotations matrix for $s_{1}$, in the first column $a_{2}$ precedes $a_{1}$ while in the last column $a_{1}$ precedes $a_{2}$. At the same time, in both the first and the last column of the cyclic rotations matrix for $s_{2}$, $a_{2}$ precedes $a_{1}$. Reasoning as in the proof of Lemma 5.6 we get that $BWT_{K}$ cannot be rank-invertible.\qed

The following lemma shows that $BWT_{K}$ is not rank-invertible if $K$ contains anywhere a triplet $(Rev,Id,\pi)$, with $\pi\neq Rev$.

Lemma 5.8

Let $|\Sigma|\geq 2$ and $K=(Id,\pi_{1},\ldots,\pi_{i-1},Rev,Id,\pi,\pi_{i+3},\ldots,\pi_{k-1})$, $i\geq 0$, with $\pi\neq Rev$. Then $BWT_{K}$ is not rank-invertible.

Proof 8

We reason as in the proof of Lemma 5.7 considering again the words $s_{1}=ab^{i}ab^{i+1}bb$ and $s_{2}=ab^{i+1}ab^{i+1}b$ in case of binary alphabet and the words $s_{1}=ac^{i}bac^{i}ccc$ and $s_{2}=ac^{i+1}bac^{i+1}c$ in the general case.\qed

We are now ready to establish the main result of this section.

Theorem 5.9

If $|\Sigma|\geq 2$, $BWT$ and $ABWT$ are the only transformations $BWT_{K}$ which are rank invertible.

Proof 9

For $|K|=2$, the result follows from Lemma 5.4. Let us suppose $K=(Id,\pi_{1},\ldots,\pi_{k-1})$ with $k>2$ and assume $BWT_{K}$ is rank invertible. Both in the case of binary alphabet and in the general case, by Corollary 5.5, we must have $\pi_{1}=Id$ or $\pi_{1}=Rev$. If $\pi_{1}=Id$ and $BWT_{K}\neq BWT$ then the $k$-tuple $K$ must contain the triplet $(Id,Id,\pi)$ with $\pi\neq Id$ which is impossible by Lemma 5.6. If $\pi_{1}=Rev$, by Lemma 5.7 $\pi_{2}=Id$. We have therefore established that $K$ has the form $K=(Id,Rev,Id,\pi_{3},\ldots,\pi_{k-1})$. By Lemma 5.8 it is $\pi_{3}=Rev$. By iterating the same reasoning we can conclude that $BWT_{k}$ coincides with $ABWT$, concluding the proof.\qed

6 Efficient computation of the ABWT

The bottleneck for the computation of $ABWT$ (as well as of any transformation $BWT_{K}$) of a given string $w$ is the $\preceq_{alt}$-based (the $\preceq_{K}$-based) sorting of its cyclic rotations. In Remark 4.3 we have observed that, if a unique end-of-string symbol, which is smaller than any other symbol in the alphabet, is appended to the input string, all transformations in the class $BWT_{K}$ can be computed in linear time by first building the suffix tree for the input string. However, for computing the $BWT$ this strategy has never been used in practice. The reason is that the algorithms for building the suffix tree, although they take linear time, have a large multiplicative constant and are not fast in practice. In addition, the suffix tree itself requires a space of about ten/fifteen times the size of the input which is a huge amount of temporary space that is not necessarily available (considering also that saving space is the primary reason for using the $BWT$). For the above reasons the $BWT$ is usually computed by first building the Suffix Array [24, 37] which is the array giving the lexicographic order of all the suffixes of the input string.

A fundamental result on Suffix Array construction is the technique in [25] that, using the concept of difference cover, makes it possible to design efficient Suffix Array construction algorithms for different models of computation including RAM, External Memory, and Cache Oblivious.

In this section, we show that this technique can be adapted to compute the $ABWT$ within the same time bound of the $BWT$.

Firstly, in order to use the notion of suffix array for the computation of $ABWT$ we need to extend the definition of alternating lexicographic order also for strings having different length.

Definition 6.1

Let $x=x_{0}x_{1}\ldots x_{s-1}$ and $y=y_{0}y_{1}\ldots y_{t-1}$ with $s<t$.

If $x$ is not a prefix of $y$ and $i$ is the smallest index in which $x_{i}\neq y_{i}$ Then, if $i$ is even $x\prec_{alt}y$ iff $x_{i}<y_{i}$. Otherwise, if $i$ is odd $x\prec_{alt}y$ iff $x_{i}>y_{i}$. 2. 2.

If $x$ is a prefix of $y$, we say that $x\prec_{alt}y$ if $|x|$ is even, $y\prec_{alt}x$ if $|x|$ is odd.

Suffix array algorithms often assume that the input string ends with a unique end-of-string symbol smaller than any other in the alphabet $\Sigma$. Remark that if we append the end-of-string symbol \$$ to the string w$, the$\prec_{alt}$-order relation between two suffixes of$w$is determined by using Definition [6.1](#S6.Thmtheorem1) (case 1). Moreover, using the end-of-string symbol$$implies that the$\preceq_{alt}$-based sorting of the cyclic rotations of input string is induced by the$\prec_{alt}$-based sorting of its suffixes. Note that this property does not hold in general. However, it is easy to verify that, apart from the symbol$$$, the output$abwt(w$)$may be different from$abwt(w)$ and the number of equal letter runs can be greater (see Fig. 4).

Here we assume that the input string $w$ contains a unique end-of-string symbol $$$, but, in the next section, we show how to remove this hypothesis by using combinatorial properties of some special rotations of the input string.

To illustrate the idea behind difference cover algorithms, in the following, given a positive integer $v$, we denote by $[0,v)$ the set $\{0,1,\ldots,v-1\}$.

Definition 6.2

A set $D\subseteq[0,v)$ is a difference cover modulo $v$ if every integer in $[0,v)$ can be expressed as a difference, modulo $v$, of two elements of $D$, i.e.

[TABLE]

\qed

For example, for $v=7$ the set $\{0,1,3\}$ is a difference cover, since $0=0-0$, $1=1-0$, $2=3-1$, $3=3-0$, $4=0-3\bmod 7$, and so on. An algorithm by Colbourn and Ling [6] ensures that for any $v$ a difference cover modulo $v$ of size at most $\sqrt{1.5v}+6$ can be computed in $O(\sqrt{v})$ time. The suffix array construction algorithms described in [25] are based on the general strategy shown in Algorithm 1. Steps 3 and 4 rely heavily on the following property of Difference covers: for any $0\leq i,j<n$ there exists $k<v$ such that $(i+k)\bmod v\in D$ and $(j+k)\bmod v\in D$. This implies that to compare lexicographically suffixes $w[i,n-1]$ and $w[j,n-1]$ it suffices to compare at most $v$ symbols since $w[i+k,n-1]$ and $w[j+k,n-1]$ are both sampled suffixes and their relative order has been determined at Step 2.

To see how the algorithm works consider for example $v=6$, $D=\{0,1,3\}$ and the string w=abaacabaacab\$$. The sampled suffixes are those starting at positions 0,1,3,6,7,9,12$. To sort them, consider the string over$\Sigma^{v}$whose elements are the$v$-tuples starting at the sampled positions in the order$0,6,12,1,7,3,9$:

[TABLE]

(note we have added additional $’s to make sure all blocks contain$v$symbols). The difference cover algorithm then renames each$v$-tuple with its lexicographic rank. Since

[TABLE]

the renamed string is $R_{bwt}=[2,4,0,4,1,3,5]$. The crucial observation is that the suffix array for $R_{bwt}$, which in our example is $SA(R_{bwt})=[2,4,0,5,1,3,6]$, provides the lexicographic ordering of the sampled suffixes. Indeed $R[2]=w[12,18]$ is the smallest sampled suffix, followed by $R[4]=w[7,12]$, followed by $R[0]=w[0,5]$, and so on. The Suffix Array of $R_{bwt}$ is computed with a recursive call at Step 2, and is later used in Steps 3 and 4 to complete the sorting of all suffixes.

To compute $abwt(w)$ with the difference cover algorithm, we consider the same string $R$ but we sort the $v$-tuples according to the alternating lexicographic order. Since

[TABLE]

it is $R_{abwt}=[2,4,0,4,3,1,5]$. Next, we compute the Suffix Array of $R_{abwt}$ according to the standard lexicographic order, $SA(R_{abwt})=[2,5,0,4,1,3,6]$. We now show that, since $v=6$ is even, $SA(R_{abwt})$ provides the correct alternating lexicographic order of the sampled suffixes.

To see this, assume $w[i,n-1]$ and $w[j,n-1]$ are sampled suffixes with a common prefix of length $\ell$. Hence $w[i,i+\ell-1]=w[j,j+\ell-1]$ while $w[i+\ell]\neq w[j+\ell]$. Let $R_{abwt}[t_{i}]$ and $R_{abwt}[t_{j}]$ denote the entries in $R_{abwt}$ corresponding to $w[i,i+v-1]$ and $w[j,j+v-1]$. By construction, the suffixes $R_{abwt}[t_{i},r]$ and $R_{abwt}[t_{j},r]$ have a common prefix of $\lfloor\ell/v\rfloor$ entries (each one corresponding to a length-$v$ block in $w$) followed respectively by $R_{abwt}[t_{i}+\lfloor\ell/v\rfloor]$ and $R_{abwt}[t_{j}+\lfloor\ell/v\rfloor]$ which are different since they correspond to the $v$-tuples $R[t_{i}+\lfloor\ell/v\rfloor]$ and $R[t_{j}+\lfloor\ell/v\rfloor]$ which differ since they contain the symbols $w[i+\ell]$ and $w[j+\ell]$ in position $(\ell\bmod v)$. Assuming for example that $w[i+\ell]<w[j+\ell]$, it is $w[i,n-1]\prec_{alt}w[j,n-1]$ depending on whether $\ell$ is odd or even. Since $v$ is even, $\ell$ is even iff $\ell\bmod v$ is even, hence

[TABLE]

which shows that the standard Suffix Array for $R_{abwt}$ provides the alternating lexicographic order of the sampled suffixes, as claimed.

Summing up, after building the string $R_{abwt}$, at Step 2 we compute $SA(R_{abwt})$ using the standard Difference cover algorithm, or any other suffix sorting algorithm. Finally, Step 3 and 4 can be easily adapted to the alternating lexicographic order using its property that for any symbol $c\in\Sigma$ and strings $\alpha,\beta\in\Sigma^{*}$ it is

[TABLE]

For example, to compare $w[0,12]$ with $w[5,12]$ we notice that after $w[0]=w[5]$ we reach the sampled suffixes $w[1,12]$ and $w[6,12]$ corresponding to $R[3,6]$ and $R[1,6]$. According to $SA(R_{abwt})$ it is $R[1,6]\preceq_{lex}R[3,6]$ which implies $w[6,12]\prec_{alt}w[1,12]$, and by (4) $w[0,12]\preceq_{lex}w[5,12]$. Since from the alternating lexicographic order of $w$’s suffixes $abwt(w)$ can be computed in linear time, the results in [25] can be translated as follows.

Theorem 6.3

Given a string $w[0,n-1]$ ending with a unique end-of-string symbol, we can compute $abwt(w)$ in RAM in ${\cal O}(n)$ time, or in ${\cal O}(n\log\log n)$ time but using only $n+o(n)$ words of working space. In external memory, using $D$ disks with block size $B$ and a fast memory of size $M$, $abwt(w)$ can be computed in ${\cal O}(\frac{n}{DB}\log_{M/B}n/B)$ I/Os and ${\cal O}(n\log_{M/B}n/B)$ internal work.\qed

We point out that the above results cannot be easily extended to the generalized BWTs introduced in Section 4. The reason is that Step 3 and 4 of the modified Difference cover algorithm hinge on Property (4) that does not hold in general for the lexicographic orders introduced by Definition 4.1.

7 Galois words and ABWT computation for arbitrary rotations

Galois words, introduced in [43], are generalization of Lyndon words for the alternating lexicographic order. Roughly speaking, a Galois word is the smallest cyclic rotation within its conjugacy class, with respect to $\preceq_{alt}$ order. Although, in general, Galois and Lyndon words are distinct within a conjugacy class, some properties that hold for Lyndon words are preserved. Some characterizations of Galois words by using infinite words and some properties of words that are obtained as a nonincreasing factorization in Galois words, are studied in [10].

In this section, we explore some combinatorial properties of Galois words and, in particular, we show a linear time and space algorithm to find the Galois rotation of a word. These results, on one hand, give an answer to a question posed in [10]. On the other hand, they allow to prove that, for the computation of the $ABWT$ of a string $w$, Galois words play a role similar to that of Lyndon words for $BWT$ [22, 2], i.e. the computation of $ABWT$ can be linearly performed, even if no end-of-string symbol is appended to the input.

Definition 7.1

A primitive word $w$ is a Galois word if for each nontrivial factorization $w=uv$, one has $w\preceq_{alt}vu$.\qed

Example 7.2

The words $w=ababba$ and $v=aababb$ are, respectively, the Galois word and the Lyndon word within the same conjugacy class. Another example is $w=ababaa$ and $v=aaabab$.\qed

Firstly, we show that a string $w$ is a Galois word if it is smaller than its proper suffixes, with respect to $\prec_{alt}$ order introduced in Definition 6.1 (see Fig. 3 for an example).

The following result has been proved in [43] (Proposition 3.1). Here we report the proof by using our notation.

Lemma 7.3

If a Galois word $w$ has a border, then it has odd length.

Proof 10

Let $u$ be both suffix and prefix of $w$. This means that $w=uv^{\prime}=v^{\prime\prime}u$. By definition, $w=uv^{\prime}\prec_{alt}uv^{\prime\prime}$. If $|u|$ would be even, then it should be $v^{\prime}\prec_{alt}v^{\prime\prime}$. On the other hand, $w=v^{\prime\prime}u\prec_{alt}v^{\prime}u$ implies that $v^{\prime\prime}\prec_{alt}v^{\prime}$, a contradiction.\qed

Lyndon words can be defined as the strings that are smaller of its proper suffixes. Such a characterization also holds for Galois words, as shown in the following proposition. A different proof of this result, involving infinite words, is given in [10].

Proposition 7.4

A primitive word $w$ is a Galois word if and only if $w$ is smaller than any of its suffix, with respect to $\prec_{alt}$ order.

Proof 11

Let $w$ be a Galois word and let $v$ a suffix of $w$. This means that $w=uv$. If $v$ is also prefix of $w$, then by Lemma 7.3 $v$ has odd length, i.e. $w\prec_{alt}v$. If $v$ is not a prefix of $w$, then there exists $0\leq i<|v|-1$ such that $v_{i}\neq w_{i}$. Since $w$ is a Galois word, $uv\prec_{alt}vu$. This fact implies that $w=uv\prec_{alt}v$. Conversely, let $w=uv$. Since $uv\prec_{alt}v$, we can distinguish two cases, whether $v$ is prefix of $w$ or not. If $v$ is not a prefix of $w$ then $uv\prec_{alt}vu$. If $v$ is a prefix of $w$, then the length of $v$ is odd. Therefore if it would be $vu\prec_{alt}uv=vu^{\prime}$ then $u^{\prime}\prec_{alt}u$ that implies $u^{\prime}\prec_{alt}uv$ that is a contradiction.\qed

It is known that, when Lyndon words are considered, the lexicographic sorting of its suffixes induces the $\preceq_{lex}$-sorting of the conjugates. Such a result is useful to compute the $BWT$ of a string without using any end-of-string symbol [22]. The following proposition shows that this property also holds for Galois words. In fact, the $\preceq_{alt}$-based sorting of the cyclic rotations of a primitive Galois word can be reduced to the $\prec_{alt}$-based sorting of its suffixes. An example of this property is reported in Fig. 3.

Proposition 7.5

Let $w$ be a primitive Galois word and let $u^{\prime},u^{\prime\prime},v^{\prime},v^{\prime\prime}$ be factors of $w$ such that $w=u^{\prime}v^{\prime}=u^{\prime\prime}v^{\prime\prime}$. Then, $v^{\prime}u^{\prime}\prec_{alt}v^{\prime\prime}u^{\prime\prime}\iff v^{\prime}\prec_{alt}v^{\prime\prime}$.

Proof 12

Let us assume that $w^{\prime}=v^{\prime}u^{\prime}\prec_{alt}w^{\prime\prime}=v^{\prime\prime}u^{\prime\prime}$. There exists $0\leq i<|w|$ such that $w^{\prime}_{i}\neq w^{\prime\prime}_{i}$. Firstly, let us assume that $i<\min\{|v^{\prime}|,|v^{\prime\prime}|\}$. In this case $v^{\prime}\prec_{alt}v^{\prime\prime}$. Let us assume now that $v^{\prime}$ is a prefix of $v^{\prime\prime}$, i.e. $v^{\prime\prime}=v^{\prime}s$, for some non-empty string $s$. If $|v^{\prime}|$ would be odd, then $v^{\prime}u^{\prime}\prec_{alt}v^{\prime\prime}u^{\prime\prime}=v^{\prime}su^{\prime\prime}\Rightarrow su^{\prime\prime}\prec_{alt}u^{\prime}\Rightarrow su^{\prime\prime}v^{\prime}\prec_{alt}u^{\prime}v^{\prime}$, that is a contradiction. So, $|v^{\prime}|$ is even and by definition $v^{\prime}\prec_{alt}v^{\prime\prime}$. Let us consider the case $v^{\prime\prime}$ is a prefix of $v^{\prime}$, i.e. $v^{\prime}=v^{\prime\prime}t$, for some string $t$. If $|v^{\prime\prime}|$ would be even then $v^{\prime\prime}tu^{\prime}=v^{\prime}u^{\prime}\prec_{alt}v^{\prime\prime}u^{\prime\prime}\Rightarrow tu^{\prime}\prec_{alt}u^{\prime\prime}\Rightarrow tu^{\prime}v^{\prime\prime}\prec_{alt}u^{\prime\prime}v^{\prime\prime}$, that is a contradiction. So, $|v^{\prime\prime}|$ is odd then, by definition, $v^{\prime}\prec_{alt}v^{\prime\prime}$.

Conversely, let us suppose that $v^{\prime}\prec_{alt}v^{\prime\prime}$. If neither $v^{\prime}$ is prefix of $v^{\prime\prime}$ nor $v^{\prime\prime}$ is prefix of $v^{\prime}$, then $v^{\prime}u^{\prime}\prec_{alt}v^{\prime\prime}u^{\prime\prime}$. Let us suppose now that $v^{\prime}$ is prefix of $v^{\prime\prime}$, i.e. $v^{\prime\prime}=v^{\prime}s$ then $v^{\prime}$ has even length. Since $w$ is a Galois word, $u^{\prime}v^{\prime}\prec_{alt}su^{\prime\prime}v^{\prime}\Rightarrow u^{\prime}\prec_{alt}su^{\prime\prime}\Rightarrow v^{\prime}u^{\prime}\prec_{alt}v^{\prime}su^{\prime\prime}=v^{\prime\prime}u^{\prime\prime}$. Let us suppose now that $v^{\prime\prime}$ is prefix of $v^{\prime}$, i.e. $v^{\prime}=v^{\prime\prime}t$ then $v^{\prime\prime}$ has odd length. The fact that $w$ is a Galois word implies that $u^{\prime\prime}v^{\prime\prime}\prec_{alt}tu^{\prime}v^{\prime\prime}\Rightarrow u^{\prime\prime}\prec_{alt}tu^{\prime}\Rightarrow v^{\prime\prime}tu^{\prime}=v^{\prime}u^{\prime}\prec_{alt}v^{\prime\prime}u^{\prime\prime}$. \qed

It is known that the unique Lyndon conjugate of a string $w$ is one of the elements in the non-increasing factorization of $ww$ into Lyndon words [11]. As proved in [10], this strategy does not work for Galois words. Hence, we introduce a new linear time and space algorithm, named FindGaloisRotation, to find, for each primitive string $w$ of length $n$, its unique cyclic rotation that is a Galois word. Our algorithm is a variant of the one in [3, 28] to find the Lyndon conjugate of a given string. The algorithm FindGaloisRotation uses a border array $B$ of length $n+1$ that stores in each position $j>0$ the length of the border of the $j$-length prefix of $w[k,(k+j-1)\bmod n]$, i.e. the Galois rotation starting at position $k$, and $B[0]=-1$.

At each iteration of the main while loop (lines 3–13), $k$ is the starting position of the current candidate for the smallest cyclic rotation (with respect to $\preceq_{alt}$ order), $w[k,(k+j-1)\bmod n]$ is a Galois word and $B[j]=i$ is the length of its border. This means that $w[k,(k+i-1)\bmod n]=w[(k+j-i),(k+j-1)\bmod n]$. So, the characters $w[(k+i)\bmod n]$ and $w[(k+j)\bmod n]$ are compared. If those characters are equal, the length of the border is increased. If they are distinct, different alphabet orders are used depending on whether $i$ is even or not, and the value of $k$ is consequently updated. Note that, even if $k$ is changed, the computed value $B[j+1]$ is the same and the values $B[i]$, with $i\leq j$, do not need to be re-computed.

Theorem 7.6

Given a primitive string $w$, its unique cyclic rotation that is a Galois word can be computed in linear time and space.

Proof 13

We note that the auxiliary memory consists solely of the border array and that the execution time depends linearly on the number of comparisons between the characters in $w$. To prove that FindGaloisRotation requires at most $4n-3$ comparisons, we consider the quantity $2(k+j)-i$ and show that it always increases after each comparison between the characters $w[(k+j)\bmod n]$ and $w[(k+i)\bmod n]$. If the two characters are equal, then both $i$ and $j$ are increased by one at Line 13. If the two characters are different, then the quantity $k+j$ remains unchanged and the value of $i$ is decreased. Finally, note that if $n\geq 2$, the quantity $2(k+j)-i$ is equal to $2$ for the first comparison and it is at most $2(2n-1)$, so the overall number of comparisons is at most $4n-3$ as claimed.\qed

The next corollary shows how to use FindGaloisRotation procedure to compute in linear time the $ABWT$ of an input string without using any end-of-string symbol.

Corollary 7.7

The $ABWT$ of a generic string $w$ can be computed in linear time.

Proof 14

We apply FindGaloisRotation to $w$ to find its Galois conjugate $w^{\prime}$. Then we apply to w^{\prime}\$$ the algorithm described in previous section. By using Remark [3.1](#S3.Thmtheorem1), we can deduce that abwt(w)$can be obtained from$abwt(w^{\prime}$)$by just removing$$$ from the output (see Fig. 4 for an example).\qed

In literature some improvements of the algorithms for finding the Lyndon conjugate have been proposed [46]. It is open the question whether similar improvements can be found for Galois words, by reducing the number of comparison or the amount of auxiliary memory used by algorithm FindGaloisRotation. Moreover, it would be interesting to investigate whether similar strategies can be applied to other generalized $BWT$s.

Acknowledgements

RG and GM are partially supported by INdAM-GNCS project 2019 “Innovative methods for the solution of medical and biological big data” and MIUR-PRIN project “Multicriteria Data Structures and Algorithms: from compressed to learned indexes, and beyond” grant n. 2017WR7SHH.

GR and MS are partially supported by the project MIUR-SIR CMACBioSeq “Combinatorial methods for analysis and compression of biological sequences” grant n. RBSI146R5L.