Synchronization Strings: Codes for Insertions and Deletions Approaching the Singleton Bound

TL;DR

The paper introduces synchronization strings, a novel technique for efficiently handling synchronization errors like insertions and deletions, enabling near-optimal insdel codes that approach the Singleton bound.

Contribution

It presents synchronization strings as a new method to transform any error-correcting code into an efficient insdel code with near-optimal rate and distance.

Findings

Transforms ECCs into insdel codes with minimal alphabet increase

Achieves insdel codes close to the Singleton bound

Provides a simple black-box construction for insdel codes

Abstract

We introduce synchronization strings as a novel way of efficiently dealing with synchronization errors, i.e., insertions and deletions. Synchronization errors are strictly more general and much harder to deal with than commonly considered half-errors, i.e., symbol corruptions and erasures. For every $\epsilon >0$, synchronization strings allow to index a sequence with an $\epsilon^{-O(1)}$ size alphabet such that one can efficiently transform $k$ synchronization errors into $(1+\epsilon)k$ half-errors. This powerful new technique has many applications. In this paper, we focus on designing insdel codes, i.e., error correcting block codes (ECCs) for insertion deletion channels. While ECCs for both half-errors and synchronization errors have been intensely studied, the later has largely resisted progress. Indeed, it took until 1999 for the first insdel codes with constant rate, constant distance, and constant alphabet size to be constructed by Schulman and Zuckerman. Insdel codes for asymptotically large or small noise rates were given in 2016 by Guruswami et al. but these codes are still polynomially far from the optimal rate-distance tradeoff. This makes the understanding of insdel codes up to this work equivalent to what was known for regular ECCs after Forney introduced concatenated codes in his doctoral thesis 50 years ago. A direct application of our synchronization strings based indexing method gives a simple black-box construction which transforms any ECC into an equally efficient insdel code with a slightly larger alphabet size. This instantly transfers much of the highly developed understanding for regular ECCs over large constant alphabets into the realm of insdel codes. Most notably, we obtain efficient insdel codes which get arbitrarily close to the optimal rate-distance tradeoff given by the Singleton bound for the complete noise spectrum.

Figures

Tables

Table 1. Table 1 : Properties and quality of ( n , δ ) 𝑛 𝛿 (n,\delta) -indexing algorithms with S 𝑆 S being a ε 𝜀 \varepsilon -synchronization string

Algorithm	Type	Misdecodings	Error-free	Streaming	Complexity
Section 5.2	ins/del	$\left(2+\epsilon \right)\cdot n\delta$		✓	$O\left(n^4\right)$
Section 6.3	ins/del	$\sqrt{\epsilon }\cdot n$			$O\left(n^2/\sqrt{\epsilon }\right)$
Section 6.4	del	$\epsilon \cdot n\delta$		✓	$O\left(n\right)$
Section 6.5	ins	$\left(1+\epsilon \right)\cdot n\delta$	✓		$O\left(n\right)$
Section 6.5	del	$\epsilon \cdot n\delta$	✓		$O\left(n\right)$
Section 6.6	ins/del	$\left(1+\epsilon \right)\cdot n\delta$		✓	$O\left(n^4\right)$

Equations

$$(\tau_{1}[k],\tau_{2}[k])=\Bigg{\{}\begin{tabular}[]{ll}$(S[i],*)$&if $S[i]$ is deleted\\ $(S[i],S_{\tau}[j])$&if $S[i]$ is delivered as $S_{\tau}[j]$\\ $(*,S_{\tau}[j])$&if $S_{\tau}[j]$ is inserted\end{tabular}$$

$$R_{\mathcal{C}}\cdot \left(1-\frac{\log {\Sigma }_S}{\log {\Sigma }_{\mathcal{C}}}\right)$$

$$RSD(S,S^{\prime })=\underset{k>0}{\max }\frac{ED\left(S(|S|-k,|S|],S^{\prime }(|S^{\prime }|-k,|S^{\prime }|]\right)}{2k}$$

$$ED\left(S[i-k,i),S[i,j)\right)>(1-\epsilon )2k.$$

$$\mathbb{P}\left[A_i\right]\le x_i\prod _{(i,j)\in E}\left(1-x_j\right)$$

$$\mathbb{P}\left[\underset{i=1}{\overset{n}{\bigwedge }}{\bar{A}}_i\right]\ge \prod _{i=1}^n\left(1-x_i\right)>0.$$

$$\underset{I\sim {\Sigma }^l}{\mathrm{Pr}}[I\text{ is bad}]$$

$$Pr\left[\text{Interval }R[p,q)\text{ is bad}\right]\le x_{p,q}\prod _{R[p,q)\cap R[p^{\prime },q^{\prime })\ne \varnothing }(1-x_{p^{\prime },q^{\prime }}).$$

$$C^{\epsilon (p-q)}\le D^{\epsilon (p-q)}\prod _{l=t}^n{\left(1-D^{-\epsilon l}\right)}^{l+(q-p)},$$

$$C^{-l^{\prime }}\le D^{-l^{\prime }}\prod _{l=t}^n{\left(1-D^{-\epsilon l}\right)}^{\frac{l+l^{\prime }}{\epsilon }},$$

$$C\ge \frac{D}{{\prod }_{l=t}^n{\left(1-D^{-\epsilon l}\right)}^{\frac{1+l/l^{\prime }}{\epsilon }}}.$$

$$C\ge \frac{D}{{\prod }_{l=t}^{\infty }{\left(1-D^{-\epsilon l}\right)}^{\frac{l+1}{\epsilon }}}.$$

$$L=\underset{D>1}{\min }\left\{\frac{D}{{\prod }_{l=t}^{\infty }{\left(1-D^{-\epsilon l}\right)}^{\frac{l+1}{\epsilon }}}\right\}.$$

$$C=\frac{\epsilon \sqrt{|\Sigma |}}{e}\ge L,$$

$$\forall l\ge t{D}^{-\epsilon l}\cdot \frac{l+1}{\epsilon }\ll 1.$$

$$\frac{D}{{\prod }_{l=t}^{\infty }{\left(1-D^{-\epsilon l}\right)}^{\frac{l+1}{\epsilon }}}$$

$$<$$

$$=$$

$$\sum _{l=t}^{\infty }(l+1)x^l=\frac{x^t(1+t-tx)}{(1-x{)}^2}<\frac{2t{x}^t}{(1-x{)}^2}.$$

$$\underset{i\ge 1}{\max }\frac{\mathcal{E}\left(|S|-i,|S|\right)}{i}$$

$$RSD(S,S^{\prime })$$

$$=$$

$$\le$$

$$\Phi (i)=\underset{1\le s\le i}{\max }\left\{\frac{t\cdot \mathcal{E}(i-s,i)}{1-\epsilon }-s\right\}$$

$$\Phi (i)$$

$$\Phi (i)=\underset{1\le s\le i}{\max }\left\{\frac{t\cdot \mathcal{E}(i-s,i)}{1-\epsilon }-s\right\}=0$$

$$M=\{(a_1,b_1),\cdots ,(a_m,b_m)\}$$

$$b\le \epsilon (n-g)$$

$$k_1$$

$$k_2\le \frac{\epsilon (|T_2|+|T_2^{\prime }|)}{2}$$

Full text

While ECCs for both half-errors and synchronization errors have been intensely studied, the later has largely resisted progress. As Mitzenmacher puts it in his 2009 survey [22]: “Channels with synchronization errors …are simply not adequately understood by current theory. Given the near-complete knowledge we have for channels with erasures and errors … our lack of understanding about channels with synchronization errors is truly remarkable.” Indeed, it took until 1999 for the first insdel codes with constant rate, constant distance, and constant alphabet size to be constructed and only since 2016 are there constructions of constant rate insdel codes for asymptotically large noise rates. Even in the asymptotically large or small noise regime these codes are polynomially far from the optimal rate-distance tradeoff. This makes the understanding of insdel codes up to this work equivalent to what was known for regular ECCs after Forney introduced concatenated codes in his doctoral thesis 50 years ago.

A straight forward application of our synchronization strings based indexing method gives a simple black-box construction which transforms any ECC into an equally efficient insdel code with only a small increase in the alphabet size. This instantly transfers much of the highly developed understanding for regular ECCs into the realm of insdel codes. Most notably, for the complete noise spectrum we obtain efficient “near-MDS” insdel codes which get arbitrarily close to the optimal rate-distance tradeoff given by the Singleton bound. In particular, for any $\delta\in(0,1)$ and $\varepsilon>0$ we give insdel codes achieving a rate of $1-\delta-\varepsilon$ over a constant size alphabet that efficiently correct a $\delta$ fraction of insertions or deletions.

1 Introduction

Since the fundamental works of Shannon, Hamming, and others the field of coding theory has advanced our understanding of how to efficiently correct symbol corruptions and erasures. The practical and theoretical impact of error correcting codes on technology and engineering as well as mathematics, theoretical computer science, and other fields is hard to overestimate. The problem of coding for timing errors such as closely related insertion and deletion errors, however, while also studied intensely since the 60s, has largely resisted such progress and impact so far. An expert panel [8] in 1963 concluded: “There has been one glaring hole in [Shannon’s] theory; viz., uncertainties in timing, which I will propose to call time noise, have not been encompassed …. Our thesis here today is that the synchronization problem is not a mere engineering detail, but a fundamental communication problem as basic as detection itself!” however as noted in a comprehensive survey [21] in 2010: “Unfortunately, although it has early and often been conjectured that error-correcting codes capable of correcting timing errors could improve the overall performance of communication systems, they are quite challenging to design, which partly explains why a large collection of synchronization techniques not based on coding were developed and implemented over the years.” or as Mitzenmacher puts in his survey [22]: “Channels with synchronization errors, including both insertions and deletions as well as more general timing errors, are simply not adequately understood by current theory. Given the near-complete knowledge we have for channels with erasures and errors …our lack of understanding about channels with synchronization errors is truly remarkable.” We, too, believe that the current lack of good codes and general understanding of how to handle synchronization errors is the reason why systems today still spend significant resources and efforts on keeping very tight controls on synchronization while other noise is handled more efficiently using coding techniques. We are convinced that a better theoretical understanding together with practical code constructions will eventually lead to systems which naturally and more efficiently use coding techniques to address synchronization and noise issues jointly. In addition, we feel that better understanding the combinatorial structure underlying (codes for) insertions and deletions will have impact on other parts of mathematics and theoretical computer science.

In this paper, we introduce synchronization strings, a new combinatorial structure which allows efficient synchronization and indexing of streams under insertions and deletions. Synchronization strings and our indexing abstraction provide a powerful and novel way to deal with synchronization issues. They make progress on the issues raised above and have applications in a large variety of settings and problems. We already found applications to channel simulations, synchronization sequences [21], interactive coding schemes [4, 17, 15, 7, 6, 5], edit distance tree codes [2], and error correcting codes for insertion and deletions and suspect there will be many more. In this paper we focus on the last application, namely, designing efficient error correcting block codes over large alphabets for worst-case insertion-deletion channels.

The knowledge on efficient error correcting block codes for insertions and deletions, also called insdel codes, severely lacks behind what is known for codes for Hamming errors. While Levenshtein [18] introduced and pushed the study of such codes already in the 60s it took until 1999 for Schulman and Zuckerman [25] to construct the first insdel codes with constant rate, constant distance, and constant alphabet size. Very recent work of Guruswami et al. [13, 10] in 2015 and 2016 gave the first constant rate insdel codes for asymptotically large noise rates, via list decoding. These codes are however still polynomially far from optimal in their rate or decodable distance respectively. In particular, they achieve a rate of $\Omega(\varepsilon^{5})$ for a relative distance of $1-\varepsilon$ or a relative distance of $O(\varepsilon^{2})$ for a rate of $1-\varepsilon$, for asymptotically small $\varepsilon>0$ (see Section 1.5 for a more detailed discussion of related work).

This paper essentially closes this line of work by designing efficient “near-MDS” insdel codes which approach the optimal rate-distance trade-off given by the Singleton bound. We prove that for any $0\leq\delta<1$ and any constant $\varepsilon>0$, there is an efficient insdel code over a constant size alphabet with block length $n$ and rate $1-\delta-\varepsilon$ which can be uniquely and efficiently decoded from any $\delta n$ insertions and deletions. The code construction takes polynomial time; and encoding and decoding can be done in linear and quadratic time, respectively. More formally, let us define the edit distance of two given strings as the minimum number of insertions and deletions required to convert one of them to the other one.

Theorem 1.1**.**

For any $\varepsilon>0$ and $\delta\in(0,1)$ there exists an encoding map $E:\Sigma^{k}\rightarrow\Sigma^{n}$ and a decoding map $D:\Sigma^{*}\rightarrow\Sigma^{k}$ such that if $EditDistance(E(m),x)\leq\delta n$ then $D(x)=m$. Further $\frac{k}{n}>1-\delta-\varepsilon$, $|\Sigma|=f(\varepsilon)$, and $E$ and $D$ are explicit and can be computed in linear and quadratic time in $n$.

We obtain this code via a black-box construction which transforms any ECC into an equally efficient insdel code with only a small increase in the alphabet size. This transformation, which is a straight forward application of our new synchronization strings based indexing method, is so simple that it can be summarized in one sentence:

For any efficient length $n$ ECC with alphabet bit size $\frac{\log\varepsilon^{-1}}{\varepsilon}$, attaching to every codeword, symbol by symbol, a random or suitable pseudorandom string over an alphabet of bit size $\log\varepsilon^{-1}$ results in an efficient insdel code with a rate and decodable distance that changed by at most $\varepsilon$.

Far beyond just implying Theorem 1.1, this allows to instantly transfer much of the highly developed understanding for regular ECCs into the realm of insdel codes.

Theorem 1.1 is obtained by using the “near-MDS” expander codes of Guruswami and Indyk [9] as a base ECC. These codes generalize the linear time codes of Spielman [27] and can be encoded and decoded in linear time. Our simple encoding strategy, as outlined above, introduces essentially no additional computational complexity during encoding. Our quadratic time decoding algorithm, however, is slower than the linear time decoding of the base codes from [9] but still pretty fast. In particular, a quadratic time decoding for an insdel code is generally very good given that, in contrast to Hamming codes, even computing the distance between the received and the sent/decoded string is an edit distance computation. Edit distance computations in general do usually not run in sub-quadratic time, which is not surprising given the recent SETH-conditional lower bounds [1]. For the settings of for insertion-only and deletion-only errors we furthermore achieve analogs of Theorem 1.1 with linear decoding complexities.

In terms of the dependence of the alphabet bit size on the parameter $\varepsilon$, which characterizes how close a code is to achieving an optimal rate/distance pair summing to one, our transformation seem to inherently produce an alphabet bit size that is near linear in $\frac{1}{\varepsilon}$. However, the same is true for the state of the art linear-time base ECCs [9] which have an alphabet bit size of $\Theta(\frac{1}{\varepsilon^{2}})$. Existentially it is known that an alphabet bit size logarithmic in $\frac{1}{\varepsilon}$ is necessary and sufficient and ECCs based on algebraic geometry [29] achieving such a bound up to constants are known, but their encoding and decoding complexities are higher.

1.1 High-level Overview, Intuition and Overall Organization

While extremely powerful, the concept and idea behind synchronization strings is easily demonstrated. In this section, we explain the high-level approach taken and provide intuition for the formal definitions and proofs to follow. This section also explains the overall organization of the rest of the paper.

1.1.1 Synchronization Errors and Half-Errors

Consider a stream of symbols over a large but constant size alphabet $\Sigma$ in which some constant fraction $\delta$ of symbols is corrupted.

There are two basic types of corruptions we will consider, half-errors and synchronization errors. Half-errors consist of erasures, that is, a symbol being replaced with a special “?” symbol indicating the erasure, and symbol corruptions in which a symbol is replaced with any other symbol in $\Sigma$. The wording half-error comes from the realization that when it comes to code distances erasures are half as bad as symbol corruptions. An erasure is thus counted as one half-error while a symbol corruption counts as two half-errors (see Section 2 for more details). Synchronization errors consist of deletions, that is, a symbol being removed without replacement, and insertions, where a new symbol from $\Sigma$ is added anywhere.

It is clear that synchronization errors are strictly more general and harsher than half-errors. In particular, any symbol corruption, worth two half-errors, can also be achieved via a deletion followed by an insertion. Any erasure can furthermore be interpreted as a deletion together with the often very helpful extra information where this deletion took place. This makes synchronization errors at least as hard as half-errors. The real problem that synchronization errors bring with them however is that they cause sending and receiving parties to become “out of synch”. This easily changes how received symbols are interpreted and makes designing codes or other systems tolerant to synchronization errors an inherently difficult and significantly less well understood problem.

1.1.2 Indexing and Synchronization Strings: Reducing Synchronization Errors to Half-Errors

There is a simple folklore strategy, which we call indexing, that avoids these synchronization problems: Simply enhance any element with a time stamp or element count. More precisely, consecutively number the elements and attach this position count or index to each stream element. Now, if we deal with only deletions it is clear that the position of any deletion is easily identified via a missing index, thus transforming it into an erasure. Insertions can be handled similarly by treating any stream index which is received more than once as erased. If both insertions and deletions are allowed one might still have elements with a spoofed or incorrectly received index position caused by a deletion of an indexed symbol which is then replaced by a different symbol with the same index. This however requires two insdel errors. Generally this trivial indexing strategy can seen to successfully transform any $k$ synchronization errors into at most $k$ half-errors.

In many applications, however, this trivial indexing cannot be used, because having to attach a $\log n$ bit222Throughout this paper all logarithms are binary. long index description to each element of an $n$ long stream is prohibitively costly. Consider for example an error correcting code of constant rate $R$ over some potentially large but nonetheless constant size alphabet $\Sigma$, which encodes $\frac{Rn}{\log|\Sigma|}$ bits into $n$ symbols from $\Sigma$. Increasing $\Sigma$ by a factor of $n$ to allow each symbol to carry its $\log n$ bit index would destroy the desirable property of having an alphabet which is independent from the block length $n$ and would furthermore reduce the rate of the code from $R$ to $\Theta(\frac{R}{\log n})$, which approaches zero for large block lengths. For streams of unknown or infinite length such problems become even more pronounced.

This is where synchronization strings come to the rescue. Essentially, synchronization strings allow to index every element in an infinite stream using only a constant size alphabet while achieving an arbitrarily good approximate reduction from synchronization errors to half-errors. In particular, using synchronization strings $\bm{k}$** synchronization errors can be transformed into at most $\bm{(1+\varepsilon)k}$ half-errors using an alphabet of size independent of the stream length** and in fact only polynomial in $\frac{1}{\varepsilon}$. Moreover, these synchronization strings have simple constructions and fast and easy decoding procedures.

Attaching our synchronization strings to the codewords of any efficient error correcting code, which efficiently tolerates the usual symbol corruptions and erasures, transforms any such code into an efficiently decodable insdel code while only requiring a negligible increasing in the alphabet size. This allows to use the decades of intense research in coding theory for Hamming-type errors to be transferred into the much harder and less well understood insertion-deletion setting.

1.2 Synchronization Strings: Definition, Construction, and Decoding

Next, we want to briefly motivate and explain how we arrive at a natural definition of these magical indexing sequences $S$ over a finite alphabet $\Sigma$ and what intuition lies behind their efficient constructions and decoding procedures.

Suppose a sender has attached some indexing sequence $S$ one-by-one to each element in a stream and consider a time $t$ at which a receiver has received a corrupted sequence of the first $t$ index descriptors, i.e., a corrupted version of the length $t$ prefix of $S$. When the receiver tries to guess or decode the current index it should naturally consider all indexing symbols received so far and find the “best” prefix of $S$. This suggests that the prefix of length $l$ of a synchronization string $S$ acts as a codeword for the index position $l$ and that one should think of the set of prefixes of $S$ as a code associated with the synchronization string $S$. Naturally one would want such a code to have good distance properties between any two codewords under some distance measure. While edit distance, i.e., the number of insertions and deletions needed to transform one string into another seems like the right notion of distance for insdel errors in general, the prefix nature of the codes under consideration will guarantee that codewords for indices $l$ and $l^{\prime}>l$ will have edit distance exactly $l^{\prime}-l$. This implies that even two very long codewords only have a tiny edit distance. On the one hand, this precludes synchronization codes with a large relative edit distance between its codewords. On the other hand, one should see this phenomenon as simply capturing the fact that at any time a simple insertion of an incorrect symbol carrying the correct next indexing symbol will lead to an unavoidable decoding error. Given this natural and unavoidable sensitivity of synchronization codes to recent corruptions, it makes sense to instead use a distance measure which captures the recent density of errors. In this spirit, we suggest the definition of a, to our knowledge, new string distance measure which we call relative suffix distance, which intuitively measures the worst fraction of insdel errors to transform suffixes, i.e., recently sent parts of two strings, into each other. This natural measure, in contrast to a similar measure defined in [2], turns out to induce a metric space on any set of strings.

With this natural definitions for an induced set of codewords and a natural distance metric associated with any such set the next task is to design a string $S$ for which the set of codewords has as large of a minimum pairwise distance as possible. When looking for (infinite) sequences that induce such a set of codewords and thus can be successfully used as synchronization strings it became apparent that one is looking for highly irregular and non-self-similar strings over a fixed alphabet $\Sigma$. It turns out that the correct definition to capture these desired properties, which we call $\varepsilon$-synchronization property, states that any two neighboring intervals of $S$ with total length $l$ should require at least $(1-\varepsilon)l$ insertions and deletions to transform one into the other, where $\varepsilon\geq 0$. A one line calculation also shows that this clean property also implies a large minimum relative suffix distance between any two codewords. Not surprisingly, random strings essentially satisfy this $\varepsilon$-synchronization property, except for local imperfections of self-similarity, such as, symbols repeated twice in a row, which would naturally occur in random sequences about every $|\Sigma|$ positions. This allows us to use the probabilistic method and the general Lovász Local Lemma to prove the existence $\varepsilon$-synchronization strings. This also leads to an efficient randomized construction.

Finally, decoding any string to the closest codeword, i.e., the prefix of the synchronization string $S$ with the smallest relative suffix distance, can be easily done in polynomial time because the set of synchronization codewords is linear and not exponential in $n$ and (edit) distance computations (to each codeword individually) can be done via the classical Wagner-Fischer dynamic programming approach.

1.3 More Sophisticated Decoding Procedures

All this provides an indexing solution which transforms any $k$ synchronization errors into at most $(5+\varepsilon)k$ half-errors. This already leads to insdel codes which achieve a rate approaching $1-5\delta$ for any $\delta$ fraction of insdel errors with $\delta<\frac{1}{5}$. While this is already a drastic improvement over the previously best $1-O(\sqrt{\delta})$ rate codes from [10], which worked only for sufficiently small $\delta$, it is a far less strong result than the near-MDS codes we promised in Theorem 1.1 for every $\delta\in(0,1)$.

We were able to improve upon the above strategy slightly by considering an alternative to the relative suffix distance measure, which we call relative suffix pseudo distance RSPD. RSPD was introduced in [2] and while neither being symmetric nor satisfying the triangle inequality, can act as a pseudo distance in the minimum-distance decoder. For any set of $k=k_{i}+k_{d}$ insdel errors consisting of $k_{i}$ insertions and $k_{d}$ deletions this improved indexing solution leads to at most $(1+\varepsilon)(3k_{i}+k_{d})$ half-errors which already implies near-MDS codes for deletion-only channels but still falls short for general insdel errors. We leave open the question whether an improved pseudo distance definition can achieve an indexing solution with negligible number of misdecodings for a minimum-distance decoder.

In order to achieve our main theorem we developed an different strategy. Fortunately, it turned out that achieving a better indexing solution and the desired insdel codes does not require any changes to the definition of synchronization codes, the indexing approach itself, or the encoding scheme but solely required a very different decoding strategy. In particular, instead of decoding indices in a streaming manner we consider more global decoding algorithms. We provide several such decoding algorithms in Section 6. In particular, we give a simple global decoding algorithm which for which the number of misdecodings goes to zero as the quality $\varepsilon$ of the $\varepsilon$-synchronization string used goes to zero, irrespectively of how many insdel errors are applied.

Our global decoding algorithms crucially build on another key-property which we prove holds for any $\varepsilon$-synchronization string $S$, namely that there is no monotone matching between $S$ and itself which mismatches more than a $\varepsilon$ fraction of indices. Besides being used in our proofs, considering this $\varepsilon$-self-matching property has another advantage. We show that this property is achieved easier than the full $\varepsilon$-synchronization property and that indeed a random string satisfies it with good probability. This means that, in the context of error correcting codes, one can even use a simple uniformly random string as a “synchronization string”. Lastly, we show that even a $n^{-O(1)}$-approximate $O\left(\frac{\log n}{\log\frac{1}{\varepsilon}}\right)$-wise independent random strings satisfy the desired $\varepsilon$-self-matching property which, using the celebrated small sample space constructions from [24] also leads to a deterministic polynomial time construction.

Lastly, we provide simpler and faster global decoding algorithms for the setting of deletion-only and insertion-only corruptions. These algorithms are essentially greedy algorithms which run in linear time. They furthermore guarantee that their indexing decoding is error-free, i.e., they only output “I don’t know” for some indices but never produce an incorrectly decoded index. Such decoding schemes have the advantage that one can use them in conjunction with error correcting codes that efficiently recover from erasures (and not necessarily also symbol corruptions).

1.4 Organization of this Paper

The organization of this paper closely follows the flow of the high-level description above.

We start by giving more details on related work in Section 1.5 and introduce notation used in the paper in Section 2 together with a formal introduction of the two different error types as well as (efficient) error correcting codes and insdel codes. In Section 3, we formalize the indexing problem and (approximate) solutions to it. Section 4 shows how any solution to the indexing problem can be used to transform any regular error correcting codes into an insdel code. Section 5 introduces the relative suffix distance and $\varepsilon$-synchronization strings, proves the existence of $\varepsilon$-synchronization strings and provides an efficient construction. Section 5.2 shows that the minimum suffix distance decoder is efficient and leads to a good indexing solution. We elaborate on the connection between $\varepsilon$-synchronization strings and the $\varepsilon$-self-matching property in Section 6.1 and provide our improved decoding algorithms in the remainder of Section 6.

1.5 Related Work

Shannon was the first to systematically study reliable communication. He introduced random error channels, defined information quantities, and gave probabilistic existence proofs of good codes. Hamming was the first to look at worst-case errors and code distances as introduced above. Simple counting arguments on the volume of balls around codewords given in the 50’s by Hamming and Gilbert-Varshamov produce simple bounds on the rate of $q$-ary codes with relative distance $\delta$. In particular, they show the existence of codes with relative distance $\delta$ and rate at least $1-H_{q}(\delta)$ where $H_{q}(x)=x\log(q-1)-\frac{x\log x-(1-x)\log(1-x)}{\log q}$ is the $q$-ary entropy function. This means that for any $\delta<1$ and $q=\omega(1/\delta)$ there exists codes with distance $\delta$ and rate approaching $1-\delta$. Concatenated codes and the generalized minimum distance decoding procedure introduced by Forney in 1966 led to the first codes which could recover from constant error fractions $\delta\in(0,1)$ while having polynomial time encoding and decoding procedures. The rate achieved by concatenated codes for large alphabets with sufficiently small distance $\delta$ comes out to be $1-O(\sqrt{\delta})$. On the other hand, for $\delta$ sufficiently close to one, one can achieve a constant rate of $O(\delta^{2})$. Algebraic geometry codes suggested by Goppa in 1975 later lead to error correcting codes which for every $\varepsilon>0$ achieve the optimal rate of $1-\delta-\varepsilon$ with an alphabet size polynomial in $\varepsilon$ while being able to efficiently correct for a $\delta$ fraction of half-errors [29].

While this answered the most basic questions, research since then has developed a tremendously powerful toolbox and selection of explicit codes. It attests to the importance of error correcting codes that over the last several decades this research direction has developed into the incredibly active field of coding theory with hundreds of researchers studying and developing better codes. A small and highly incomplete subset of important innovations include rateless codes, such as, LT codes [20], which do not require to fix a desired distance at the time of encoding, explicit expander codes [27, 9] which allow linear time encoding and decoding, polar codes [14, 12] which can approach Shannon’s capacity polynomially fast, network codes [19] which allow intermediate nodes in a network to recombine codewords, and efficiently list decodable codes [11] which allow to list-decode codes of relative distance $\delta$ up to a fraction of about $\delta$ symbol corruptions.

While error correcting codes for insertions and deletions have also been intensely studied, our understanding of them is much less well developed. We refer to the 2002 survey by Sloan [26] on single-deletion codes, the 2009 survey by Mitzenmacher [22] on codes for random deletions and the most general 2010 survey by Mercier et al. [21] for the extensive work done around codes for synchronization errors and only mention the results most closely related to Theorem 1.1 here: Insdel codes were first considered by Levenshtein [18] and since then many bounds and constructions for such codes have been given. However, while essentially the same volume and sphere packing arguments as for regular codes show that there exists insdel codes capable of correcting a fraction $\delta$ of insdel erros with rate $1-\delta$, no efficient constructions anywhere close to this rate-distance tradeoff are known. Even the construction of efficient insdel codes over a constant alphabet with any (tiny) constant relative distance and any (tiny) constant rate had to wait until Schulman and Zuckerman gave the first such code in 1999 [25]. Over the last two years Guruswami et al. provided new codes improving over this state of the art the asymptotically small or large noise regime by giving the first codes which achieve a constant rate for noise rates going to one and codes which provide a rate going to one for an asymptotically small noise rate. In particular, [13] gave the first efficient codes codes over fixed alphabets to correct a deletion fraction approaching $1$, as well as efficient binary codes to correct a small constant fraction of deletions with rate approaching $1$. These codes could, however, only be efficiently decoded for deletions and not insertions. A follow-up work gave new and improved codes with similar rate-distance tradeoffs which can be efficiently decoded from insertions and deletions [10]. In particular, these codes achieve a rate of $\Omega(\delta^{5})$ and $1-\tilde{O}(\sqrt{\delta})$ while being able to efficiently recover from a $\delta$ fraction of insertions and deletions. These works put the current state of the art for error correcting codes for insertions and deletions pretty much equal to what was known for regular error correcting codes 50 years ago, after Forney’s 1965 doctoral thesis.

2 Definitions and Preliminaries

In this section, we provide the notation and definitions we will use throughout the rest of the paper.

2.1 String Notation and Edit Distance

String Notation. For two strings $S\in\Sigma^{n}$ and $S^{\prime}\in\Sigma^{n^{\prime}}$ be two strings over alphabet $\Sigma$. We define $S\cdot S^{\prime}\in\Sigma^{n+n^{\prime}}$ to be their concatenation. For any positive integer $k$ we define $S^{k}$ to equal $k$ copies of $S$ concatenated together. For $i,j\in\{1,\dots,n\}$, we denote the substring of $S$ from the $i^{th}$ index through and including the $j^{th}$ index as $S[i,j]$. Such a consecutive substring is also called a factor of $S$. For $i<1$ we define $S[i,j]=\bot^{-i+1}\cdot S[1,j]$ where $\bot$ is a special symbol not contained in $\Sigma$. We refer to the substring from the $i^{th}$ index through, but not including, the $j^{th}$ index as $S[i,j)$. The substrings $S(i,j]$ and $S[i,j]$ are similarly defined. Finally, $S[i]$ denotes the $i^{th}$ symbol of $S$ and $|S|=n$ is the length of $S$. Occasionally, the alphabets we use are the cross-product of several alphabets, i.e. $\Sigma=\Sigma_{1}\times\cdots\times\Sigma_{n}$. If $T$ is a string over $\Sigma,$ then we write $T[i]=\left[a_{1},\dots,a_{n}\right]$, where $a_{i}\in\Sigma_{i}$.

Edit Distance. Throughout this work, we rely on the well-known edit distance metric defined as follows.

Definition 2.1** (Edit distance).**

The edit distance $ED(c,c^{\prime})$ between two strings $c,c^{\prime}\in\Sigma^{*}$ is the minimum number of insertions and deletions required to transform $c$ into $c^{\prime}$.

It is easy to see that edit distance is a metric on any set of strings and in particular is symmetric and satisfies the triangle inequality property. Furthermore, $ED\left(c,c^{\prime}\right)=|c|+|c^{\prime}|-2\cdot LCS\left(c,c^{\prime}\right)$, where $LCS\left(c,c^{\prime}\right)$ is the longest common substring of $c$ and $c^{\prime}$.

We also use some string matching notation from  [2]:

Definition 2.2** (String matching).**

Suppose that $c$ and $c^{\prime}$ are two strings in $\Sigma^{*}$, and suppose that $*$ is a symbol not in $\Sigma$. Next, suppose that there exist two strings $\tau_{1}$ and $\tau_{2}$ in $\left(\Sigma\cup\{*\}\right)^{*}$ such that $|\tau_{1}|=|\tau_{2}|$, $del\left(\tau_{1}\right)=c$, $del(\tau_{2})=c^{\prime}$, and $\tau_{1}[i]\approx\tau_{2}[i]$ for all $i\in\left\{1,\dots,|\tau_{1}|\right\}$. Here, $del$ is a function that deletes every $*$ in the input string and $a\approx b$ if $a=b$ or one of $a$ or $b$ is $*$. Then we say that $\tau=\left(\tau_{1},\tau_{2}\right)$ is a string matching between $c$ and $c^{\prime}$ (denoted $\tau:c\to c^{\prime}$). We furthermore denote with $sc\left(\tau_{i}\right)$ the number of $*$’s in $\tau_{i}$.

Note that the edit distance $ED(c,c^{\prime})$ between strings $c,c,\in\Sigma^{*}$ is exactly equal to $\min_{\tau:c\to c^{\prime}}\left\{sc\left(\tau_{1}\right)+sc\left(\tau_{2}\right)\right\}$.

2.2 Error Correcting Codes

Next we give a quick summary of the standard definitions and formalism around error correcting codes. This is mainly for completeness and we remark that readers already familiar with basic notions of error correcting codes might want to skip this part.

Codes, Distance, Rate, and Half-Errors

An error correcting code $C$ is an injective function which takes an input string $s\in({\Sigma^{\prime}})^{n^{\prime}}$ over alphabet ${\Sigma^{\prime}}$ of length $n^{\prime}$ and generates a codeword $C(s)\in{\Sigma}^{n}$ of length $n$ over alphabet ${\Sigma}$. The length $n$ of a codeword is also called the block length. The two most important parameters of a code are its distance $\Delta$ and its rate $R$. The rate $R=\frac{n\log|{\Sigma}|}{n^{\prime}\log|{\Sigma^{\prime}}|}$ measures what fraction of bits in the codewords produced by $C$ carries non-redundant information about the input. The code distance $\Delta(C)=\min_{s,s^{\prime}}\Delta(C(s),C(s^{\prime}))$ is simply the minimum Hamming distance between any two codewords. The relative distance $\delta(C)=\frac{\Delta(C)}{n}$ measures what fraction of output symbols need to be corrupted to transform one codeword into another.

It is easy to see that if a sender sends out a codeword $C(s)$ of code $C$ with relative distance $\delta$ a receiver can uniquely recover $s$ if she receives a codeword in which less than a $\delta$ fraction of symbols are affected by an erasure, i.e., replaced by a special “$?$” symbol. Similarly, a receiver can uniquely recover the input $s$ if less than $\delta/2$ symbol corruptions, in which a symbol is replaced by any other symbol from ${\Sigma}$, occurred. More generally it is easy to see that a receiver can recover from any combination of $k_{e}$ erasures and $k_{c}$ corruptions as long as $k_{e}+2k_{c}<\delta n$. This motivates defining half-errors to incorporate both erasures and symbol corruptions where an erasure is counted as a single half-error and a symbol corruption is counted as two half-errors. In summary, any code of distance $\delta$ can tolerate any error pattern of less than $\delta n$ half-errors.

We remark that in addition to studying codes with decoding guarantees for worst-case error pattern as above one can also look at more benign error models which assume a distribution over error patterns, such as errors occurring independently at random. In such a setting one looks for codes which allow unique recovery for typical error patterns, i.e., one wants to recover the input with probability tending to $1$ rapidly as the block length $n$ grows. While synchronization strings might have applications for such codes as well, this paper focuses exclusively on codes with good distance guarantees which tolerate an arbitrary (worst-case) error pattern.

Synchronization Errors

In addition to half-errors, we study synchronization errors which consist of deletions, that is, a symbol being removed without replacement, and insertions, where a new symbol from ${\Sigma}$ is added anywhere. It is clear that synchronization errors are strictly more general and harsh than half-errors (see Section 1.1.1). The above formalism of codes, rate, and distance works equally well for synchronization errors if one replaces the Hamming distance with edit distance. Instead of measuring the number of symbol corruptions required to transform one string into another, edit distance measures the minimum number of insertions and deletions to do so. An insertion-deletion error correcting code, or insdel code for short, of relative distance $\delta$ is a set of codewords for which at least $\delta n$ insertions and deletions are needed to transformed any codeword into another. Such a code can correct any combination of less than $\delta n/2$ insertions and deletions. We remark that it is possible for two codewords of length $n$ to have edit distance up to $2n$ putting the (minimum) relative edit distance between zero and two and allowing for constant rate codes which can tolerate $(1-\varepsilon)n$ insdel errors.

Efficient Codes

In addition to codes with a good minimum distance, one furthermore wants efficient algorithms for the encoding and error-correction tasks associated with the code. Throughout this paper we say a code is efficient if it has encoding and decoding algorithms running in time polynomial in the block length. While it is often not hard to show that random codes exhibit a good rate and distance, designing codes which can be decoded efficiently is much harder. We remark that most codes which can efficiently correct for symbol corruptions are also efficient for half-errors. For insdel codes the situation is slightly different. While it remains true that any code that can uniquely be decoded from any $\delta(C)$ fraction of deletions can also be decoded from the same fraction of insertions and deletions [18] doing so efficiently is often much easier for the deletion-only setting than the fully general insdel setting.

.

3 The Indexing Problem

In this section, we formally define the indexing problem. In a nutshell, this problem is that of sending a suitably chosen string $S$ of length $n$ over an insertion-deletion channel such that the receiver will be able to figure out the indices of most of the symbols he receives correctly. This problem can be trivially solved by sending the string $S=1,2,\ldots,n$ over the alphabet $\Sigma=\{1,\ldots,n\}$ of size $n$. Interesting solution to the indexing problem, however, do almost as well while using a finite size alphabet. While very intuitive and simple, the formalization of this problem and its solutions enables an easy use in many applications.

To set up an $(n,\delta)$-indexing problem, we fix $n$, i.e., the number of symbols which are being sent, and the maximum fraction $\delta$ of symbols that can be inserted or deleted. We further call the string $S$ the synchronization string. Lastly, we describe the influences of the $n\delta$ worst-case insertions and deletions which transform $S$ into the related string $S_{\tau}$ in terms of a string matching $\tau$. In particular, $\tau=(\tau_{1},\tau_{2})$ is the string matching from $S$ to $S_{\tau}$ such that $del(\tau_{1})=S$, $del(\tau_{2})=S_{\tau}$, and for every $k$

[TABLE]

where $i=|del(\tau_{1}[1,k])|$ and $j=|del(\tau_{2}[1,k])|$.

Definition 3.1** ($(n,\delta)$-Indexing Algorithm).**

The pair $(S,\mathcal{D}_{S})$ consisting of a synchronization string $S\in\Sigma^{n}$ and an algorithm $\mathcal{D}_{S}$ is called a $(n,\delta)$-indexing algorithm over alphabet $\Sigma$ if for any set of $n\delta$ insertions and deletions represented by $\tau$ which alter $S$ to a string $S_{\tau}$, the algorithm $\mathcal{D}_{S}(S_{\tau})$ outputs either $\bot$ or an index between $1$ and $n$ for every symbol in $S_{\tau}$.

The $\bot$ symbol here represents an “I don’t know” response of the algorithm while an index $j$ output by $\mathcal{D}_{S}(S_{\tau})$ for the $i^{th}$ symbol of $S_{\tau}$ should be interpreted as the $(n,\delta)$-indexing algorithm guessing that this was the $j^{th}$ symbol of $S$. One seeks algorithms that decode as many indices as possible correctly. Naturally, one can only correctly decode indices that were correctly transmitted. Next we give formal definitions of both notions:

Definition 3.2** (Correctly Decoded Index).**

An $(n,\delta)$ indexing algorithm $(S,\mathcal{D}_{S})$ decodes index $j$ correctly under $\tau$ if $\mathcal{D}_{S}(S_{\tau})$ outputs $i$ and there exists a $k$ such that $i=|del(\tau_{1}[1,k])|,~{}j=|del(\tau_{2}[1,k])|,~{}\tau_{1}[k]=S[i],~{}\tau_{2}[k]=S_{\tau}[j]$

We remark that this definition counts any $\bot$ response as an incorrect decoding.

Definition 3.3** (Successfully Transmitted Symbol).**

For string $S_{\tau}$, which was derived from a synchronization string $S$ via $\tau=(\tau_{1},\tau_{2})$, we call the $j^{th}$ symbol $S_{\tau}[j]$ successfully transmitted if it stems from a symbol coming from $S$, i.e., if there exists a $k$ such that $|del(\tau_{2}[1,k])|=j$ and $\tau_{1}[k]=\tau_{2}[k]$.

We now define the quality of an $(n,\delta)$-indexing algorithm by counting the maximum number of misdecoded indices among those that were successfully transmitted. Note that the trivial indexing strategy with $S=1,\ldots,n$ which outputs for each symbol the symbol itself has no misdecodings. One can therefore also interpret our quality definition as capturing how far from this ideal solution an algorithm is (stemming likely due to the smaller alphabet which is used for $S$).

Definition 3.4** (Misdecodings of an $(n,\delta)$-Indexing Algorithm).**

Let $(S,\mathcal{D}_{S})$ be an $(n,\delta)$-indexing algorithm. We say this algorithm has at most $k$ misdecodings if for any $\tau$ corresponding to at most $n\delta$ insertions and deletions the number of correctly transmitted indices that are incorrectly decoded is at most $k$.

Now, we introduce two further useful properties that a $(n,\delta)$-indexing algorithm might have.

Definition 3.5** (Error-free Solution).**

We call $(S,\mathcal{D}_{S})$ an error-free $(n,\delta)$-indexing algorithm with respect to a set of deletion or insertion patterns if every index output is either $\bot$ or correctly decoded. In particular, the algorithm never outputs an incorrect index, even for indices which are not correctly transmitted.

It is noteworthy that error-free solutions are essentially only obtainable when dealing with the insertion-only or deletion-only setting. In both cases, the trivial solution with $S=1,\cdots,n$ which decodes any index that was received exactly once is error-free. We later give some algorithms which preserve this nice property, even over a smaller alphabet, and show how error-freeness can be useful in the context of error correcting codes.

Lastly, another very useful property of some $(n,\delta)$-indexing algorithms is that their decoding process operates in a streaming manner, i.e, the decoding algorithm decides the index output for $S_{\tau}[j]$ independently of $S_{\tau}[j^{\prime}]$ where $j^{\prime}>j$. While this property is not particularly useful for the error correcting block code application put forward in this paper, it is an extremely important and strong property which is crucial in several applications we know of, such as, rateless error correcting codes, channel simulations, interactive coding, edit distance tree codes, and other settings.

Definition 3.6** (Streaming Solutions).**

We call $(S,\mathcal{D}_{S})$ a streaming solution if the decoded index for the $i$th element of the received string $S_{\tau}$ only depends on $S_{\tau}[1,i]$.

Again, the trivial solution for $(n,\delta)$-index decoding problem over an alphabet of size $n$ with zero misdecodings can be made streaming by outputting for every received symbols the received symbol itself as an index. This solution is also error-free for the deletion-only setting but not error-free for the insertion-only setting. In fact, it is easy to show that an algorithm cannot be both streaming and error-free in any setting which allows insertions.

Overall, the important characteristics of an $(n,\delta)$-indexing algorithm are (a) its alphabet size $|\Sigma|$, (b) the bound on the number of misdecodings, (c) the complexity of the decoding algorithm $\mathcal{D}$, (d) the preprocessing complexity of constructing the string $S$, (e) whether the algorithm works for the insertion-only, the deletion-only or the full insdel setting, and (f) whether the algorithm satisfies the streaming or error-freeness property. Table 1 gives a summary over the different solutions for the $(n,\delta)$-indexing problem we give in this paper.

4 Insdel Codes via Indexing Algorithms

Next, we show how a good $(n,\delta)$-indexing algorithms $(S,\mathcal{D}_{S})$ over alphabet $\Sigma_{S}$ allows one to transform any regular ECC $\mathcal{C}$ with block length $n$ over alphabet $\Sigma_{\mathcal{C}}$ which can efficiently correct half-errors, i.e., symbol corruptions and erasures, into a good insdel code over alphabet $\Sigma=\Sigma_{\mathcal{C}}\times\Sigma_{S}$.

To this end, we simply attach $S$ symbol-by-symbol to every codeword of $\mathcal{C}$. On the decoding end, we first decode the indices of the symbols arrived using the indexing part of each received symbol and then interpret the message parts as if they have arrived in the decoded order. Indices where zero or multiple symbols are received get considered as erased. We will refer to this procedure as the indexing procedure. Finally, the decoding algorithm $\mathcal{D}_{\mathcal{C}}$ for $\mathcal{C}$ is used. These two straight forward algorithms are formally described as Algorithm 1 and Algorithm 2.

Theorem 4.1**.**

If $(S,\mathcal{D}_{S})$ guarantees $k$ misdecodings for the $(n,\delta)$-index problem, then the indexing procedure recovers the codeword sent up to $n\delta+2k$ half-errors, i.e., half-error distance of the sent codeword and the one recovered by the indexing procedure is at most $n\delta+2k$. If $(S,\mathcal{D}_{S})$ is error-free, the indexing procedure recovers the codeword sent up to $n\delta+k$ half-errors.

Proof.

Consider a set insertions and deletions described by $\tau$ consisting of $D_{\tau}$ deletions and $I_{\tau}$ insertions. Note that among $n$ encoded symbols, at most $D_{\tau}$ were deleted and less than $k$ of are decoded incorrectly. Therefore, at least $n-D_{\tau}-k$ indices are decoded correctly. On the other hand at most $D_{\tau}+k$ of the symbols sent are not decoded correctly. Therefore, if the output only consisted of correctly decoded indices for successfully transmitted symbols, the output would have contained up to $D_{\tau}+k$ erasures and no symbol corruption, resulting into a total of $D_{\tau}+k$ half-errors. However, any symbol which is being incorrectly decoded or inserted may cause a correctly decoded index to become an erasure by making it appear multiple times or change one of original $I_{\tau}+k$ erasures into a corruption error by making the indexing procedure mistakenly decode an index. Overall, this can increase the number of half-errors by at most $I_{\tau}+k$ for a total of at most $D_{\tau}+k+I_{\tau}+k=D_{\tau}+I_{\tau}+2k=n\delta+2k$ half-errors. For error-free indexing algorithms, any misdecoding does not result in an incorrect index and the number of incorrect indices is $I_{\tau}$ instead of $I_{\tau}+k$ leading to the reduced number of half-errors in this case. ∎

This makes it clear that applying an ECC $\mathcal{C}$ which is resilient to $n\delta+2k$ half-errors enables the receiver side to fully recover $m$.

Next, we formally state how a good $(n,\delta)$-indexing algorithm $(S,\mathcal{D}_{S})$ over alphabet $\Sigma_{S}$ allows one to transform any regular ECC $\mathcal{C}$ with block length $n$ over alphabet $\Sigma_{\mathcal{C}}$ which can efficiently correct half-errors, i.e., symbol corruptions and erasures, into a good insdel code over alphabet $\Sigma=\Sigma_{\mathcal{C}}\times\Sigma_{S}$. The following Theorem is a corollary of Theorem 4.1 and the definition of the indexing procedure:

Theorem 4.2**.**

Given an (efficient) $(n,\delta)$-indexing algorithm $(S,\mathcal{D}_{S})$ over alphabet $\Sigma_{S}$ with at most $k$ misdecodings, and decoding complexity $T_{\mathcal{D}_{S}}(n)$ and an (efficient) ECC $\mathcal{C}$ over alphabet $\Sigma_{\mathcal{C}}$ with rate $R_{\mathcal{C}}$, encoding complexity $T_{\mathcal{E}_{\mathcal{C}}}$, and decoding complexity $T_{\mathcal{D}_{\mathcal{C}}}$ that corrects up to $n\delta+2k$ half-errors, one obtains an insdel code that can be (efficiently) decoded from up to $n\delta$ insertions and deletions. The rate of this code is

[TABLE]

*The encoding complexity remains $T_{\mathcal{E}_{\mathcal{C}}}$, the decoding complexity is $T_{\mathcal{D}_{\mathcal{C}}}+T_{\mathcal{D}_{S}}(n)$ and the preprocessing complexity of constructing the code is the complexity of constructing $\mathcal{C}$ and $S$.

Furthermore, if $(S,\mathcal{D}_{S})$ is error-free, then choosing a $\mathcal{C}$ which can recover only from $n\delta+k$ erasures is sufficient to produce the same quality code.*

Note that if one chooses $\Sigma_{\mathcal{C}}$ such that $\frac{\log\Sigma_{S}}{\log\Sigma_{\mathcal{C}}}=o(\delta)$, the rate loss due to the attached symbols will be negligible. With all this in place one can obtain Theorem 1.1 as a consequence of Theorem 4.2.

Proof of Theorem 1.1.

Given the $\delta$ and $\varepsilon$ from the statement of Theorem 1.1 we choose $\varepsilon^{\prime}=O\left(\left(\frac{\varepsilon}{6}\right)^{2}\right)$ and use Theorem 6.13 to construct a string $S$ of length $n$ over alphabet $\Sigma_{S}$ of size $\varepsilon^{-O(1)}$ with the $\varepsilon^{\prime}$-self-matching property. We then use the $(n,\delta)$-indexing algorithm $(S,\mathcal{D}_{S})$ where given in Section 6.3 and line 2 of Table 1 which guarantees that it has at most $\sqrt{\varepsilon^{\prime}}=\frac{\varepsilon}{3}$ misdecodings. Finally, we choose a near-MDS expander code [9] $\mathcal{C}$ which can efficiently correct up to $\delta_{\mathcal{C}}=\delta+\frac{\varepsilon}{3}$ half-errors and has a rate of $R_{\mathcal{C}}>1-\delta_{\mathcal{C}}-\frac{\varepsilon}{3}$ over an alphabet $\Sigma_{\mathcal{C}}=\exp(\varepsilon^{-O(1)})$ such that $\log|\Sigma_{\mathcal{C}}|\geq\frac{3\log|\Sigma_{S}|}{\varepsilon}$. This ensures that the final rate is indeed at least $R_{\mathcal{C}}-\frac{\log\Sigma_{S}}{\log\Sigma_{\mathcal{C}}}=1-\delta-3\frac{\varepsilon}{3}$ and the number of insdel errors that can be efficiently corrected is $\delta_{\mathcal{C}}-2\frac{\varepsilon}{3}\geq\delta$. The encoding and decoding complexities are furthermore straight forward and as is the polynomial time preprocessing time given Theorem 6.13 and [9]. ∎

5 Synchronization Strings

In this section, we formally define and develop $\varepsilon$-synchronization strings, which can be used as our base synchronization string $S$ in our $(n,\delta)$-indexing algorithms.

As explained in Section 1.2 it makes sense to think of the prefixes $S[1,l]$ of a synchronization string $S$ as codewords encoding their length $l$, as the prefix $S[1,l]$, or a corrupted version of it, will be exactly all the indexing information that has been received by the time the $l^{th}$ symbol is communicated:

Definition 5.1** (Codewords Associated with a Synchronization String).**

Given any synchronization string $S$ we define the set of codewords associated with $S$ to be the set of prefixes of $S$, i.e., $\{S[1,l]\ |\ 1\leq l\leq|S|\}$.

Next, we define a distance metric on any set of strings, which will be useful in quantifying how good a synchronization string $S$ and its associated set of codewords is:

Definition 5.2** (Relative Suffix Distance).**

For any two strings $S,S^{\prime}\in\Sigma^{*}$ we define their relative suffix distance $RSD$ as follows:

[TABLE]

Next we show that RSD is indeed a distance which satisfies all properties of a metric for any set of strings. To our knowledge, this metric is new. It is, however, similar in spirit to the suffix “distance” defined in [2], which unfortunately is non-symmetric and does not satisfy the triangle inequality but can otherwise be used in a similar manner as RSD in the specific context here (see also Section 6.6).

Lemma 5.3**.**

For any strings $S_{1},S_{2},S_{3}$ we have

• •

Symmetry:* $RSD(S_{1},S_{2})=RSD(S_{2},S_{1})$,*

• •

Non-Negativity and Normalization:* $0\leq RSD(S_{1},S_{2})\leq 1$,*

• •

Identity of Indiscernibles:* $RSD(S_{1},S_{2})=0\Leftrightarrow S_{1}=S_{2}$, and*

• •

Triangle Inequality:* $RSD(S_{1},S_{3})\leq RSD(S_{1},S_{2})+RSD(S_{2},S_{3})$.*

In particular, RSD defines a metric on any set of strings.

Proof.

Symmetry and non-negativity follow directly from the symmetry and non-negativity of edit distance. Normalization follows from the fact that the edit distance between two length $k$ strings can be at most $2k$. To see the identity of indiscernibles note that $RSD(S_{1},S_{2})=0$ if and only if for all $k$ the edit distance of the $k$ prefix of $S_{1}$ and $S_{2}$ is zero, i.e., if for every $k$ the $k$-prefix of $S_{1}$ and $S_{2}$ are identical. This is equivalent to $S_{1}$ and $S_{2}$ being equal. Lastly, the triangle inequality also essentially follows from the triangle inequality for edit distance. To see this let $\delta_{1}=RSD(S_{1},S_{2})$ and $\delta_{2}=RSD(S_{2},S_{3})$. By the definition of RSD this implies that for all $k$ the $k$-prefixes of $S_{1}$ and $S_{2}$ have edit distance at most $2\delta_{1}k$ and the $k$-prefixes of $S_{2}$ and $S_{3}$ have edit distance at most $2\delta_{2}k$. By the triangle inequality for edit distance, this implies that for every $k$ the $k$-prefix of $S_{1}$ and $S_{3}$ have edit distance at most $(\delta_{1}+\delta_{2})\cdot 2k$ which implies that $RSD(S_{1},S_{3})\leq\delta_{1}+\delta_{2}$. ∎

With these definitions in place, it remains to find synchronization strings whose prefixes induce a set of codewords, i.e., prefixes, with large RSD distance. It is easy to see that the RSD distance for any two strings ending on a different symbol is one. This makes the trivial synchronization string, which uses each symbol in $\Sigma$ only once, induce an associated set of codewords of optimal minimum-RSD-distance one. Such trivial synchronization strings, however, are not interesting as they require an alphabet size linear in the length $n$. To find good synchronization strings over constant size alphabets, we give the following important definition of an $\varepsilon$-synchronization string. The parameter $0<\varepsilon<1$ should be thought of measuring how far a string is from the perfect synchronization string, i.e., a string of $n$ distinct symbols.

Definition 5.4** ($\varepsilon$-Synchronization String).**

String $S\in\Sigma^{n}$ is an $\varepsilon$-synchronization string if for every $1\leq i<j<k\leq n+1$ we have that $ED\left(S[i,j),S[j,k)\right)>(1-\varepsilon)(k-i)$. We call the set of prefixes of such a string an $\varepsilon$-synchronization string.

The next lemma shows that the $\varepsilon$-synchronization string property is strong enough to imply a good minimum RSD distance between any two codewords associated with it.

Lemma 5.5**.**

If $S$ is an $\varepsilon$-synchronization string, then $RSD(S[1,i],S[1,j])>1-\varepsilon$ for any $i<j$, i.e., any two codewords associated with $S$ have RSD distance of at least $1-\varepsilon$.

Proof.

Let $k=j-i$. The $\varepsilon$-synchronization string property of $S$ guarantees that

[TABLE]

Note that this holds even if $i-k<1$. To finish the proof we note that the maximum in the definition of RSD includes the term $\frac{ED\left(S[i-k,i),S[i,j)\right)}{2k}>1-\varepsilon$, which implies that $RSD(S[1,i],S[1,j])>1-\varepsilon$. ∎

5.1 Existence and Construction

The next important step is to show that the $\varepsilon$-synchronization strings we just defined exist, particularly, over alphabets whose size is independent of the length $n$. We show the existence of $\varepsilon$-synchronization strings of arbitrary length for any $\varepsilon>0$ using an alphabet size which is only polynomially large in $1/\varepsilon$. We remark that $\varepsilon$-synchronization strings can be seen as a strong generalization of square-free sequences in which any two neighboring substrings $S[i,j)$ and $S[j,k)$ only have to be different and not also far from each other in edit distance. Thue [28] famously showed the existence of arbitrarily large square-free strings over a trinary alphabet. Thue’s methods for constructing such strings however turns out to be fundamentally too weak to prove the existence of $\varepsilon$-synchronization strings, for any constant $\varepsilon<1$.

Our existence proof requires the general Lovász local lemma which we recall here first:

Lemma 5.6** (General Lovász local lemma).**

Let $A_{1},\dots,A_{n}$ be a set of “bad” events. The directed graph $G(V,E)$ is called a dependency graph for this set of events if $V=\{1,\dots,n\}$ and each event $A_{i}$ is mutually independent of all the events $\{A_{j}:(i,j)\not\in E\}$.

Now, if there exists $x_{1},\dots,x_{n}\in[0,1)$ such that for all $i$ we have

[TABLE]

then there exists a way to avoid all events $A_{i}$ simultaneously and the probability for this to happen is bounded by

[TABLE]

Theorem 5.7**.**

For any $\varepsilon\in(0,1)$, $n\geq 1$, there exists an $\varepsilon$-synchronization string of length $n$ over an alphabet of size $\Theta(1/\varepsilon^{4})$.

Proof.

Let $S$ be a string of length $n$ obtained by concatenating two strings $T$ and $R$, where $T$ is simply the repetition of $0,\dots,t-1$ for $t=\Theta\left(\frac{1}{\varepsilon^{2}}\right)$, and $R$ is a uniformly random string of length $n$ over alphabet $\Sigma$. In particular, $S_{i}=\left(i\bmod t,R_{i}\right)$.

We prove that $S$ is an $\varepsilon$-synchronization string by showing that there is a positive probability that $S$ contains no bad triple, where $(x,y,z)$ is a bad triple if $ED(S[x,y),S[y,z))\leq(1-\varepsilon)(z-x)$.

First, note that a triple $(x,y,z)$ for which $z-x<t$ cannot be a bad triple as it consists of completely distinct symbols by courtesy of $T$. Therefore, it suffices to show that there is no bad triple $(x,y,z)$ in $R$ for $x,y,z$ such that $z-x>t$.

Let $(x,y,z)$ be a bad triple and let $a_{1}a_{2}\cdots a_{k}$ be the longest common subsequence of $R[x,y)$ and $R[y,z)$. It is straightforward to see that $ED(R[x,y),R[y,z))=(y-x)+(z-y)-2k=z-x-2k.$ Since $(x,y,z)$ is a bad triple, we have that $z-x-2k\leq(1-\varepsilon)(z-x)$, which means that $k\geq\frac{\varepsilon}{2}(z-x)$. With this observation in mind, we say that $R[x,z)$ is a bad interval if it contains a subsequence $a_{1}a_{2}\cdots a_{k}a_{1}a_{2}\cdots a_{k}$ such that $k\geq\frac{\varepsilon}{2}(z-x)$.

To prove the theorem, it suffices to show that a randomly generated string does not contain any bad intervals with a non-zero probability. We first upper bound the probability that an interval of length $l$ is bad:

[TABLE]

where the first inequality holds because if an interval of length $l$ is bad, then it must contain a repeating subsequence of length $\frac{l\varepsilon}{2}$. Any such sequence can be specified via $\varepsilon l$ positions in the $l$ long interval and the probability that a given fixed sequence is valid for a random string is ${|\Sigma|}^{-\frac{\varepsilon l}{2}}$. The second inequality comes from the fact that ${n\choose k}<\left(\frac{ne}{k}\right)^{k}$.

The resulting inequality shows that the probability of an interval of length $l$ being bad is bounded above by $C^{-\varepsilon l}$, where $C$ can be made arbitrarily large by taking a sufficiently large alphabet size $|\Sigma|$.

To show that there is a non-zero probability that the uniformly random string $R$ contains no bad interval $I$ of size $t$ or larger, we use the general Lovász local lemma stated in Lemma 5.6. Note that the badness of interval $I$ is mutually independent of the badness of all intervals that do not intersect $I$. We need to find real numbers $x_{p,q}\in[0,1)$ corresponding to intervals $R[p,q)$ for which

[TABLE]

We have seen that the left-hand side can be upper bounded by $C^{-\varepsilon\left|R[p,q)\right|}=C^{\varepsilon(p-q)}$. Furthermore, any interval of length $l^{\prime}$ intersects at most $l+l^{\prime}$ intervals of length $l$. We propose $x_{p,q}=D^{-\varepsilon\left|R[p,q)\right|}=D^{\varepsilon(p-q)}$ for some constant $D>1$. This means that it suffices to find a constant $D$ that for all substrings $R[p,q)$ satisfies

[TABLE]

or more clearly, for all $l^{\prime}\in\{1,\cdots,n\}$,

[TABLE]

which means that

[TABLE]

For $D>1$, the right-hand side of Equation (1) is maximized when $n=\infty$ and $l^{\prime}=1$, and since we want Equation (1) to hold for all $n$ and all $l^{\prime}\in\{1,\cdots,n\}$, it suffices to find a $D$ such that

[TABLE]

To this end, let

[TABLE]

Then, it suffices to have $\Sigma$ large enough so that

[TABLE]

which means that $|\Sigma|\geq\frac{e^{2}L^{2}}{\varepsilon^{2}}$ suffices to allow us to use the Lovász local lemma. We claim that $L=\Theta(1)$, which will complete the proof. Since $t=\omega\left(\frac{\log\frac{1}{\varepsilon}}{\varepsilon}\right)$,

[TABLE]

Therefore, we can use the fact that $(1-x)^{k}>1-xk$ to show that:

[TABLE]

Equation (3) is derived using the fact that $\prod_{i=1}^{\infty}(1-x_{i})\geq 1-\sum_{i=1}^{\infty}x_{i}$ and Equation (5) is a result of the following equality for $x<1$:

[TABLE]

One can see that for $D=7$, $\max_{\varepsilon}\left\{\frac{2}{\varepsilon^{3}}\frac{D^{-\frac{1}{\varepsilon}}}{(1-D^{-\varepsilon})^{2}}\right\}<0.9$, and therefore step (3) is legal and (6) can be upper-bounded by a constant. Hence, $L=\Theta(1)$ and the proof is complete. ∎

Remarks on the alphabet size: Theorem 5.7 shows that for any $\varepsilon>0$ there exists an $\varepsilon$-synchronization string over alphabets of size $O(\varepsilon^{-4})$. A polynomial dependence on $\varepsilon$ is also necessary. In particular, there do not exist any $\varepsilon$-synchronization string over alphabets of size smaller than $\varepsilon^{-1}$. In fact, any consecutive substring of size $\varepsilon^{-1}$ of an $\varepsilon$-synchronization string has to contain completely distinct elements. This can be easily proven as follows: For sake of contradiction let $S[i,i+\varepsilon^{-1})$ be a substring of an $\varepsilon$-synchronization string where $S[j]=S[j^{\prime}]$ for $i\leq j<j^{\prime}<i+\varepsilon^{-1}$. Then, $ED\left(S[j],S[j+1,j^{\prime}+1))\right)=j^{\prime}-j-1=(j^{\prime}+1-j)-2\leq(j^{\prime}+1-j)(1-2\varepsilon)$. We believe that using the Lovász Local Lemma together with a more sophisticated non-uniform probability space, which avoids any repeated symbols within a small distance, allows avoiding the use of the string $T$ in our proof and improving the alphabet size to $O(\varepsilon^{-2})$. It seems much harder to improved the alphabet size to $o(\varepsilon^{-2})$ and we are not convinced that it is possible. This work thus leaves open the interesting question of closing the quadratic gap between $O(\varepsilon^{-2})$ and $\Omega(\varepsilon^{-1})$ from either side.

Theorem 5.7 also implies an efficient randomized construction.

Lemma 5.8**.**

There exists a randomized algorithm which for any $\varepsilon>0$ constructs a $\varepsilon$-synchronization string of length $n$ over an alphabet of size $O(\varepsilon^{-4})$ in expected time $O(n^{5})$.

Proof.

Using the algorithmic framework for the Lovász local lemma given by Moser and Tardos [23] and the extensions by Haeupler et al.[16] one can get such a randomized algorithm from the proof in Theorem 5.7. The algorithm starts with a random string over any alphabet $\Sigma$ of size $\varepsilon^{-C}$ for some sufficiently large $C$. It then checks all $O(n^{2})$ intervals for a violation of the $\varepsilon$-synchronization string property. For every interval this is an edit distance computation which can be done in $O(n^{2})$ time using the classical Wagner-Fischer dynamic programming algorithm. If a violating interval is found the symbols in this interval are assigned fresh random values. This is repeated until no more violations are found. [16] shows that this algorithm performs only $O(n)$ expected number of re-samplings. This gives an expected running time of $O(n^{5})$ overall, as claimed. ∎

Lastly, since synchronization strings can be encoded and decoded in a streaming fashion they have many important applications in which the length of the required synchronization string is not known in advance. In such a setting it is advantageous to have an infinite synchronization string over a fixed alphabet. In particular, since every consecutive substring of an $\varepsilon$-synchronization string is also an $\varepsilon$-synchronization string by definition, having an infinite $\varepsilon$-synchronization string also implies the existence for every length $n$, i.e., Theorem 5.7. Interestingly, a simple argument shows that the converse is true as well, i.e., the existence of an $\varepsilon$-synchronization string for every length $n$ implies the existence of an infinite $\varepsilon$-synchronization string over the same alphabet:

Lemma 5.9**.**

For any $\varepsilon\in(0,1)$ there exists an infinite $\varepsilon$-synchronization string over an alphabet of size $\Theta(1/\varepsilon^{4})$.

Proof of Lemma 5.9.

Fix any $\varepsilon\in(0,1)$. According to Theorem 5.7 there exist an alphabet $\Sigma$ of size $O(1/\varepsilon^{4})$ such that there exists an at least one $\varepsilon$-synchronization strings over $\Sigma$ for every length $n\in\mathbb{N}$. We will define a synchronization string $S=s_{1}\cdot s_{2}\cdot s_{3}\ldots$ with $s_{i}\in\Sigma$ for any $i\in\mathbb{N}$ for which the $\varepsilon$-synchronization property holds for any $i,j,k\in\mathbb{N}$. We define this string inductively. In particular, we fix an ordering on $\Sigma$ and define $s_{1}\in\Sigma$ to be the first symbol in this ordering such that an infinite number of $\varepsilon$-synchronization strings over $\Sigma$ starts with $s_{1}$. Given that there is an infinite number of $\varepsilon$-synchronization over $\Sigma$ such an $s_{1}$ exists. Furthermore, the set of $\varepsilon$-synchronization strings over $\Sigma$ which start with $s_{1}$ remains infinite by definition, allowing us to define $s_{2}\in\Sigma$ to be the lexicographically first symbol in $\Sigma$ such there exists an infinite number of $\varepsilon$-synchronization strings over $\Sigma$ starting with $s_{1}\cdot s_{2}$. In the same manner, we inductively define $s_{i}$ to be the lexicographically first symbol in $\Sigma$ for which there exists and infinite number of $\varepsilon$-synchronization strings over $\Sigma$ starting with $s_{1}\cdot s_{2}\cdot\ldots\cdot s_{i}$. To see that the infinite string defined in this manner does indeed satisfy the edit distance requirement of the $\varepsilon$-synchronization property defined in Definition 5.4, we note that for every $i<j<k$ with $i,j,k\in\mathbb{N}$ there exists, by definition, an $\varepsilon$-synchronization string, and in fact an infinite number of them, which contains $S[1,k]$ and thus also $S[i,k]$ as a consecutive substring implying that indeed $ED\left(S[i,j),S[j,k)\right)>(1-\varepsilon)(k-i)$ as required. Our definition thus produces the unique lexicographically first infinite $\varepsilon$-synchronization string over $\Sigma$. ∎

We remark that any string produced by the randomized construction of Lemma 5.8 is guaranteed to be a correct $\varepsilon$-synchronization string (not just with probability one). This randomized synchronization string construction is furthermore only needed once as a pre-processing step. The encoder or decoder of any resulting error correcting codes do not require any randomization. Furthermore, in Section 6 we will provide a deterministic polynomial time construction of a relaxed version of $\varepsilon$-synchronization strings that can still be used as a basis for good $(n,\delta)$-indexing algorithms thus leading to insdel codes with a deterministic polynomial time code construction as well.

It nonetheless remains interesting to obtain fast deterministic constructions of finite and infinite $\varepsilon$-synchronization strings. In a subsequent work we achieve such efficient deterministic constructions for $\varepsilon$-synchronization strings. Our constructions even produce the infinite $\varepsilon$-synchronization string $S$ proven to exist by Lemma 5.9, which is much less explicit: While for any $n$ and any $\varepsilon$ an $\varepsilon$-synchronization string of length $n$ can in principle be found using an exponential time enumeration there is no straight forward algorithm which follows the proof of Lemma 5.9 and given an $i\in\mathbb{N}$ produces the $i^{th}$ symbol of such an $S$ in a finite amount of time (bounded by some function in $i$). Our constructions require significantly more work but in the end lead to an explicit deterministic construction of an infinite $\varepsilon$-synchronization string for any $\varepsilon>0$ for which the $i^{th}$ symbol can be computed in only $O(\log i)$ time – thus satisfying one of the strongest notions of constructiveness that can be achieved.

5.2 Decoding

We now provide an algorithm for decoding synchronization strings, i.e., an algorithm that can form a solution to the indexing problem along with $\varepsilon$-synchronization strings. In the beginning of Section 5, we introduced the notion of relative suffix distance between two strings. Theorem 5.5 stated a lower bound of $1-\varepsilon$ for relative suffix distance between any two distinct codewords associated with an $\varepsilon$-synchronization string, i.e., its prefixes. Hence, a natural decoding scheme for detecting the index of a received symbol would be finding the prefix with the closest relative suffix distance to the string received thus far. We call this algorithm the minimum relative suffix distance decoding algorithm.

We define the notion of relative suffix error density at index $i$ which presents the maximized density of errors taken place over suffixes of $S[1,i]$. We will introduce a very natural decoding approach for synchronization strings that simply works by decoding a received string by finding the codeword of a synchronization string $S$ (prefix of synchronization string) with minimum distance to the received string. We will show that this decoding procedure works correctly as long as the relative suffix error density is not larger than $\frac{1-\varepsilon}{2}$. Then, we will show that if adversary is allowed to perform $c$ many insertions or deletions, the relative suffix distance may exceed $\frac{1-\varepsilon}{2}$ upon arrival of at most $\frac{2c}{1-\varepsilon}$ many successfully transmitted symbols. Finally, we will deduce that this decoding scheme decodes indices of received symbols correctly for all but $\frac{2c}{1-\varepsilon}$ many of successfully transmitted symbols. Formally, we claim that:

Theorem 5.10**.**

Any $\varepsilon$-synchronization string of length $n$ along with the minimum relative suffix distance decoding algorithm form a solution to $(n,\delta)$-indexing problem that guarantees $\frac{2}{1-\varepsilon}n\delta$ or less misdecodings. This decoding algorithm is streaming and can be implemented so that it works in $O(n^{4})$ time.

Before proceeding to the formal statement and the proofs of the claims above, we first provide the following useful definitions.

Definition 5.11** (Error Count Function).**

Let $S$ be a string sent over an insertion-deletion channel. We denote the error count from index $i$ to index $j$ with $\mathcal{E}(i,j)$ and define it to be the number of insdels applied to $S$ from the moment $S[i]$ is sent until the moment $S[j]$ is sent. $\mathcal{E}(i,j)$ counts the potential deletion of $S[j]$. However, it does not count the potential deletion of $S[i]$.

Definition 5.12** (Relative Suffix Error Density).**

Let string $S$ be sent over an insertion-deletion channel and let $\mathcal{E}$ denote the corresponding error count function. We define the relative suffix error density of the communication as:

[TABLE]

The following lemma relates the suffix distance of the message being sent by sender and the message being received by the receiver at any point of a communication over an insertion-deletion channel to the relative suffix error density of the communication at that point.

Lemma 5.13**.**

Let string $S$ be sent over an insertion-deletion channel and the corrupted message $S^{\prime}$ be received on the other end. The relative suffix distance $RSD(S,S^{\prime})$ between the string $S$ that was sent and the string $S^{\prime}$ which was received is at most the relative suffix error density of the communication.

Proof.

Let $\tilde{\tau}=(\tilde{\tau}_{1},\tilde{\tau}_{2})$ be the string matching from $S$ to $S^{\prime}$ that characterizes insdels that have turned $S$ into $S^{\prime}$. Then:

[TABLE]

where $\tau^{\prime}$ is $\tilde{\tau}$ limited to its suffix corresponding to $S(|S|-k,|S|])$. Note that Steps (7) and (8) follow from the definitions of edit distance and relative suffix distance. Moreover, to see Step (9), one has to note that one single insertion or deletion on the $k$-element suffix of a string may result into a string with $k$-element suffix of edit distance two of the original string’s $k$-element suffix; one stemming from the inserted/deleted symbol and the other one stemming from a symbol appearing/disappearing at the beginning of the suffix in order to keep the size of suffix $k$. ∎

A key consequence of Lemma 5.13 is that if an $\varepsilon$-synchronization string is being sent over an insertion-deletion channel and at some step the relative suffix error density corresponding to corruptions is smaller than $\frac{1-\varepsilon}{2}$, the relative suffix distance of the sent string and the received one at that point is smaller than $\frac{1-\varepsilon}{2}$; therefore, as RSD of all pairs of codewords associated with an $\varepsilon$-synchronization string are greater than $1-\varepsilon$, the receiver can correctly decode the index of the corrupted codeword he received by simply finding the codeword with minimum relative suffix distance.

The following lemma states that such a guarantee holds most of the time during transmission of a synchronization string:

Lemma 5.14**.**

Let $\varepsilon$-synchronization string $S$ be sent over an insertion-channel channel and corrupted string $S^{\prime}$ be received on the other end. If there are $c_{i}$ symbols inserted and $c_{d}$ symbols deleted, then, for any integer $t$, the relative suffix error density is smaller than $\frac{1-\varepsilon}{t}$ upon arrival of all but $\frac{t(c_{i}+c_{d})}{1-\varepsilon}-c_{d}$ many of the successfully transmitted symbols.

Proof.

Let $\mathcal{E}$ denote the error count function of the communication. We define the potential function $\Phi$ over $\{0,1,\cdots,n\}$ as follows:

[TABLE]

Also, set $\Phi(0)=0$. We prove the theorem by showing the correctness of the following claims:

If $\mathcal{E}(i-1,i)=0$, i.e., the adversary does not insert or delete any symbols in the interval starting right after the moment $S[i-1]$ is sent and ending at when $S[i]$ is sent, then the value of $\Phi$ drops by 1 or becomes/stays zero, i.e., $\Phi(i)=\max\left\{0,\Phi(i-1)-1\right\}$. 2. 2.

If $\mathcal{E}(i-1,i)=k$, i.e., adversary inserts or deletes $k$ symbols in the interval starting right after the moment $S[i-1]$ is sent and ending at when $S[i]$ is sent, then the value of $\Phi$ increases by at most $\frac{tk}{1-\varepsilon}-1$, i.e., $\Phi(i)\leq\Phi(i-1)+\frac{tk}{1-\varepsilon}-1$. 3. 3.

If $\Phi(i)=0$, then the relative suffix error density of the string that is received when $S[i]$ arrives at the receiving side is not larger than $\frac{1-\varepsilon}{t}$.

Given the correctness of claims made above, the lemma can be proved as follows. As adversary can apply at most $c_{i}+c_{d}$ insertions or deletions, $\Phi$ can gain a total increase of $\frac{t\cdot(c_{i}+c_{d})}{1-\varepsilon}$. Therefore, the value of $\Phi$ can be non-zero for at most $\frac{t\cdot(c_{i}+c_{d})}{1-\varepsilon}$ many inputs. As value of $\Phi(i)$ is non-zero for all $i$’s where $S[i]$ has been removed by adversary, there are at most $\frac{t\cdot(c_{i}+c_{d})}{1-\varepsilon}-c_{d}$ indices $i$ where $\Phi(i)$ is non-zero and $i$ is successfully transmitted. Hence, at most $\frac{t\cdot(c_{i}+c_{d})}{1-\varepsilon}-c_{d}$ many of correctly transmitted symbols can possibly be decoded incorrectly.

We now proceed to the proof of each of the above claims to finish the proof:

In this case, $\mathcal{E}(i-s,i)=\mathcal{E}(i-s,i-1)$. So,

[TABLE] 2. 2.

In this case, $\mathcal{E}(i-s,i)=\mathcal{E}(i-s,i-1)+k$. So,

[TABLE] 3. 3.

[TABLE]

These finish the proof of the lemma. ∎

Now, we have all necessary tools to analyze the performance of the minimum relative suffix distance decoding algorithm:

Proof of Theorem 5.10.

As adversary is allowed to insert or delete up $n\delta$ symbols, by Lemma 5.14, there are at most $\frac{2n\delta}{1-\varepsilon}$ successfully transmitted symbols during the arrival of which at the receiving side, the relative suffix error density is greater than $\frac{1-\varepsilon}{2}$; Hence, by Lemma 5.13, there are at most $\frac{2n\delta}{1-\varepsilon}$ misdecoded successfully transmitted symbols.

Further, we remark that this algorithm can be implemented in $O(n^{4})$ as follows: Using dynamic programming, we can pre-process the edit distance of any consecutive substring of $S$, like $S[i,j]$ to any consecutive substring of $S^{\prime}$, like $S^{\prime}[i^{\prime},j^{\prime}]$, in $O(n^{4})$. Then, for each symbol of the received string, like $S^{\prime}[l^{\prime}]$, we can find the codeword with minimum relative suffix distance to $S^{\prime}[1,l^{\prime}]$ by calculating the relative suffix distance of it to all $n$ codewords. Finding suffix distance of $S^{\prime}[1,l^{\prime}]$ and a codeword like $S[1,l]$ can also be simply done by minimizing $\frac{ED(S(l-k,l],S^{\prime}(l^{\prime}-k,l^{\prime}])}{k}$ over $k$ which can be done in $O(n)$. With a $O(n^{4})$ pre-process and a $O(n^{3})$ computation as mentioned above, we have shown that the decoding process can be implemented in $O(n^{4})$. ∎

We remark that by taking $\varepsilon=o(1)$, one can obtain a solution to the $(n,\delta)$-indexing problem with a misdecoding guarantee of $2n\delta(1+o(1))$ which, using Theorem 4.1, results into a translation of $n\delta$ insertions and deletions into $n\delta(5+o(1))$ half-errors. As explained in Section 1.3, such a guarantee however falls short of giving Theorem 1.1. In Section 6.6, we show that this guarantee of the min-distance-decoder can be slightly improved to work beyond an RSD distance of $\frac{1-\varepsilon}{2}$, at the cost of some simplicity, by considering an alternative distance measure. In particular, the relative suffix pseudo distance RSPD, which was introduced in [2], can act as a metric stand-in for the minimum-distance decoder and lead to slightly improved decoding guarantees, despite neither being symmetric nor satisfying the triangle inequality. For any set of $k=k_{i}+k_{d}$ insdel errors consisting of $k_{i}$ insertions and $k_{d}$ deletions the RSPD based indexing solution leads to at most $(1+\varepsilon)(3k_{i}+k_{d})$ half-errors which does imply “near-MDS” codes for deletion-only channels but still falls short for general insdel errors.

This leaves open the intriguing question whether a further improved (pseudo) distance definition can achieve an indexing solution with negligible number of misdecodings for the minimum-distance decoder.

6 More Advanced Global Decoding Algorithms

Thus far, we have introduced $\varepsilon$-synchronization strings as fitting solutions to the indexing problem. In Section 5.2, we provided an algorithm to solve the indexing problem along with synchronization strings with an asymptotic guarantee of $2n\delta$ misdecodings. As explained in Section 1.3, such a guarantee falls short of giving Theorem 1.1. In this section, we thus provide a variety of more advanced decoding algorithms that provide a better decoding guarantees, in particular achieve a misdecoding fraction which goes to zero as $\varepsilon$ goes to zero.

We start by pointing out a very useful property of $\varepsilon$-synchronization strings in Section 6.1. We define a monotone matching between two strings as a common subsequence of them. We will next show that in a monotone matching between an $\varepsilon$-synchronization string and itself, the number of matches that both correspond to the same element of the string is fairly large. We will refer to this property as $\varepsilon$-self-matching property. We show that one can very formally think of this $\varepsilon$-self-matching property as a robust global guarantee in contrast to the factor-closed strong local requirements of the $\varepsilon$-synchronization property. One advantage of this relaxed notion of $\varepsilon$-self-matching is that one can show that a random string over alphabets polynomially large in $\varepsilon^{-1}$ satisfies this property (Section 6.2). This leads to a particularly simple generation process for $S$. Finally, showing that this property even holds for approximately $\log n$-wise independent strings directly leads to a deterministic polynomial time algorithm generating such strings as well.

In Section 6.3, we propose a decoding algorithm for insdel errors that basically works by finding monotone matchings between the received string and the synchronization string. Using the $\varepsilon$-self-matching property we show that this algorithm guarantees $O\left(n\sqrt{\varepsilon}\right)$ misdecodings. This algorithm works in time $O(n^{2}/\sqrt{\varepsilon})$ and is exactly what we need to prove our main theorem.

Lastly, in Sections 6.4 and 6.5 we provide two simpler linear time algorithms that solve the indexing problem under the assumptions that the adversary can only delete symbols or only insert new symbols. These algorithms not only guarantee asymptotically optimal $\frac{\varepsilon}{1-\varepsilon}n\delta$ misdecodings but are also error-free. Table 1 provides a break down of the decoding schemes presented in this paper, describing the type of error they work under, the number of misdecodings they guarantee, whether they are error-free or streaming, and their decoding complexity.

6.1 Monotone Matchings and the $\varepsilon$-Self Matching Property

Before proceeding to the main results of this section, we start by defining monotone matchings which provide a formal way to refer to common substrings of two strings:

Definition 6.1** (Monotone Matchings).**

A monotone matching between $S$ and $S^{\prime}$ is a set of pairs of indices like:

[TABLE]

where $a_{1}<\cdots<a_{m}$, $b_{1}<\cdots<b_{m}$, and $S[a_{i}]=S^{\prime}[b_{i}]$.

We now point out a key property of synchronization strings that will be broadly used in our decoding algorithms. Basically, Theorem 6.2 states that two similar subsequences of an $\varepsilon$-synchronization string cannot disagree on many positions. More formally, let $M=\left\{(a_{1},b_{1}),\cdots,(a_{m},b_{m})\right\}$ be a monotone matching between $S$ and itself. We call the pair $(a_{i},b_{i})$ a good pair if $a_{i}=b_{i}$ and a bad pair otherwise. Then:

Theorem 6.2**.**

Let $S$ be an $\varepsilon$-synchronization string of size $n$ and $M=\left\{(a_{1},b_{1}),\cdots,(a_{m},b_{m})\right\}$ be a monotone matching of size $m$ from $S$ to itself containing $g$ good pairs and $b$ bad pairs. Then,

[TABLE]

Proof.

Let $(a^{\prime}_{1},a^{\prime}_{2}),\cdots,(a^{\prime}_{m^{\prime}},b^{\prime}_{m^{\prime}})$ indicate the set of bad pairs in $M$ indexed as $a^{\prime}_{1}<\cdots<a^{\prime}_{m^{\prime}}$ and $b^{\prime}_{1}<\cdots<b^{\prime}_{m^{\prime}}$. Without loss of generality, assume that $a^{\prime}_{1}<b^{\prime}_{1}$. Let $k_{1}$ be the largest integer such that $a^{\prime}_{k_{1}}<b^{\prime}_{1}$. Then, the pairs $(a^{\prime}_{1},a^{\prime}_{2}),\cdots,(a^{\prime}_{k_{1}},b^{\prime}_{k_{1}})$ form a common substring of size $k_{1}$ between $T_{1}=S[a^{\prime}_{1},b^{\prime}_{1})$ and $T^{\prime}_{1}=S[b^{\prime}_{1},b^{\prime}_{k_{1}}]$. Now, the synchronization string guarantee implies that:

[TABLE]

Note that the monotonicity of the matching guarantees that there are no good matches occurring on indices covered by $T_{1}$ and $T^{\prime}_{1}$, i.e., $a^{\prime}_{1},\cdots,b^{\prime}_{k_{1}}$. One can repeat very same argument for the remaining bad matches to rule out bad matches $(a^{\prime}_{k_{1}+1},b^{\prime}_{k_{1}+1}),\cdots,(a^{\prime}_{k_{1}+k_{2}},b^{\prime}_{k_{1}+k_{2}})$ for some $k_{2}$ having the following inequality guaranteed:

[TABLE]

where

[TABLE]

For a pictorial representation see Figure 1.

Continuing the same procedure, one can find $k_{1},\cdots,k_{l}$, $T_{1},\cdots,T_{l}$, and $T^{\prime}_{1},\cdots,T^{\prime}_{l}$ for some $l$. Summing up all inequalities of form (10), we will have:

[TABLE]

Note that $\sum_{i=1}^{l}k_{i}=u$ and $T_{i}$s are mutually exclusive and contain no indices where a good pair occurs at. Same holds for $T^{\prime}_{i}$s. Hence, $\sum_{i=1}^{l}|T_{i}|\leq n-g$ and $\sum_{i=1}^{l}|T^{\prime}_{i}|\leq n-g$. All these along with (11) give that:

[TABLE]

∎

We define the $\varepsilon$-self-matching property as follows:

Definition 6.3** ($\varepsilon$-self-matching property).**

String $S$ satisfies $\varepsilon$-self-matching property if any monotone matching between $S$ and itself contains less than $\varepsilon|S|$ bad pairs.

Note that $\varepsilon$-synchronization property concerns all substrings of a string while the $\varepsilon$-self-matching property only concerns the string itself. Granted that, we now show that $\varepsilon$-synchronization property and satisfying $\varepsilon$-self-matching property on all substrings are equivalent up to a factor of two:

Theorem 6.4**.**

$\varepsilon$-synchronization and $\varepsilon$-self matching properties are related as follows:

• a)

If $S$ is an $\varepsilon$-synchronization string, then all substrings of $S$ satisfy $\varepsilon$-self-matching property.

• b)

If all substrings of string $S$ satisfy the $\frac{\varepsilon}{2}$-self-matching property, then $S$ is $\varepsilon$-synchronization string.

Proof of Theorem 6.4 (a).

This part is a straightforward consequence of Theorem 6.2.∎

Proof of Theorem 6.4 (b).

Assume by contradiction that there are $i<j<k$ such that $ED(S[i,j),S[j,k))\leq(1-\varepsilon)(k-i)$. Then, $LCS(S[i,j),S[j,k))\geq\frac{k-i-(1-\varepsilon)(k-i)}{2}=\frac{\varepsilon}{2}(k-i)$. The corresponding pairs of such longest common substring form a monotone matching of size $\frac{\varepsilon}{2}(k-i)$ which contradicts $\frac{\varepsilon}{2}$-self-matching property of $S$. ∎

As a matter of fact, the decoding algorithms we will propose for $\varepsilon$-synchronization strings in Sections 6.3, 6.4, and 6.5 only make use of the $\varepsilon$-self-matching property of the $\varepsilon$-synchronization string.

We now proceed to the definition of $\varepsilon$-bad-indices which will enable us to show that $\varepsilon$-self matching property, as opposed to the $\varepsilon$-synchronization property, is robust against local changes.

Definition 6.5** ($\varepsilon$-bad-index).**

We call index $k$ of string $S$ an $\varepsilon$-bad-index if there exists a factor $S[i,j]$ of $S$ with $i\leq k\leq j$ where $S[i,j]$ does not satisfy the $\varepsilon$-self-matching property. In this case, we also say that index $k$ blames interval $[i,j]$.

Using the notion of $\varepsilon$-bad indices, we now present Lemma 6.6. This lemma suggests that a string containing limited fraction of $\varepsilon$-bad indices would still be an $\varepsilon^{\prime}$-self matching string for some $\varepsilon^{\prime}>\varepsilon$. An important consequence of this result is that if one changes a limited number of elements in a given $\varepsilon$-self matching string, the self matching property will be essentially preserved to a lesser extent. Note that $\varepsilon$-synchronization property do not satisfy any such robustness quality.

Lemma 6.6**.**

If the fraction of $\varepsilon$-bad indices in string $S$ is less than $\gamma$, then $S$ satisfies $(\varepsilon+2\gamma)$-self matching property.

Proof.

Consider a matching from $S$ to itself. The number of bad matches whose both ends refer to non-$\varepsilon$-bad indices of $S$ is at most $|S|(1-\gamma)\varepsilon$ by definition. Further, each $\varepsilon$-bad index can appear at most once in each end of bad pairs. Therefore, the number of bad pairs in $S$ can be at most:

[TABLE]

which, by definition, implies that $S$ satisfies the $(\varepsilon+2\gamma)$-self-matching property. ∎

On the other hand, in the following theorem, we will show that within a given $\varepsilon$-self matching string, there can be a limited number of $\varepsilon^{\prime}$-bad indices for sufficiently large $\varepsilon^{\prime}>\varepsilon$.

Lemma 6.7**.**

Let $S$ be an $\varepsilon$-self matching string of length $n$. Then, for any $3\varepsilon<\varepsilon^{\prime}<1$, at most $\frac{3n\varepsilon}{\varepsilon^{\prime}}$ many indices of $S$ can be $\varepsilon^{\prime}$-bad.

Proof.

Let $s_{1},s_{2},\cdots,s_{k}$ be bad indices of $S$ and $\varepsilon^{\prime}$-bad index $s_{i}$ blame substring $S[a_{i},b_{i})$. As intervals $S[a_{i},b_{i})$ are supposed to be bad, there has to be a $\varepsilon^{\prime}$-self matching within each $S[a_{i},b_{i})$ like $M_{i}$ for which $|M_{i}|\geq\varepsilon^{\prime}\cdot|[a_{i},b_{i})|$. We claim that one can choose a subset of $[1..k]$ like $I$ for which

• •

Corresponding intervals to the indices in $I$ are mutually exclusive. In other words, for any $i,j\in I$ where $i\not=j$, $[A_{i},b_{i})\cap[a_{j},b_{j})=\emptyset$.

• •

$\sum_{i\in I}\left|[A_{i},b_{i})\right|\geq\frac{k}{3}$.

If such $I$ exists, one can take $\bigcup_{i\in I}M_{i}$ as a self matching in $S$ whose size is larger than $\frac{k\varepsilon^{\prime}}{3}$. As $S$ is an $\varepsilon$-self matching string,

[TABLE]

which finishes the proof. The only remaining piece is proving the claim. Note that any index in $\bigcup_{i\in I}[a_{i},b_{i})$ is a $\varepsilon^{\prime}$-bad index as they by definition belong to an interval with a $\varepsilon^{\prime}$-self matching. Therefore, $\left|\bigcup_{i\in I}[a_{i},b_{i})\right|=k$. In order to find set $I$, we greedily choose the largest substring $[a_{i},b_{i})$, put its corresponding index into $I$ and then remove any interval intersecting $[a_{i},b_{i})$. We continue repeating this procedure until all substrings are removed. The set $I$ obtained by this procedure clearly satisfies the first claimed property. Moreover, note that if $l_{i}=\left|[a_{i},b_{i})\right|$, any interval intersecting $[a_{i},b_{i})$ falls into $[a_{i}-l_{i},b_{i}+l_{i})$ which is an interval of length $3l_{i}$. This certifies the second property and finishes the proof. ∎

As the final remark on the $\varepsilon$-self matching property and its relation with the more strict $\varepsilon$-synchronization property, we show that using the minimum RSD decoder for indexing together with an $\varepsilon$-self matching string leads to guarantees on the misdecoding performance which are only slightly weaker than the guarantee obtained by $\varepsilon$-synchronization strings. In order to do so, we first show that the $(1-\varepsilon)$ RSD distance property of prefixes holds for any non-$\varepsilon$-bad index in any arbitrary string in Theorem 6.8. Then, using Theorem 6.8 and Lemma 6.7, we upper-bound the number of misdecodings that may happen using a minimum RSD decoder along with an $\varepsilon$-self matching string in Theorem 6.9.

Theorem 6.8**.**

Let $S$ be an arbitrary string of length $n$ and $1\leq i\leq n$ be such that $i$’th index of $S$ is not an $\varepsilon$-bad index. Then, for any $j\not=i$, $RSD(S[1,i],S[1,j])>1-\varepsilon$.

Proof.

Without loss of generality assume that $j<i$. Consider the interval $[2j-i+1,i]$. As $i$ is not an $\varepsilon$-bad index, there is no self matching of size $2\varepsilon(i-j)$ within $[2j-i,i]$. In particular, the edit distance of $S[2j-i+1,j]$ and $[j+1,i]$ has to be larger than $(1-\varepsilon)\cdot 2(i-j)$ which equivalently means $RSD(S[1,i],S[1,j])>1-\varepsilon$. Note that if $2j-i+1<0$ the proof goes through by simply replacing $2j-i+1$ with zero. ∎

Theorem 6.9**.**

Using any $\varepsilon$-self matching string along with minimum RSD algorithm, one can solve the $(n,\delta)$-indexing problem with a guarantee of $n(4\delta+6\varepsilon)$ misdecodings.

Proof.

Note that applying Lemma 6.7 for $\varepsilon^{\prime}$ gives that there are at most $\frac{3n\varepsilon}{\varepsilon^{\prime}}$ indices in $S$ that are $\varepsilon^{\prime}$-bad. Further, using Theorems 5.10 and 6.8, at most $\frac{2n\delta}{1-\varepsilon^{\prime}}$ many of the other indices might be decoded incorrectly upon their arrivals. Therefore, this solution for the $(n,\delta)$-indexing problem can contain at most $n\left(\frac{3\varepsilon}{\varepsilon^{\prime}}+\frac{2\delta}{1-\varepsilon^{\prime}}\right)$ many incorrectly decoded indices. Setting $\varepsilon^{\prime}=\frac{3\varepsilon}{3\varepsilon+2\delta}$ gives an upper bound of $n(4\delta+6\varepsilon)$ on the number of misdecodings. ∎

6.2 Efficient Polynomial Construction of $\varepsilon$-self matching strings

In this section, we will use Lemma 6.6 to show that there is a polynomial deterministic construction of a string of length $n$ with the $\varepsilon$-self-matching property, which can then for example be used to obtain a deterministic code construction. We start by showing that even random strings satisfy the $\varepsilon$-selfmatching property for an $\varepsilon$ polynomial in the alphabet size:

Theorem 6.10**.**

A random string on an alphabet of size $O(\varepsilon^{-3})$ satisfies $\varepsilon$-selfmatching property with a constant probability.

Proof.

Let $S$ be a random string on alphabet $\Sigma$ of size $|\Sigma|=\varepsilon^{-3}$. We are going to find the expected number of $\varepsilon$-bad indices in $S$. We first count the expected number of $\varepsilon$-bad indices that blame intervals of length $\frac{2}{\varepsilon}$ or smaller. If index $k$ blames interval $S[i,j]$ where $j-i<2\varepsilon^{-1}$, there has to be two identical symbols appearing in $S[i,j]$ which gives that there are two identical elements in $4\varepsilon^{-1}$ neighborhood of $S$. Therefore, the probability of index $k$ being $\varepsilon$-bad blaming $S[i,j]$ for $j-i<2\varepsilon^{-1}$ can be upper-bounded by ${4\varepsilon^{-1}\choose 2}\frac{1}{|\Sigma|}\leq 8\varepsilon$. Thus, the expected fraction of $\varepsilon$-bad indices that blame intervals of length $\frac{2}{\varepsilon}$ or smaller is less than $8\varepsilon$.

We now proceed to finding the expected fraction of $\varepsilon$-bad indices in $S$ blaming intervals of length $2\varepsilon^{-1}$ or more. Since every interval of length $l$ which does not satisfy $\varepsilon$-self-matching property causes at most $l$ $\varepsilon$-bad indices, we get that the expected fraction of such indices, i.e., $\gamma^{\prime}$, is at most:

[TABLE]

Last inequality holds because the number of possible matchings is at most ${l\choose l\varepsilon}^{2}$. Further, fixing the matching edges, the probability of the elements corresponding to pair $(a,b)$ of the matching being identical is independent from all pairs $(a^{\prime},b^{\prime})$ where $a^{\prime}<a$ and $b^{\prime}<b$. Hence, the probability of the set of pairs being a matching between random string $S$ and itself is $\frac{1}{|\Sigma|^{l\varepsilon}}$. Then,

[TABLE]

Note that series $\sum_{l=2\varepsilon^{-1}}^{\infty}lx^{l}=\frac{2\varepsilon^{-1}x^{2\varepsilon^{-1}}-(2\varepsilon^{-1}-1)x^{2\varepsilon^{-1}+1}}{(1-x)^{2}}$ for $|x|<1$. Therefore, for $0<x<\frac{1}{2}$, $\sum_{l=l_{0}}^{\infty}lx^{l}<8\varepsilon^{-1}x^{2\varepsilon^{-1}}$. So,

[TABLE]

Using Lemma 6.6, this random structure has to satisfy $(\varepsilon+2\gamma)$-self-matching property where

[TABLE]

Therefore, using Markov inequality, a randomly generated string over alphabet $O(\varepsilon^{-3})$ satisfies $\varepsilon$-matching property with constant probability. The constant probability can be as high as one wishes by applying higher constant factor in alphabet size. ∎

As the next step, we prove a similar claim for strings of length $n$ whose symbols are chosen from an $\Theta\left(\frac{\log n}{\log(1/\varepsilon)}\right)$-wise independent [24] distribution over a larger, yet still $\varepsilon^{-O(1)}$ size, alphabet. This is the key step in allowing for a derandomization using the small sample spaces of Naor and Naor [24]. The proof of Theorem 6.11 follows a similar strategy as was used in [3] to derandomize the constructive Lovász local lemma. In particular the crucial idea, given by Claim 6.12, is to show that for any large obstruction there has to exist a smaller yet not too small obstruction. This allows one to prove that in the absence of any small and medium size obstructions no large obstructions exist either.

Theorem 6.11**.**

A $\frac{c\log n}{\log(1/\varepsilon)}$-wise independent random string of size $n$ on an alphabet of size $O(\varepsilon^{-6})$ satisfies $\varepsilon$-matching property with a non-zero constant probability. $c$ is a sufficiently large constant.

Proof.

Let $S$ be a pseudo-random string of length $n$ with $\frac{c\log n}{\log(1/\varepsilon)}$-wise independent symbols. Then, Step (12) is invalid as the proposed upper-bound does not work for $l>\frac{c\log n}{\varepsilon\log(1/\varepsilon)}$. To bound the probability of intervals of size $\Omega\left(\frac{c\log n}{\varepsilon\log(1/\varepsilon)}\right)$ not satisfying $\varepsilon$-self matching property, we claim that:

Claim 6.12**.**

Any string of size $l>100m$ which contains an $\varepsilon$-self-matching contains two sub-intervals $I_{1}$ and $I_{2}$ of size $m$ where there is a matching of size $0.99\frac{m\varepsilon}{2}$ between $I_{1}$ and $I_{2}$.

Using Claim 6.12, one can conclude that any string of size $l>100\frac{c\log n}{\varepsilon\log(1/\varepsilon)}$ which contains an $\varepsilon$-self-matching contains two sub-intervals $I_{1}$ and $I_{2}$ of size $\frac{c\log n}{\varepsilon\log(1/\varepsilon)}$ where there is a matching of size $\frac{c\log n}{2\log(1/\varepsilon)}$ between $I_{1}$ and $I_{2}$. Then, Step (12) can be revised by upper-bounding the probability of a long interval having an $\varepsilon$-self-matching by a union bound over the probability of pairs of its subintervals having a dense matching. Namely, for $l>100\frac{c\log n}{\varepsilon\log(1/\varepsilon)}$, let us denote the event of $S[i,i+l)$ containing a $\varepsilon$-self-matching by $A_{i,l}$. Then,

[TABLE]

where first inequality follows from the fact there can be at most $n^{2}$ pairs of intervals of size $\frac{c\log n}{\varepsilon\log(1/\varepsilon)}$ in $S$ and the number of all possible matchings of size $\frac{c\log n}{\log(1/\varepsilon)}$ between them is at most ${\varepsilon^{-1}c\log n/\log(1/\varepsilon)\choose c\log n/2\log(1/\varepsilon)}^{2}$. Further, for small enough $\varepsilon$, constant $c^{\prime}$ can be as large as one desires by setting constant $c$ large enough. Thus, Step (12) can be revised as:

[TABLE]

For an appropriately chosen $c$, $2-c^{\prime}<0$; hence, the later term vanishes as $n$ grows. Therefore, the conclusion $\mathbb{E}[\gamma]\leq O(\varepsilon)$ holds for the limited $\frac{\log n}{\log(1/\varepsilon)}$-wise independent string as well. ∎

Proof of Claim 6.12.

Let $M$ be a self-matching of size $l\varepsilon$ or more between $S$ and itself containing only bad edges. We chop $S$ into $\frac{l}{m}$ intervals of size $m$. On the one hand, the size of $M$ is greater than $l\varepsilon$ and on the other hand, we know that the size of $M$ is exactly $\sum_{i,j}|E_{i,j}|$ where $E_{i,j}$ denotes the number of edges between interval $i$ and $j$. Thus:

[TABLE]

Note that $\frac{|E_{i,j}|}{m}$ represents the density of edges between interval $i$ and interval $j$. Further, Since $M$ is monotone, there are at most $\frac{2l}{m}$ intervals for which $|E_{i,j}|\not=0$ and subsequently $\frac{|E_{i,j}|}{m}\not=0$. Hence, on the right hand side we have the average of $\frac{2l}{m}$ many non-zero terms which is greater than $\varepsilon/2$. So, there has to be some $i^{\prime}$ and $j^{\prime}$ for which:

[TABLE]

To analyze more accurately, if $l$ is not divisible by $m$, we simply throw out up to $m$ last elements of the string. This may decrease $\varepsilon$ by $\frac{m}{l}<\frac{\varepsilon}{100}$. ∎

Note that using the polynomial sample spaces of [24] Theorem 6.11 directly leads to a deterministic algorithm for finding a string of size $n$ with $\varepsilon$-self-matching property. For this one simply checks all possible points in the sample space of the $\frac{c\log n}{\log(1/\varepsilon)}$-wise independent strings and finds a string $S$ with $\gamma_{S}\leq\mathbb{E}[\gamma]=O(\varepsilon)$. In other words, using brute-force, one can find a string satisfying $O(\varepsilon)$-self-matching property in $O\left(|\Sigma|^{\frac{c\log n}{\log(1/\varepsilon)}}\right)=n^{O(1)}$.

Theorem 6.13**.**

There is a deterministic algorithm running in $n^{O(1)}$ that finds a string of length $n$ satisfying $\varepsilon$-self-matching property over an alphabet of size $O(\varepsilon^{-6})$.

6.3 Insdel Errors

Now, we provide an alternative indexing algorithm to be used along with $\varepsilon$-synchronization strings. Throughout the following sections, we let $\varepsilon$-synchronization string $S$ be sent as the synchronization string in an instance of $(n,\delta)$-indexing problem and string $S^{\prime}$ be received at the receiving end being affected by up to $n\delta$ insertions or deletions. Furthermore, let $d_{i}$ symbols be inserted into the communication and $d_{r}$ symbols be deleted from it.

The algorithm works as follows. On the first round, the algorithm finds the longest common subsequence between $S$ and $S^{\prime}$. Note that this common subsequence corresponds to a monotone matching $M_{1}$ between $S$ and $S^{\prime}$. On the next round, the algorithm finds the longest common subsequence between $S$ and the subsequence of unmatched elements of $S^{\prime}$ (those that have not appeared in $M_{1}$). This common subsequence corresponds to a monotone matching between $S$ and the elements of $S^{\prime}$ that do not appear in $M_{1}$. The algorithm repeats this procedure $\frac{1}{\beta}$ times to obtain $M_{1},\cdots,M_{1/\beta}$ where $\beta$ is a parameter that we will fix later.

In the output of this algorithm, $S^{\prime}[t_{i}]$ is decoded as $S[i]$ if and only if $S[i]$ is only matched to $S^{\prime}[t_{i}]$ in all $M_{1},\cdots,M_{1/\beta}$. Note that the longest common subsequence of two strings of length $O(n)$ can be found in $O(n^{2})$ using dynamic programming. Therefore, the whole algorithm runs in $O\left(n^{2}/\beta\right)$.

Now we proceed to analyzing the performance of the algorithm by bounding the number of misdecodings.

Theorem 6.14**.**

This decoding algorithm guarantees a maximum misdecoding count of $(n+d_{i}-d_{r})\beta+\frac{\varepsilon}{\beta}n$. More specifically, for $\beta=\sqrt{\varepsilon}$, the number misdecodings will be $O\left(n\sqrt{\varepsilon}\right)$ and running time will be $O\left(n^{2}/\sqrt{\varepsilon}\right)$.

Proof.

First, we claim that at most $(n+d_{i}-d_{r})\beta$ many of the symbols that have been successfully transmitted are not matched in any of $M_{1},\cdots,M_{1/\beta}$. Assume by contradiction that more than $(n+d_{i}-d_{r})\beta$ of the symbols that pass through the channel successfully are not matched in any of $M_{1},\cdots,M_{1/\beta}$. Then, there exists a monotone matching of size greater than $(n+d_{i}-d_{r})\beta$ between the unmatched elements of $S^{\prime}$ and $S$ after $\frac{1}{\beta}$ rounds of finding longest common substrings. Hence, size of any of $M_{i}$s is at least $(n+d_{i}-d_{r})\beta$. So, the summation of their sizes exceeds $(n+d_{i}-d_{r})\beta\times\frac{1}{\beta}=n+d_{i}-d_{r}=|S^{\prime}|$ which brings us to a contradiction.

Furthermore, as a result of Theorem 6.4, any of $M_{i}$s contain at most $\varepsilon n$ many incorrectly matched elements. Hence, at least $\frac{\varepsilon}{\beta}n$ many of the matched symbols are matched to incorrect index.

Hence, the total number of misdecodings can be bounded by $(n+d_{i}-d_{r})\beta+\frac{\varepsilon}{\beta}n$. ∎

6.4 Deletion Errors Only

We now introduce a very simple linear time streaming algorithm that decodes a received synchronization string of length $n$ which can be affected by up to $n\delta$ many deletions. Our scheme is guaranteed to have less than $\frac{\varepsilon}{1-\varepsilon}\cdot n\delta$ misdecodings.

Before proceeding to the algorithm description, let $d_{r}$ denote the number of symbols removed by adversary. As adversary is restricted to symbol deletion, each symbol received at the receiver corresponds to a symbol sent by the sender. Hence, there exists a monotone matching of size $|S^{\prime}|=n^{\prime}=n-d_{r}$ like $M=\{(t_{1},1),(t_{2},2),\cdots,(t_{n-d_{r}},n-d_{r})\}$ between $S$ and $S^{\prime}$ which matches each of the received symbols to their actual indices.

Our simple streaming algorithm greedily matches $S$ to the left-most possible subsequence of $S$. To put it another words, the algorithm matches $S^{\prime}[1]$ to $S[t^{\prime}_{1}]$ where $S[t^{\prime}_{1}]=S^{\prime}[1]$ and $t^{\prime}_{1}$ is as small as possible, then matches $S^{\prime}[2]$ to the smallest $t^{\prime}_{2}>t^{\prime}_{1}$ where $S[t^{\prime}_{2}]=S^{\prime}[2]$ and construct the whole matching $M^{\prime}$ by repeating this procedure. Note that as there is a matching of size $|S^{\prime}|$ between $S$ and $S^{\prime}$, the size of $M^{\prime}$ will be $|S^{\prime}|$ too.

This algorithm clearly works in a streaming manner and runs in linear time. To analyze the performance, we basically make use of the fact that $M$ and $M^{\prime}$ are both monotone matchings of size $|S^{\prime}|$ between $S$ and $S^{\prime}$. Therefore, $\bar{M}=\{(t_{1},t^{\prime}_{1}),(t_{2},t^{\prime}_{2}),\cdots,(t_{n-d_{r}},t^{\prime}_{n-d_{r}})\}$ is a monotone matching between $S$ and itself. Note that if $t_{i}\not=t^{\prime}_{i}$, then the algorithm has decoded the index $t_{i}$ incorrectly. Let $p$ be the number of indices $i$ where $t_{i}\not=t^{\prime}_{i}$. Then matching $\bar{M}$ consists of $n-d_{r}-p$ good pairs and $p$ bad pairs. Therefore, using Theorem 6.2

[TABLE]

This proves the following theorem:

Theorem 6.15**.**

Any $\varepsilon$-synchronization string along with the algorithm described in Section 6.4 form a linear-time streaming solution for deletion-only $(n,\delta)$-indexing problem guaranteeing $\frac{\varepsilon}{1-\varepsilon}\cdot n\delta$ misdecodings.

6.5 Insertion Errors Only

We now depart to another simplified case where adversary is restricted to only insert symbols. We propose a decoding algorithm whose output is guaranteed to be error-free and contain less than $\frac{n\delta}{1-\varepsilon}$ misdecodings.

Assume that $d_{i}$ symbols are inserted into the string $S$ to turn it in into $S^{\prime}$ of size $n+d_{i}$ on the receiving side. Again, it is clear that there exists a monotone matching $M$ of size $n$ like $M=\{(1,t_{1}),(2,t_{2}),\cdots,({n},t_{n})\}$ between $S$ and $S^{\prime}$ that matches each symbol in $S$ to its actual index when it arrives at the receiver.

The decoding algorithm we present, matches $S[i]$ to $S^{\prime}[t^{\prime}_{i}]$ in its output, $M^{\prime}$, if and only if in all possible monotone matchings between $S$ and $S^{\prime}$ that saturate $S$ (i.e., are of size $|S|=n$), $S[i]$ is matched to $S^{\prime}[t^{\prime}_{i}]$. Note that any symbol $S[i]$ that is matched to $S^{\prime}[t^{\prime}_{i}]$ in $M^{\prime}$ has to be matched to the same element in $M$; therefore, the output of this algorithm does not contain any incorrectly decoded indices; therefore, the algorithm is error-free.

Now, we are going to first provide a linear time approach to implement this algorithm and then prove an upper-bound of $\frac{d_{i}}{1-\varepsilon}$ on the number of misdecodings. To this end, we make use of the following lemma:

Lemma 6.16**.**

Let $M_{L}=\{(1,l_{1}),(2,l_{2}),\cdots,(n,l_{n})\}$ be a monotone matching between $S$ and $S^{\prime}$ such that $l_{1},\cdots,l_{n}$ has the smallest possible value lexicographically. We call $M_{L}$ the left-most matching between $S$ and $S^{\prime}$. Similarly, let $M_{R}=\{(1,r_{1}),\cdots,(n,r_{n})\}$ be the monotone matching such that $r_{n},\cdots,r_{1}$ has the largest possible lexicographical value. Then $S[i]$ is matched to $S^{\prime}[t^{\prime}_{i}]$ in all possible monotone matchings of size $n$ between $S$ and $S^{\prime}$ if and only if $(i,t^{\prime}_{i})\in M_{R}\cap M_{L}$.

This lemma can be proved by a simple contradiction argument. Our algorithm starts by computing left-most and right-most monotone matchings between $S$ and $S^{\prime}$ using the trivial greedy algorithm introduced in Section 6.4 on $(S,S^{\prime})$ and them reversed. It then outputs the intersection of these two matching as the answer. This algorithm clearly runs in linear time.

To analyze this algorithm, we bound the number of successfully transmitted symbols that the algorithm refuses to decode, denoted by $p$. To bound the number of such indices, we make use of the fact that $n-p$ elements of $S^{\prime}$ are matched to the same element of $S$ in both $M_{L}$ and $M_{R}$. As there are $p$ elements in $S$ that are matched to different elements in $S^{\prime}$ and there is a total of $n+d_{i}$ elements in $S^{\prime}$, there has to be at least $2p-[(n+d_{i})-(n-p)]=p-d_{i}$ elements in $S^{\prime}$ who are matched to different elements of $S$ in $M_{L}$ and $M_{R}$.

Consider the following monotone matching from $S$ to itself as follows:

[TABLE]

Note that monotonicity follows the fact that both $M_{L}$ and $M_{R}$ are both monotone matchings between $S$ and $S^{\prime}$. We have shown that the size of the second set is at least $p-d_{i}$ and the size of the first set is by definition $n-p$. Also, all pairs in the first set are good pairs and all in the second one are bad pairs. Therefore, by Theorem 6.2:

[TABLE]

which proves the efficiency claim and gives the following theorem.

Theorem 6.17**.**

Any $\varepsilon$-synchronization string along with the algorithm described in Section 6.5 form a linear-time error-free solution for insertion-only $(n,\delta)$-indexing problem guaranteeing $\frac{1}{1-\varepsilon}\cdot n\delta$ misdecodings.

Finally, we remark that a similar non-streaming algorithm can be applied to the case of deletion-only errors. Namely, one can compute the left-most and right-most matchings between the received string and string that is supposed to be received and output the common edges. By a similar argument as above, one can prove the following:

Theorem 6.18**.**

Any $\varepsilon$-synchronization string along with the algorithm described in Section 6.5 form a linear-time error-free solution for deletion-only $(n,\delta)$-indexing problem guaranteeing $\frac{\varepsilon}{1-\varepsilon}\cdot n\delta$ misdecodings.

In the same manner as Theorem 1.1, we can derive the following theorem concerning deletion-only and insertion-only codes based on Theorems 6.17 and 6.18.

Theorem 6.19**.**

For any $\varepsilon>0$ and $\delta\in(0,1)$:

• •

There exists an encoding map $E:\Sigma^{k}\rightarrow\Sigma^{n}$ and a decoding map $D:\Sigma^{*}\rightarrow\Sigma^{k}$ such that if $x$ is a subsequence of $E(m)$ where $|x|\geq n-n\delta$ then $D(x)=m$. Further $\frac{k}{n}>1-\delta-\varepsilon$, $|\Sigma|=f(\varepsilon)$, and $E$ and $D$ are explicit and have linear running times in $n$.

• •

There exists an encoding map $E:\Sigma^{k}\rightarrow\Sigma^{n}$ and a decoding map $D:\Sigma^{*}\rightarrow\Sigma^{k}$ such that if $E(m)$ is a subsequence of $x$ where $|x|\leq n+n\delta$ then $D(x)=m$. Further $\frac{k}{n}>1-\delta-\varepsilon$, $|\Sigma|=f(\varepsilon)$, and $E$ and $D$ are explicit and have linear running times in $n$.

Finally, we remark that since indexing solutions offered in Theorems 6.17 and 6.18 are error free, it suffices to use good erasure codes along with synchronization strings to obtain Theorem 6.19.

6.6 Decoding Using the Relative Suffix Pseudo-Distance (RSPD)

In this section, we show how one can slightly improve the constants in the results obtained in Section 5.2 by replacing RSD with a related notion of “distance” between two strings introduced in [2]. We call this notion relative suffix pseudo-distance or RSPD both to distinguish it from our RSD relative suffix distance and also because RSPD is not a metric distance per se – it is neither symmetric nor satisfies the triangle inequality.

Definition 6.20** (Relative Suffix Pseudo-Distance (RSPD)).**

Given any two strings $c,\tilde{c}\in\Sigma^{*}$, the suffix distance between $c$ and $\tilde{c}$ is

[TABLE]

We derive our algorithms by proving a useful property of synchronization strings:

Lemma 6.21**.**

Let $S\in\Sigma^{n}$ be an $\varepsilon$-synchronization string and $\tilde{c}\in\Sigma^{m}$. Then there exists at most one $c\in\bigcup_{i=1}^{n}S[1..i]$ such that $RSPD(c,\tilde{c})\leq 1-\varepsilon$.

Before proceeding to the proof of Lemma 6.21, we prove the following lemma:

Lemma 6.22**.**

Let $RSPD(S,T)\leq 1-\varepsilon$, then:

For every $1\leq s\leq|S|$, there exists $t$ such that $ED\left(S[s,|S|],T\left[t,|T|\right]\right)\leq(1-\varepsilon)(|S|-s+1)$. 2. 2.

For every $1\leq t\leq|T|$, there exists $s$ such that $ED\left(S[s,|S|],T\left[t,|T|\right]\right)\leq(1-\varepsilon)(|S|-s+1)$.

Proof.

Part 1

Let $\tau$ be the string matching chosen in $RSPD(S,T)$. There exist some $r$ such that $del(\tau_{1}\left[r,|\tau_{1}|\right])=S[s,|S|]$. Note that $del(\tau_{2}[r,|\tau_{2}|])$ is a suffix of $T$. Therefore, there exists some $t$ such that $T[t,|T|]=del(\tau_{2}[r,|\tau_{2}|])$. Now,

[TABLE]

Part 2

Similarly, let $\tau$ be the string matching chosen in $RSPD(S,T)$. There exists some $r$ such that $del(\tau_{2}\left[r,|\tau_{2}|\right])=T[t,|T|]$. Now, $del(\tau_{1}[r,|\tau_{1}|])$ is a suffix of $S$. Therefore, there exists some $s$ such that $S[s,|S|]=del(\tau_{1}[r,|\tau_{1}|])$. Now, all the steps we took to prove equation (13) hold and the proof is complete. ∎

Proof of Lemma 6.21.

For a contradiction, suppose that there exist a $\tilde{c}$, $l$ and $l^{\prime}$ such that $l<l^{\prime}$ and $RSPD(S[1,l],\tilde{c})\leq 1-\varepsilon$ and $RSPD(S[1,l^{\prime}],\tilde{c})\leq 1-\varepsilon$. Now, using part 1 of Lemma 6.22, there exists $k$ such that $ED\left(S[l+1,l^{\prime}],\tilde{c}[k,|\tilde{c}|]\right)\leq(1-\varepsilon)(l^{\prime}-l)$. Further, part 2 of Lemma 6.22 gives that there exist $l^{\prime\prime}$ such that $ED\left(S[l^{\prime\prime}+1,l],\tilde{c}[k,|\tilde{c}|]\right)\leq(1-\varepsilon)(l-l^{\prime\prime})$. Hence,

[TABLE]

which contradicts the fact that $S$ is an $\varepsilon$-synchronization string. ∎

Lemma 6.21 implies a natural algorithm for decoding $\tilde{c}$: simply search over all prefixes of $S$ for the one with small enough suffix distance from $\tilde{c}$. We prove that this is possible in $O(n^{5})$ via dynamic programming.

Theorem 6.23**.**

Let $S\in\Sigma^{n}$ be an $\varepsilon$-synchronization string, and $\tilde{c}\in\Sigma^{m}$. Then Algorithm 3, given input $S$ and $\tilde{c}$, either returns the unique prefix $c$ of $S$ such that $RSPD(c,\tilde{c})\leq 1-\varepsilon$ or returns $\emptyset$ if no such prefix exists. Moreover, Algorithm 3 runs in time $O(n^{5})$; spending $O(n^{4})$ for each received symbol.

Proof.

To find $c$, we calculate the RSPD of $\tilde{c}$ and all prefixes of $S$ one by one. We only need to show that the RSPD of two strings of length at most $n$ can be found in $O(n^{3})$. We do this using dynamic programming. Let us try to find $RSPD(s,t)$. Further, let $s(i)$ represent the suffix of $s$ of length $i$ and $t(j)$ represent the suffix of $t$ of length $j$. Now, let $d[i][j][l]$ be the minimum string matching $(\tau_{1},\tau_{2})$ from $s(i)$ to $t(j)$ such that $sc(\tau_{1})=l$. In other words,

[TABLE]

where $\tau$ is a string matching for $s(i)$ and $t(j)$. Note that for any $\tau:s(i)\rightarrow t(j)$, one the following three scenarios might happen:

$\tau_{1}(1)=\tau_{2}(1)=s\left(|s|-(i-1)\right)=t(|t|-(j-1))$: In this case, removing the first elements of $\tau_{1}$ and $\tau_{2}$ gives a valid string matching from $s(i-1)$ to $t(j-1)$. 2. 2.

$\tau_{1}(1)=*\text{ and }\tau_{2}(1)=t(|t|-(j-1))$: In this case, removing the first element of $\tau_{1}$ and $\tau_{2}$ gives a valid string matching from $s(i)$ to $t(j-1)$. 3. 3.

$\tau_{2}(1)=*\text{ and }\tau_{1}(1)=s(|s|-(i-1))$: In this case, removing the first element of $\tau_{1}$ and $\tau_{2}$ gives a valid string matching from $s(i-1)$ to $t(j)$.

This implies that

[TABLE]

Hence, one can find $RSPD(s,t)$ by minimizing $d[|s|][|t|][l]$ over all possible values of $l$, as Algorithm 3 does in Step 4 for all prefixes of $S$. Finally, Algorithm 3 returns the prefix $c$ such that $RSPD(s,t)\leq\frac{1-\varepsilon}{2}$ if one exists, and otherwise it returns $\emptyset$. ∎

We conclude by showing that if an $\varepsilon$-synchronization string of length $n$ is used along with the minimum RSPD algorithm, the number of misdecodings will be at most $\frac{n\delta}{1-\varepsilon}$.

Theorem 6.24**.**

Suppose that $S$ is an $\varepsilon$-synchronization string of length $n$ over alphabet $\Sigma$ that is sent over an insertion-deletion channel with $c_{i}$ insertions and $c_{d}$ deletions. By using Algorithm 3 for decoding the indices, the outcome will contain less than $\frac{c_{i}}{1-\varepsilon}+\frac{c_{d}\varepsilon}{1-\varepsilon}$ misdecodings.

Proof.

The proof of this theorem is similar to the proof of Theorem 5.10. Let prefix $S[1,i]$ be sent through the channel $S_{\tau}[1,j]$ be received on the other end as the result of adversary’s set of actions $\tau$. Further, assume that $S_{\tau}[j]$ is successfully transmitted and is actually $S[i]$ sent be the other end. We first show that $RSPD(S[1,i],S^{\prime}[1,j])$ is less than the relative suffix error density:

[TABLE]

Now, using Theorem 5.14, we know that the relative suffix error density is smaller than $1-\varepsilon$ upon arrival of all but at most $\frac{c_{i}+d_{d}}{1-\varepsilon}-c_{d}$ of successfully transmitted symbols. Along with Lemma 6.21, this results into the conclusion that the minimum RSPD decoding guarantees $\frac{c_{i}}{1-\varepsilon}+c_{d}\left(\frac{1}{1-\varepsilon}-1\right)$ misdecodings. This finishes the proof of the theorem. ∎

Acknowledgements

The authors thank Ellen Vitercik and Allison Bishop for valuable discussions in the early stages of this work.