Streaming for Aibohphobes: Longest Palindrome with Mismatches

TL;DR

This paper introduces streaming algorithms for finding long approximate palindromes with mismatches, providing space-efficient solutions with approximation guarantees and lower bounds, applicable to DNA analysis and document repair.

Contribution

It presents the first streaming algorithms for longest $d$-near-palindromes with approximation guarantees and space complexity analysis, including lower bounds and exact two-pass algorithms.

Findings

Algorithm achieves $(1+psilon)$-approximation with space $\u2113rac{d\u2217 ext{polylog}(n)}{\u2208}$

Lower bounds show $\u2205 d\u2207 ext{polylog}(n)$ space necessary for estimation

Exact two-pass algorithm uses $\u2206 d^2 ext{polylog}(n)$ space

Abstract

A palindrome is a string that reads the same as its reverse, such as "aibohphobia" (fear of palindromes). Given an integer $d>0$, a $d$-near-palindrome is a string of Hamming distance at most $d$ from its reverse. We study the natural problem of identifying a longest $d$-near-palindrome in data streams. The problem is relevant to the analysis of DNA databases, and to the task of repairing recursive structures in documents such as XML and JSON. We present an algorithm that returns a $d$-near-palindrome whose length is within a multiplicative $(1+\epsilon)$-factor of the longest $d$-near-palindrome. Our algorithm also returns the set of mismatched indices of the $d$-near-palindrome, using $\mathcal{O}\left(\frac{d\log^7 n}{\epsilon\log(1+\epsilon)}\right)$ bits of space, and $\mathcal{O}\left(\frac{d\log^6 n}{\epsilon\log(1+\epsilon)}\right)$ update time per arriving symbol. We show that $\Omega(d\log n)$ space is necessary for estimating the length of longest $d$-near-palindromes with high probability. We further obtain an additive-error approximation algorithm and a comparable lower bound, as well as an exact two-pass algorithm that solves the longest $d$-near-palindrome problem using $\mathcal{O}\left(d^2\sqrt{n}\log^6 n\right)$ bits of space.

Tables

Table 1. Table 1: Summary of our results and comparison to related work

	Space for Algorithms		Lower Bounds
Model	$d$-Near-Palindrome	Palindrome	$d$-Near-Palindrome	Palindrome
$1$-Pass, Multiplicative $\left(1+ϵ\right)$	$𝒪\left(\frac{d{\log }^{7}n}{ϵ\log \left(1+ϵ\right)}\right)$	$𝒪\left(\frac{{\log }^{2}n}{ϵ\log \left(1+ϵ\right)}\right)$[BEMS14]	$\Omega \left(d\log n\right)$	$\Omega \left(\frac{\log n}{\log \left(1+ϵ\right)}\right)$[GMSU16]
$1$-Pass, Additive $E$	$𝒪\left(\frac{dn{\log }^{6}n}{E}\right)$	$𝒪\left(\frac{n\log n}{E}\right)$[BEMS14]	$\Omega \left(\frac{dn}{E}\right)$	$\Omega \left(\frac{n}{E}\right)$[BEMS14]
$2$-Pass, Exact	$𝒪\left(d^2\sqrt{n}{\log }^{6}n\right)$	$𝒪\left(\sqrt{n}\log n\right)$[BEMS14]	-	-

Equations

$${\varphi }^F(S)=\left(\sum _{x=1}^{|S|}S[x]\cdot B^x\right)\mathrm{mod}P,{\varphi }^R(S)=\left(\sum _{x=1}^{|S|}S[x]\cdot B^{-x}\right)\mathrm{mod}P.$$

$$1.$$

$$2.$$

$$3.$$

$${\varphi }_{a,b}^F(S)={\varphi }^F(S_{a,b})$$

$${\varphi }_{a,b}^R(S)={\varphi }^R(S_{a,b})$$

$$1.$$

$$2.$$

$$3.$$

$$\frac{\frac{(34-2)d}{\beta }{\log }^{2}n}{\log \left(\frac{34d}{\beta }{\log }^{2}n\right)}\ge \frac{\frac{32d}{\beta }{\log }^{2}n}{\log n}\ge \frac{32d}{\beta }\log n.$$

$${\varphi }_{a,b}^F(S)=\left(\sum _{x\equiv a\mathrm{mod}b}f(S[x])\cdot B^{\lceil x/b\rceil }\right)\mathrm{mod}P.$$

$$2(m-c)\ge 2m-2y+4(1+\alpha {)}^{k-1}-2\alpha (1+\alpha {)}^{k-3}=2(1+\alpha {)}^{k-1}-2\alpha (1+\alpha {)}^{k-3}.$$

$$\frac{{\ell }_{max}}{\tilde{\ell }}\le \frac{2(1+\alpha {)}^k}{2(1+\alpha {)}^{k-1}-2\alpha (1+\alpha {)}^{k-3}}=\frac{(1+\alpha {)}^3}{(1+\alpha {)}^2-\alpha }\le (1+\alpha {)}^2=1+ϵ.$$

$$\frac{\frac{|X|}{4}\left(\genfrac{}{}{0ex}{}{n/4-2d}{d}\right)}{|X|\left(\left(\genfrac{}{}{0ex}{}{n/4}{d}\right)+\left(\genfrac{}{}{0ex}{}{n/4}{d+1}\right)\right)}$$

$$\ge \frac{d+1}{n+4}{\left(\frac{n/4-3d+1}{n/4-d+1}\right)}^d=\frac{d+1}{n+4}{\left(1-\frac{2d}{n/4-d+1}\right)}^d$$

$$\ge \frac{d+1}{n+4}\left(1-\frac{2d^2}{n/4-d+1}\right)\ge \frac{1}{n}$$

$$\sum _{i=0}^{\frac{|\mathcal{I}|}{2}-1}\left(\genfrac{}{}{0ex}{}{|\mathcal{I}|}{i}\right)\left(\genfrac{}{}{0ex}{}{n^{\prime }/2-|\mathcal{I}|}{d-i}\right)+\sum _{\frac{|\mathcal{I}|}{2}+1}^d\left(\genfrac{}{}{0ex}{}{|\mathcal{I}|}{i}\right)\left(\genfrac{}{}{0ex}{}{n^{\prime }/2-|\mathcal{I}|}{d+1-i}\right)$$

$$\ge \left(\sum _{i=0}^d\left(\genfrac{}{}{0ex}{}{|\mathcal{I}|}{i}\right)\left(\genfrac{}{}{0ex}{}{n^{\prime }/2-|\mathcal{I}|}{d-i}\right)\right)-\left(\genfrac{}{}{0ex}{}{|\mathcal{I}|}{|\mathcal{I}|/2}\right)\left(\genfrac{}{}{0ex}{}{n^{\prime }/2-|\mathcal{I}|}{d-|\mathcal{I}|/2}\right).$$

$$\left(\genfrac{}{}{0ex}{}{|\mathcal{I}|}{|\mathcal{I}|/2}\right)\left(\genfrac{}{}{0ex}{}{n^{\prime }/2-|\mathcal{I}|}{d-|\mathcal{I}|/2}\right)\le \left(\genfrac{}{}{0ex}{}{2}{1}\right)\left(\genfrac{}{}{0ex}{}{n^{\prime }/2-2}{d-1}\right)=\left(\genfrac{}{}{0ex}{}{n^{\prime }/2}{d}\right)\frac{2d(n^{\prime }/2-d)}{(n^{\prime }/2)(n^{\prime }/2-1)}\le \left(\genfrac{}{}{0ex}{}{n^{\prime }/2}{d}\right)\frac{1}{\sqrt{n^{\prime }}},$$

$$\frac{\frac{|X|}{16}\left(\genfrac{}{}{0ex}{}{n^{\prime }/2}{d}\right)}{|X|\left(\left(\genfrac{}{}{0ex}{}{n^{\prime }/2}{d}\right)+\left(\genfrac{}{}{0ex}{}{n^{\prime }/2}{d+1}\right)\right)}\ge \frac{\left(\genfrac{}{}{0ex}{}{n^{\prime }/2}{d}\right)}{16\left(\left(\genfrac{}{}{0ex}{}{n^{\prime }/2}{d}\right)+(d+1)\left(\genfrac{}{}{0ex}{}{n^{\prime }/2}{d}\right)\right)}\ge \frac{1}{32d}.$$

$$s(x,y)={\text{1}}^E{x}_1{\text{1}}^{\frac{E}{d}}{x}_2\dots {\text{1}}^{\frac{E}{d}}{x}_{n^{\prime }/2}{y}_{n^{\prime }/2}{\text{1}}^{\frac{E}{d}}\dots y_2{\text{1}}^{\frac{E}{d}}{y}_1{\text{1}}^E,$$

Full text

We study the natural problem of identifying a longest $d$-near-palindrome in data streams. The problem is relevant to the analysis of DNA databases, and to the task of repairing recursive structures in documents such as XML and JSON.

We present an algorithm that returns a $d$-near-palindrome whose length is within a multiplicative $(1+\epsilon)$-factor of the longest $d$-near-palindrome. Our algorithm also returns the set of mismatched indices of the $d$-near-palindrome, using $\mathcal{O}\left(\frac{d\log^{7}n}{\epsilon\log(1+\epsilon)}\right)$ bits of space, and $\mathcal{O}\left(\frac{d\log^{6}n}{\epsilon\log(1+\epsilon)}\right)$ update time per arriving symbol. We show that $\Omega(d\log n)$ space is necessary for estimating the length of longest $d$-near-palindromes with high probability.

We further obtain an additive-error approximation algorithm and a comparable lower bound, as well as an exact two-pass algorithm that solves the longest $d$-near-palindrome problem using $\mathcal{O}\left(d^{2}\sqrt{n}\log^{6}n\right)$ bits of space.

1 Introduction

A palindrome is a string that reads the same as its reverse, such as the common construct “racecar”, or the deliberate construct “aibohphobia”. Given a metric and an integer $d>0$, we say that a string is a $d$-near-palindrome if it is at distance at most $d$ from its reverse. In this paper, we study the problem of identifying the longest $d$-near-palindrome substring in the streaming model, under the Hamming distance. In the streaming model, the input data arrives one symbol at a time, and we are allowed to perform computation using only a small amount of working memory. Specifically, our goal is to approximate the length of a longest near-palindrome in a string of length $n$, using only $o(n)$ space. A related question regarding approximating the length of a longest palindrome in RNA sequences under removal of elements was explicitly asked at the Bertinoro Workshop on Sublinear Algorithms 2014 [Sub].

Finding near-palindromes is widely motivated in string processing of databases relevant to bioinformatics. Specifically, since the development of the Human Genome Project, advances in biological algorithms have quickened the sequencing for genes and proteins, leading to increasingly large databases of strings representing both nucleic acids for DNA or RNA, and amino acids for proteins. Tools to analyze these sequences, such as the basic local alignment search tool (BLAST) [AGM*+*90], often require the removal of “low-complexity” regions (long repetitive or palindromic structures). However, these long sequences frequently contain small perturbations through mutation or some other form of corruption (including human error), so that identifying “near”-palindromes under either Hamming distance or edit distance is important for preprocessing sequences before applying heuristic tools. In particular, the streaming model is relevant to contemporary data-sequencing technologies for near-palindromes, as further discussed in [CCH04, HMS*+*07].

Our contributions

We initiate the study of finding near-palindromes in the streaming model, and provide several algorithms for the longest near-palindrome substring.

Given a stream $S$ of length $n$ and integer $d=o(\sqrt{n})$, let $\ell_{max}$ be the length of a longest $d$-near-palindrome substring in $S$.

Theorem 1.1

There exists a one-pass streaming algorithm that returns a $d$-near-palindrome of length at least $\frac{1}{1+\epsilon}\cdot{\ell_{max}}$, with probability $1-\frac{1}{n}$. The algorithm uses $\mathcal{O}\left(\frac{d\log^{7}n}{\epsilon\log(1+\epsilon)}\right)$ bits of space and update time $\mathcal{O}\left(\frac{d\log^{6}n}{\epsilon\log(1+\epsilon)}\right)$ per arriving symbol.

Theorem 1.2

There exists a one-pass streaming algorithm that returns a $d$-near-palindrome of length at least $\ell_{max}-E$, with probability $1-\frac{1}{n}$. The algorithm uses $\mathcal{O}\left(\frac{dn\log^{6}n}{E}\right)$ bits of space and update time $\mathcal{O}\left(\frac{dn\log^{5}n}{E}\right)$ per arriving symbol.

If two passes over the stream are allowed, one can find an exact longest $d$-near-palindrome.

Theorem 1.3

There exists a two-pass streaming algorithm that returns a $d$-near-palindrome of length $\ell_{max}$, with probability $1-\frac{1}{n}$. It uses $\mathcal{O}\left(d^{2}\sqrt{n}\log^{6}n\right)$ bits of space and $\mathcal{O}\left(d^{2}\sqrt{n}\log^{5}n\right)$ update time per arriving symbol.

We complement our results with lower bounds for randomized algorithms.

Theorem 1.4

Let $d=o(\sqrt{n})$. Any randomized streaming algorithm that returns an estimate $\hat{\ell}$ of the length of the longest $d$-near-palindrome, where $\hat{\ell}\leq\ell_{max}\leq(1+\epsilon)\hat{\ell}$, with probability at least $1-\frac{1}{n}$, must use $\Omega\left(d\log n\right)$ bits of space.

Theorem 1.5

Let $d=o(\sqrt{n})$ and $E>d$ be an integer. Any randomized streaming algorithm that returns an estimate $\hat{\ell}$ of the length of the longest $d$-near-palindrome, where $\hat{\ell}\leq\ell_{max}\leq\hat{\ell}+E$, with probability at least $1-\frac{1}{n}$, must use $\Omega\left(\frac{dn}{E}\right)$ bits of space.

A summary of our results and comparison with related work appears in Table 1.

Background and Related Work

Our techniques extend previous work on the Longest Palindromic Substring Problem, the Pattern Matching Problem, and the $d$-Mismatch Problem in the streaming model.

In the Longest Palindromic Substring Problem, the goal is to output a longest palindromic substring of an input of length $n$, while minimizing the computation space. Manacher [Man75] introduces a linear-time online algorithm that reports whether all symbols seen at the time of query form a palindrome. Berenbrink et al.  [BEMS14] achieve $\mathcal{O}\left(\frac{\log^{2}n}{\epsilon\log(1+\epsilon)}\right)$ space for multiplicative error $(1+\epsilon)$, and show a space lower bound for algorithms with additive error. Gawrychowski et al.  [GMSU16] recently generalize the aforementioned lower bounds for additive error, and also produce a space lower bound of $\Omega\left(\frac{\log n}{\log(1+\epsilon)}\right)$ for algorithms with multiplicative error $(1+\epsilon)$.

In the Pattern Matching Problem, one is given a pattern of length $m$ and the goal is to output all occurrences of the pattern in the input string, while again minimizing space or update time. In order to achieve space sublinear in the size of the input, many pattern matching streaming algorithms use Karp-Rabin fingerprints [KR87]. Porat and Porat [PP09] present a randomized algorithm for exact pattern matching using $\mathcal{O}\left(\log m\right)$ space and $\mathcal{O}\left(\log m\right)$ update time, which Breslauer and Galil [BG14] further improve to constant update time. For a more comprehensive survey on pattern matching, see [AG97].

In the related $d$-Mismatch Problem, one is given a pattern of length $m$ and the goal is to find all substrings of the input that are at most Hamming distance $d$ from the pattern. A line of exciting work (e.g., [ALP04, PP09, CEPP11, AGMP13]) culminates in a recent algorithm by Clifford et al. [CFP*+*16] that uses $\mathcal{O}\left(d^{2}\,{{\mathsf{polylog}}}\,m\right)$ space and $\mathcal{O}\left(\sqrt{d}\log d+{{\mathsf{polylog}}}\,m\right)$ update time per arriving symbol.

For several other metrics, Clifford et al.  [CJPS13] show that linear space is necessary for algorithms identifying substrings with distance at most $d$ from a given pattern. Similarly, Andoni et al. [AGMP13] prove that any sketch estimating the edit distance between two strings requires space almost linear in the inputs. For time bounds, Backurs and Indyk [BI15] show that the strong Exponential Time Hypothesis implies the general edit distance problem cannot be solved in time better than $n^{2-\epsilon}$. On the positive side, Chakraborty et al. [CGK16] give a low distortion embedding from edit distance to Hamming distance, and Belazzougui and Zhang [BZ16] provide the first streaming algorithm for computing edit distance using $\mathcal{O}\left(d^{8}\log^{5}n\right)$ space, given the promise that the edit distance is at most $d$.

2 Preliminaries

We denote by $[n]$ the set $\{1,2,\ldots,n\}$. We assume an input stream of length $n$ over alphabet $\Sigma$. Given a string $S[1,\dots,n]$, we denote its length by $|S|$, its $i^{\mbox{\tiny{{th}}}}$ character by $S[i]$ or $S_{i}$, and the substring between locations $i$ and $j$ (inclusive) by $S[i,j]$.

The Hamming distance between $S$ and $T$, denoted ${\mathsf{HAM}}(S,T)$, is the number of indices whose symbols do not match: {\mathsf{HAM}}(S,T)=\Big{|}\{i\mid S[i]\neq T[i]\}\Big{|}. We denote the concatenation of $S$ and $T$ by $S\circ T$. Each index $i$ such that $S[i]\neq S[n-i+1]$ is a mismatch. We say $S$ is a $d$-near-palindrome if ${\mathsf{HAM}}(S,S^{R})\leq d$. Without loss of generality, our algorithms assume the lengths of $d$-near-palindromes are even, since for any odd length $d$-near-palindrome, we may apply the algorithm to $S[1]S[1]S[2]S[2]\cdots S[n]S[n]$ instead of $S[1,n]$.

Definition 2.1** (Karp-Rabin Fingerprint)**

For a string $S$, prime $P$ and integer $B$ with $1\leq B<P$, the Karp-Rabin forward and reverse fingerprints [KR87] are defined as follows:

[TABLE]

Karp-Rabin Fingerprints have the following easily verifiable properties:

[TABLE]

We use Karp-Rabin Fingerprints for certain subpatterns of $S$, as in [CFP*+*16]. For a string $S$ and integers $a\leq b$, define the first-level subpattern $S_{a,b}$ to be the subsequence $S[a]S[a+b]S[a+2b]\ldots$. In this case, define $S^{R}_{a,b}=(S_{a,b})^{R}$ (as opposed to $(S^{R})_{a,b}$). Similarly, define $S_{a,b}[x,y]=S_{a,b}\cap S[x,y]$ (as opposed to $(S[x,y])_{a,b}$). Then for $1\leq a\leq b$, define the fingerprints for $S_{a,b}$ and its reverse:

[TABLE]

For an example, see Figure 1.

Given a first-level subpattern $T=S_{a,b}=S[a]S[a+b]S[a+2b]\ldots$ and integers $r\leq s$, define the second-level subpattern $T_{r,s}=T[r]T[r+s]T[r+2s]\ldots$. Observe that $T_{r,s}=S_{a+rb,sb}$ and thus, second-level subpatterns are simply more refined first-level subpatterns. For an example, see Figure 2.

Observe the following properties of fingerprints on first-level and second-level subpatterns:

[TABLE]

We also use the following application of the Prime Number Theorem from [CFP*+*16]:

Lemma 2.2

(Adaptation of Lemma 4.1 [CFP*+*16]) Given two distinct integers $a,b\in[n]$ and a random prime number $p\in\left[\frac{d}{\beta}\log^{2}n,\frac{34d}{\beta}\log^{2}n\right]$ where $\beta=\frac{1}{16}$, then \mathbf{Pr}\Big{[}a\equiv b\bmod{p}\Big{]}\leq\frac{\beta}{32d}.

**Proof :  ** By the Prime Number Theorem (Corollary $1$ of [RS62]) it follows that the number of primes in $\left[\frac{d}{\beta}\log^{2}n,\frac{34d}{\beta}\log^{2}n\right]$ is at least

[TABLE]

If $a\equiv b\bmod{p}$, then $p$ is a divisor of $|a-b|$. Furthermore, by assumption, $p$ is prime. Thus, the probability that $p$ is one of the prime divisors of $|a-b|\leq n-1$ is at most $\frac{\log n}{(32d/\beta)\log n}=\frac{\beta}{32d}$, since $|a-b|$ can have at most $\log n$ prime divisors. $\Box$

Finally, we remark that problems in bioinformatics, such as the RNA Folding Problem [Sub], use the following notion of complementary palindromes:

Definition 2.3

Let $f\,:\,\sum\rightarrow\sum$ be a pairing of symbols in the alphabet. A string $S\in\sum^{n}$ is a complementary palindrome if $S[x]=f(S[n+1-x])$ for all $1\leq x\leq n$.

Our algorithms can be modified to recognize complementary palindromes with the same space usage and update time. Indeed, we only need to modify the forward fingerprints to use $f(S[x])$ instead of $S[x]$:

[TABLE]

3 Overview and Techniques

One-pass Multiplicative Approximation Algorithm

Our algorithm combines and extends ideas and techniques from the solution to the $d$-Mismatch Problem in [CFP*+*16] and the solution to the Longest Palindrome Problem in [BEMS14].

As the stream progresses, we keep a set of checkpoints $\mathcal{C}$, where each $c\in\mathcal{C}$ is an index for which we search $d$-near-palindromes to begin. We also maintain a sliding window that contains the $2d$ most recently seen symbols, as shown in Figure 3. The sliding window identifies any $d$-near-palindrome of length at most $2d$. It also guesses that the midpoint of the sliding window is the midpoint of a potential $d$-near-palindrome of length $>2d$.

We keep an estimate $\tilde{\ell}$ of the length $\ell_{max}$ of the longest $d$-near-palindrome seen throughout the stream, as well as its starting index $c_{start}$, and the locations of the mismatches, a set of size at most $d$. Upon reading symbol $S[x]$ of the stream, we call procedure ${\mathsf{NearPalindrome}}$ to see if $S[c_{i},x]$ is a $d$-near-palindrome, for each checkpoint $c_{i}$ such that $x-c_{i}>\tilde{\ell}$, as in Figure 4.

Using the framework of [BEMS14], we create and update checkpoints throughout the stream so that we find a $d$-near-palindrome of length at least $\frac{\ell_{max}}{1+\epsilon}$, as in Figure 5.

The algorithm in [BEMS14] also maintains a list of potential midpoints associated with each checkpoint. Although this list can be linear in size, it satisfies nice structural results that can be used to succinctly represent the list of candidate midpoints. However, directly adapting these structural results to our setting would incur an extra factor of $d$ in our space complexity. We avoid this extra factor by circumventing the list of candidate midpoints in the one-pass algorithms altogether.

We now overview the procedure ${\mathsf{NearPalindrome}}$ that we use repeatedly in our algorithms. The procedure returns whether $S[c_{i},x]$ is a $d$-near-palindrome, and if so, it returns the corresponding mismatches.

The procedure ${\mathsf{NearPalindrome}}$ adapts the data structures outlined in [CFP*+*16]. Recall that in the $d$-Mismatch Problem, we are given a pattern $R$ and a text $S$ and the algorithm is required to output all indices $x$ such that ${\mathsf{HAM}}(R,S[x,x+|R|-1])\leq d$. While the pattern is fixed in the $d$-Mismatch Problem, here we essentially use variable-length patterns. Namely, we check whether ${\mathsf{HAM}}(S[c_{i},x],S^{R}[c_{i},x])\leq d$ for each checkpoint $c_{i}$ by maintaining dynamic sets of fingerprints.

The procedure has two stages. In the first stage it eliminates strings $T$ with ${\mathsf{HAM}}(T,T^{R})\geq 2d$, while in the second stage it eliminates strings with $d<{\mathsf{HAM}}(T,T^{R})<2d$. This can be achieved by estimating the distance between $T$ and $T^{R}$ using fingerprints of equivalence classes modulo different primes.

Intuitively, picking random primes distributes the mismatches into different equivalence classes. For each prime $p$, the procedure estimates the number of mismatches by comparing the fingerprints of the substrings whose indices are in the same congruence class modulo $p$ with the reverse fingerprints, namely $T_{r,p}$ and $T^{R}_{r,p}$ for all $1\leq r\leq p$. Denote by $T_{r,p}$ and $T^{R}_{r,p}$ the first-level fingerprints.

By the second stage we are only left with the strings with a small number of mismatches. In order to recover the mismatches, one needs to refine each subpattern $\tilde{T}=T_{r,p}$ by picking smaller primes $p^{\prime}$, and comparing the fingerprints of the strings $\tilde{T}_{r^{\prime},p^{\prime}}$ and $\tilde{T}^{R}_{r^{\prime},p^{\prime}}$ for all $1\leq r^{\prime}\leq p^{\prime}$. Denote by $\tilde{T}_{r^{\prime},p^{\prime}}$ and $\tilde{T}^{R}_{r^{\prime},p^{\prime}}$ the second-level fingerprints (see Figure 2).

In the first stage, we sample $2\log n$ primes uniformly at random from $\left[\frac{d}{\beta}\log^{2}n,\frac{34d}{\beta}\log^{2}n\right]$, where $\beta=1/16$. Each prime generates $p$ subpatterns containing positions in the same equivalence class$\pmod{p}$. Therefore, there are $\mathcal{O}\left(d\,\log^{3}n\right)$ first-level subpatterns. In the second stage, we take all primes in $[\log n,3\log n]$ that together with the primes picked in the first stage generate a total of $\mathcal{O}\left(d\,\log^{5}n\right)$ second-level subpatterns.

Finally, we assume throughout the paper that the fingerprints of any subpattern do not fail. Since there are at most $n^{3}$ subpatterns, and the probability that a particular fingerprint fails is at most $\frac{1}{n^{5}}$ for $P\in[n^{5},n^{6}]$ (by Theorem 1 in [BG14]), then by a union bound, the probability that no fingerprint fails is least $1-\frac{1}{n^{2}}$.

Our choice of parameters is more space-efficient compared to the data structure given by [CFP*+*16], which uses $\mathcal{O}\left(d^{2}\log^{7}n\right)$ space, since we no longer need the starting index to slide.

We also note that [PL07] gives another data structure for determining the Hamming distance between two strings. That data structure is more space efficient than the data structure above given by [CFP*+*16], but seemingly does not suffice for our problem, as it does not support concatenation, which is needed for maintaining the checkpoints.

One-pass Additive Approximation and Two-Pass Exact Algorithms

To obtain the one-pass additive approximation, we modify our checkpoints, so that they appear in every ${\left\lfloor\frac{E}{2}\right\rfloor}$ positions. Hence, the longest $d$-near-palindrome must have some checkpoint within ${\left\lfloor\frac{E}{2}\right\rfloor}$ positions of it, and the algorithm will recover a $d$-near-palindrome with length at least $\ell_{max}-E$.

To obtain the two-pass exact algorithm, we set $E=\sqrt{n}$ and modify the additive error algorithm so that it returns a list $\mathcal{L}$ of candidate midpoints of $d$-near-palindromes. Moreover, we show a structural result in Lemma 6.2, which allows us to compress certain substrings in the first pass, so that the second pass can recover mismatches for any potential $d$-near-palindromes within these substrings.

In the second pass, we carefully keep track of the $\frac{\sqrt{n}}{2}$ characters before the starting positions of long $d$-near-palindromes identified in the first pass. We use the compressed information from the first pass to reconstruct the fingerprints and calculate the number of mismatches within these long $d$-near-palindromes identified in the first pass. However, the actual $d$-near-palindromes may extend beyond the estimate returned in the first pass. Thus, we compare the $\frac{\sqrt{n}}{2}$ characters after the $d$-near-palindromes identified in the first pass with the $\frac{\sqrt{n}}{2}$ characters that we track. This allows us to exactly identify the longest $d$-near-palindrome during the second pass.

Lower Bounds

To show lower bounds for randomized algorithms solving the $d$-near palindrome problem we use Yao’s Principle [Yao77], and construct distributions for which any deterministic algorithm fails with significant probability unless given a certain amount of space. We first show that providing a $(1+\epsilon)$ approximation to the length of longest $d$-near-palindromes inheritly solves the problem of exactly identifying whether two strings have Hamming distance at most $d$. This problem has been useful in proving other related lower bounds [EGSZ17, GSZ17] and may be of independent interest. We carefully construct hard distributions for this problem, using ideas from [GMSU16], and show via counting arguments that deterministic algorithms using a little of space will fail with significant probability on inputs from these distributions.

4 One-Pass Streaming Algorithm with Multiplicative Error $(1+\epsilon)$

In this section, we prove Theorem 1.1. Namely, we provide a one-pass streaming algorithm with multiplicative error $(1+\epsilon)$, using $\mathcal{O}\left(\frac{d\log^{7}n}{\epsilon\log(1+\epsilon)}\right)$ bits of space.

4.1 Algorithm

As described in the overview, similar to [BEMS14], we maintain a sliding window of size $2d$, along with master fingerprints, and a series of checkpoints. From the sliding window, we observe every $d$-near-palindrome with length at most $2d$, as well as every candidate midpoints. Then, prior to seeing element $S[x]$ in the stream, we initialize the following in memory:

Initialization:

(1)

Pick a prime $P$ from $[n^{5},n^{6}]$ and an integer $B<P$ (the modulo and the base of the Karp-Rabin fingerprints, respectively).

(2)

For the first-level fingerprints, create set $\mathcal{P}$ consisting of $2\log n$ primes $p_{1},p_{2},\ldots,p_{2\log n}$ sampled independently and uniformly at random from $\left[\frac{d}{\beta}\log^{2}n,\frac{34d}{\beta}\log^{2}n\right]$, where $\beta=\frac{1}{16}$.

(3)

For the second-level fingerprints, let $\mathcal{Q}$ be the set of primes in $[\log n,3\log n]$.

(4)

Initialize a sliding window of size $2d$.

(5)

Initialize the sets of Master Fingerprints, $\mathcal{F}^{F}$ and $\mathcal{F}^{R}$:

(a)

Set $\phi^{F}_{r,p}(S)=0$, $\phi^{R}_{r,p}(S)=0$ for all $p\in\mathcal{P}$ and $1\leq r\leq p$.

(b)

Set $\phi^{F}_{r^{\prime},pq}(S)=0$, $\phi^{R}_{r^{\prime},pq}(S)=0$ for all $p\in\mathcal{P}$, $q\in\mathcal{Q}$ and $1\leq r^{\prime}\leq pq$.

(c)

Let $\mathcal{F}^{F}$ be the set of all $\phi^{F}(S)$.

(d)

Let $\mathcal{F}^{R}$ be the set of all $\phi^{R}(S)$.

(6)

Set $k_{0}=\frac{\log(1/\alpha)}{\log(1+\alpha)}$, where $\alpha=\sqrt{1+\epsilon}-1$.

(7)

Initialize a list of checkpoints $\mathcal{C}=\emptyset$.

(8)

Set the starting index $c_{start}$ to be $1$, the length estimate $\tilde{\ell}$ of the longest $d$-near-palindrome found so far to be [math], and the at most $d$ mismatched indices $\mathcal{M}=\emptyset$.

We now formalize the steps outlined in the overview. The data structure relies on the procedure $\mathsf{NearPalindrome}$ that we describe and analyze in detail in Section 4.2.

Maintenance:

(1)

Read $S[x]$. Update the sliding window to $S[x-2d,x]$.

(2)

Update the Master Fingerprints to be $\mathcal{F}^{F}(1,x)$ and $\mathcal{F}^{R}(1,x)$:

(a)

Update the first-level fingerprints: for every $p\in\mathcal{P}$, let $r\equiv x\mod p$, and increment $\phi_{r,p}^{F}(S)$ by $S[x]\cdot B^{{\left\lceil x/p\right\rceil}}\bmod{P}$ and increment $\phi_{r,p}^{R}(S)$ by $S[x]\cdot B^{-{\left\lceil x/p\right\rceil}}\bmod{P}$.

(b)

Update the second-level fingerprints: for every $p\in\mathcal{P}$ and $q\in\mathcal{Q}$, let $r^{\prime}\equiv x\mod pq$, and increment $\phi_{r^{\prime},pq}^{F}(S)$ by $S[x]\cdot B^{{\left\lceil x/(pq)\right\rceil}}\bmod{P}$ and increment $\phi^{R}_{r^{\prime},pq}(S)$ by $S[x]\cdot B^{-{\left\lceil x/(pq)\right\rceil}}\bmod{P}$.

(3)

For all $k\geq k_{0}$:

(a)

If $x$ is a multiple of ${\left\lfloor\alpha(1+\alpha)^{k-2}\right\rfloor}$, then add the checkpoint $c=x$ to $\mathcal{C}$. Set $\mathit{level}\left(c\right)=k$, $\mathit{fingerprints}\left(c\right)={\cal F}^{F}(1,x)\cup{\cal F}^{R}(1,x)$.

(b)

If there exists a checkpoint $c$ with $\mathit{level}\left(c\right)=k$ and $c<x-2(1+\alpha)^{k}$, then delete $c$ from $\mathcal{C}$.

(4)

For every checkpoint $c\in\mathcal{C}$ such that $x-c>\tilde{\ell}$, we call ${\mathsf{NearPalindrome}}$ (described in Section 4.2) to see if $S[c,x]$ is a $d$-near-palindrome. If $S[c,x]$ is a $d$-near-palindrome, then set $c_{start}=c$, $\tilde{\ell}=x-c$ and $\mathcal{M}$ to be the indices returned by ${\mathsf{NearPalindrome}}$.

(5)

If $x=n$, then report $c_{start}$, $\tilde{\ell}$, and $\mathcal{M}$.

4.2 Procedure $\mathsf{NearPalindrome}$ and Analysis

In this section, we describe and analyze the randomized procedure ${\mathsf{NearPalindrome}}$ that receives as input a string, and decides whether it is a $d$-near-palindrome or not. Moreover, if the string is a $d$-near-palindrome, ${\mathsf{NearPalindrome}}$ returns the locations of the mismatched indices. As mentioned, ${\mathsf{NearPalindrome}}$ adapts ideas from [CFP*+*16] for solving the $k$-mismatch problem. Our proofs of the properties of ${\mathsf{NearPalindrome}}$ follow almost verbatim from the statements in [CFP*+*16], with the only difference being that we make the magnitudes of the chosen primes as large as to withstand patterns of length $\mathcal{O}\left(n\right)$. We also use the notations from [CFP*+*16], which we introduce next.

Given a string $S[x,y]$, and prime $p_{j}$ let $\Delta_{j}(x,y)$ be the number of $r\in[p_{j}]$ such that the subpatterns $S_{r,p_{j}}[x,y]$ and $S^{R}_{r,p_{j}}[x,y]$ are different. Note that we can compute $\Delta_{j}(x,y)$ from the fingerprints $\mathcal{F}^{F}(x,y)$ and $\mathcal{F}^{R}(x,y)$ as the number of indices $r$ such that $\phi^{F}_{r,p_{j}}[x,y]\neq B^{k+1}\cdot\phi^{R}_{r,p_{j}}[x,y]\bmod{P}$, where $k$ is the length of $S_{r,p_{j}}[x,y]$. Define $\Delta(x,y)=\max_{j}\Delta_{j}(x,y)$. We may assume throughout that $S[x,y]$ has even length. Next we summarize some useful properties of $\Delta(x,y)$.

Lemma 4.1

(Adaptation of Lemma 5.1 and Lemma 5.2 [CFP*+*16]) Let $\beta=1/16$.

1. (1)

If ${\mathsf{HAM}}(S[x,y],S^{R}[x,y])\leq d$, then $\Delta(x,y)\leq d$. 2. (2)

If ${\mathsf{HAM}}(S[x,y],S^{R}[x,y])\geq 2d$, then $\Delta(x,y)>(1+\beta)\cdot d$ with probability at least $1-\frac{1}{n^{3}}$.

**Proof :  ** Recall that $\Delta(x,y)=\max_{j}\Delta_{j}(x,y)$, where $\Delta_{j}(x,y)$ is the number of indices $r$ such that the subpatterns $S_{r,p_{j}}[x,y]$ and $S^{R}_{r,p_{j}}[x,y]$ are not the same. Also recall that a mismatch is an index $a$ s.t. $S[x+a]\neq S[y-x-a+1]$. Then (1) follows from the observation that for every $p_{j}$, the number of $r\in[p_{j}]$ for which $S_{r,p_{j}}[x,y]\neq S^{R}_{r,p_{j}}[x,y]$ is at most the number of mismatches of $S[x,y]$, and so ${\mathsf{HAM}}(S[x,y],S^{R}[x,y])\geq\Delta(x,y)$. We now show that $\Delta(x,y)\leq(1+\beta)\cdot d$ w.p. $\leq\frac{1}{n^{3}}$, thus proving (2). Assume that ${\mathsf{HAM}}(S[x,y],S^{R}[x,y])\geq 2d$, and let $\mathcal{M}$ be any set of $2d$ mismatches in $S[x,y]$.

A mismatch $a$ is $\mathcal{M}$-isolated under prime $p_{j}$ if there exists some $r\in[p_{j}]$ so that $a$ is the only mismatch from $\mathcal{M}$ in the first-level subpattern $S_{r,p_{j}}[x,y]$. Hence, the number of $\mathcal{M}$-isolated mismatches under any prime $p_{j}$ is a lower bound on $\Delta(x,y)$.

We will show that \mathbf{Pr}\Big{[}\Delta(x,y)<(1+\beta)d\Big{]}\leq 1/n^{3}.

Claim 4.2

We have $Pr_{p}[\Delta_{j}(x,y)<(1+\beta)d]<1/8$, over random prime $p$ chosen by the algorithm.

**Proof :  ** Note that $\Delta_{j}(x,y)<(1+\beta)d$ if and only if at least $(1-\beta)d$ elements in $\mathcal{M}$ are not $\mathcal{M}$-isolated under $p_{j}$. By Lemma 2.2, for any $a,b\in\mathcal{M}$, the probability $a\equiv b\pmod{p_{j}}$ is at most $\frac{\beta}{32d}$. Therefore, by a union bound, for a fixed $a\in\mathcal{M}$, $a$ is not $\mathcal{M}$-isolated under $p_{j}$ w.p. $\frac{\beta}{32d}\cdot(2d)=\beta/16$. Thus, the expected number of elements in $\mathcal{M}$ that are not $\mathcal{M}$-isolated is less than $(\beta d)/8$. By Markov’s inequality, the number of elements in $\mathcal{M}$ that are not $\mathcal{M}$-isolated exceeds $(1-\beta)d$ with probability at most $\beta/(8\cdot(1-\beta))<1/(8\cdot 15)<1/8$. $\Box$

From the claim and from the fact that $\Delta(x,y)=\max\Delta_{j}(x,y)$ it follows that after picking $2\log n$ random primes in $\left[\frac{d}{\beta}\log^{2}n,\frac{34d}{\beta}\log^{2}n\right]$ we have \mathbf{Pr}\Big{[}\Delta(x,y)<(1+\beta)d\Big{]}\leq(1/8)^{2\log n}\leq n^{-3}. $\Box$

A position $i\in[x,y]$ is an isolated mismatch under $p_{j}$ if there exists some $r\leq p_{j}$ for which the subpatterns $S_{r,p_{j}}[x,y]$ and $S^{R}_{r,p_{j}}[x,y]$ differ only in position $i$. Let $\mathcal{I}_{j}(x,y)$ be the number of isolated mismatches in $S[x,y]$ under $p_{j}$, and let $\mathcal{I}(x,y)$ be the union of $\mathcal{I}_{j}(x,y)$, over all primes $p_{j}$. The next lemma shows that if ${\mathsf{HAM}}(S[x,y],S^{R}[x,y])\leq 2d$ , then $\mathcal{I}(x,y)$ is precisely ${\mathsf{HAM}}(S[x,y],S^{R}[x,y])$ with high probability over the set of primes.

Lemma 4.3

*(Adaptation of Lemma 4.2 [CFP*+*16]) If ${\mathsf{HAM}}(S[x,y],S^{R}[x,y])\leq 2d$, then

${\mathsf{HAM}}(S[x,y],S^{R}[x,y])=\mathcal{I}(x,y)$ with probability at least $1-\frac{1}{n^{7}}$.*

**Proof :  ** Since $\mathcal{I}(x,y)$ is the union of $\mathcal{I}_{j}(x,y)$, the number of isolated mismatches in $S[x,y]$ under $p_{j}$, then ${\mathsf{HAM}}(S[x,y],S^{R}[x,y])=\mathcal{I}(x,y)$ if and only if each mismatch is isolated under $p_{j}$ for some $j$.

For fixed $a,b\in\mathcal{M}^{\prime}$, the probability that $a\equiv b\bmod{p_{j}}$ is at most $1/32d$ by Lemma 2.2. As before, since there are at most $2d$ mismatches, the probability that $a\equiv b\bmod{p_{j}}$ for some $b\in\mathcal{M}^{\prime}$ is at most $1/16$ by a union bound. This is the probability that $a$ is not isolated under $p_{j}$.

Thus, the probability that $a$ is not isolated under any of the random $2\log n$ primes in $\mathcal{P}$ is at most $(1/16)^{2\log n}=1/n^{8}$. Thus, the probability that there is some $a\in\mathcal{M}^{\prime}$ that is not isolated under any of the primes is at most $2d/n^{8}\leq 1/n^{7}$, by another union bound.

Recall that if all mismatches are isolated, then ${\mathsf{HAM}}(S[x,y],S^{R}[x,y])=\mathcal{I}(x,y)$, and so the probability that ${\mathsf{HAM}}(S[x,m],S^{R}[m+1,y])\neq\mathcal{I}(x,y)$ is at most $1/n^{7}$. $\Box$

Lemma 4.4

(Adaptation of Lemma 4.3 [CFP*+*16]) The set of mismatches can be identified using the second-level fingerprints.

**Proof :  ** Using the notion from the proof of Lemma 4.3, note that if subpattern $S_{r_{j},p_{j}}[x,y]$ contains an isolated mismatch for prime $p_{j}$ and $r_{j}\in[p_{j}]$, then this mismatch is exactly the one position that does not match in the second-level subpattern. It remains to show that the algorithm can correctly recover the isolated mismatch through the second-level subpatterns. Suppose, by way of contradiction, the algorithm recovers some index $s$ not equivalent to the mismatch $t$ isolated under $p_{j}$. Then it follows that both $s$ and $t$ are equivalent to $r_{1}\bmod{p_{j}q_{1}}$, $r_{2}\bmod{p_{j}q_{2}}$, $\ldots$, $r_{|\mathcal{Q}|}\bmod{p_{j}q_{|\mathcal{Q}|}}$. By Theorem 1 of [RS62], the product of the primes $q_{i}$ is at least $n$. Thus, by the Chinese Remainder Theorem, $s=t$, which is a contradiction. It follows that the algorithm correctly identifies the location of any isolated mismatches. $\Box$

We are now ready to present the algorithm in full.

${\mathsf{NearPalindrome}}(c_{i},x)$: (determines if $S[c_{i},x]$ is a $d$-near-palindrome)

(1)

For each $j\in[2\log n]$, initialize $\Delta_{j}=0$.

(2)

For each $j\in[2\log n]$ and $r\in[p_{j}]$:

If $\phi^{F}_{r,p_{j}}(S[c_{i},x])\neq B^{k+1}\cdot\phi^{R}_{r,p_{j}}(S[c_{i},x])\bmod{P}$, where $k$ is the length of $S_{r,p_{j}}[c_{i},x]$, then increment $\Delta_{j}(c_{i},x)=\Delta_{j}(c_{i},x)+1$.

(3)

Let $\Delta(c_{i},x)=\max_{j}\{\Delta_{j}(c_{i},x)\}$.

(4)

If $\Delta(c_{i},x)>(1+\beta)\cdot d$, then we immediately reject $S[c_{i},x]$. (Recall that $\beta=\frac{1}{16}$.)

(5)

Initialize $\mathcal{I}=\emptyset$.

(6)

For each mismatch in $S[c_{i},x]$, if there exists $q\in\mathcal{Q}$ and such that $\phi^{F}_{r^{\prime},q}(S_{r,p}[c_{i},x])\neq B^{k^{\prime}+1}\cdot\phi^{R}_{r^{\prime},q}(S_{r,p}[c_{i},x])\bmod{P}$, where $k^{\prime}$ is the length of $S_{r^{\prime}+rp,pq}[c_{i},x]$, for exactly one $r\in[p],r^{\prime}\in[q]$, then insert the mismatch into $\mathcal{I}(c_{i},x)$. (This is the set of isolated mismatches.)

(7)

If $|\mathcal{I}(c_{i},x)|>d$, then we reject $S[c_{i},x]$.

(8)

Else, if $|\mathcal{I}(c_{i},x)|\leq d$, then we accept $S[c_{i},x]$ and return $\mathcal{I}(c_{i},x)$.

Theorem 4.5

With probability at least $1-\frac{1}{n^{3}}$, procedure ${\mathsf{NearPalindrome}}$ returns whether $S[c_{i},x]$ is a $d$-near-palindrome.

**Proof :  ** If ${\mathsf{HAM}}(S[c_{i},x],S^{R}[c_{i},x])>2d$, then by Lemma 4.1, $\Delta(c_{i},x)>(1+\beta)\cdot 2d$ with probability at least $1-\frac{1}{n^{3}}$ and so ${\mathsf{NearPalindrome}}$ will reject $S[c_{i},x]$. Conditioned on ${\mathsf{HAM}}(S[c_{i},x],S^{R}[c_{i},x])\leq 2d$, by Lemma 4.3 $\mathcal{I}(c_{i},x)={\mathsf{HAM}}(S[c_{i},x],S^{R}[c_{i},x])$ with probability at least $1-\frac{1}{n^{5}}$, and so if ${\mathsf{HAM}}(S[c_{i},x],S^{R}[c_{i},x])>d$ the algorithm safely rejects, and otherwise it accepts. Finally, by Lemma 4.4 the entire set of mismatches $\mathcal{I}(c_{i},x)$ can be computed from the second-level subpattern fingerprints. $\Box$

4.3 Correctness and Space Complexity

In this section, we finish the proof of Theorem 1.1 by claiming correctness and analyzing the space used by the one-pass streaming algorithm described in Section 4.1. Since we used the spacing of the checkpoints as in [BEMS14], we have the following properties.

Observation 4.6

([BEMS14], Observation 16, Lemma 17) At reading $S[x]$, for all $k\geq k_{0}={\left\lceil\frac{\log\left(\frac{(1+\alpha)^{2}}{\alpha}\right)}{\log(1+\alpha)}\right\rceil}$, let $C_{x,k}=\{c\in\mathcal{C}\,|\,\mathit{level}\left(c\right)=k\}$.

1. (1)

$C_{x,k}\subseteq[x-2(1+\alpha)^{k},x]$. 2. (2)

The distance between two consecutive checkpoints of $C_{x,k}$ is ${\left\lfloor\alpha(1+\alpha)^{k-2}\right\rfloor}$. 3. (3)

$|C_{x,k}|={\left\lceil\frac{2(1+\alpha)^{k}}{{\left\lfloor\alpha(1+\alpha)^{k-2}\right\rfloor}}\right\rceil}$. 4. (4)

At any point in the algorithm, the number of checkpoints is $\mathcal{O}\left(\frac{\log n}{\epsilon\log(1+\epsilon)}\right)$.

Corollary 4.7

The total space used by the algorithm is $\mathcal{O}\left(\frac{d\log^{7}n}{\epsilon\log(1+\epsilon)}\right)$ bits. The update time per arriving symbol is also $\mathcal{O}\left(\frac{d\log^{6}n}{\epsilon\log(1+\epsilon)}\right)$.

**Proof :  ** The first-level and second-level Karp-Rabin fingerprints consist of integers modulo $P$ for each of the $\mathcal{O}\left(d\log^{5}n\right)$ subpatterns. Since $P\in[n^{5},n^{6}]$, then $\mathcal{O}\left(d\log^{6}n\right)$ bits of space are necessary for each fingerprint. Furthermore, by Observation 4.6, there are $\frac{\log n}{\epsilon\log(1+\epsilon)}$ checkpoints, so the total space used is $\mathcal{O}\left(d\log^{7}n\right)$ bits. For each arriving symbol $S[x]$, the algorithm checks possibly the fingerprints of each checkpoint whether the substring is a $d$-near-palindrome. There are $\mathcal{O}\left(\frac{\log n}{\epsilon\log(1+\epsilon)}\right)$ checkpoints, each with fingerprints of size $\mathcal{O}\left(d\log^{5}n\right)$. Each subpattern of a fingerprint may be compared in constant time, so the overall update time is $\mathcal{O}\left(\frac{d\log^{6}n}{\epsilon\log(1+\epsilon)}\right)$. $\Box$

We now show correctness and analyze the space complexity of the one-pass streaming algorithm described in Section 4.1.

**Proof of Theorem 1.1:  ** Let $\ell_{max}$ be the length of the longest $d$-near-palindrome, $S[x,x+\ell_{max}-1]$, with midpoint $m$. Let $k$ be the largest integer so that $2(1+\alpha)^{k-1}<\ell_{max}$, where $\alpha=\sqrt{1+\epsilon}-1$. Let $y=m+(1+\alpha)^{k-1}$ so that $x<y<x+\ell_{max}-1$. By Observation 4.6, there exists a checkpoint in the interval $[y-2(1+\alpha)^{k-1},y]$. Furthermore, Observation 4.6 implies consecutive checkpoints of level ${k-1}$ are separated by distance ${\left\lfloor\alpha(1+\alpha)^{k-2}\right\rfloor}$. Thus, there exists a checkpoint $c$ in the interval $\left[y-2(1+\alpha)^{k-1},y-2(1+\alpha)^{k-1}+\alpha(1+\alpha)^{k-3}\right]$. If procedure ${\mathsf{NearPalindrome}}$ succeeds for this checkpoint on position $m+(m-c)$, then the output $\tilde{\ell}$ of the algorithm is at least

[TABLE]

Comparing this output with $\ell_{max}$,

[TABLE]

Thus, if procedure ${\mathsf{NearPalindrome}}$ succeeds for all substrings then $\tilde{\ell}\leq\ell_{max}\leq(1+\epsilon)\tilde{\ell}$. Taking Theorem 4.5 and a simple union bound over all $\mathcal{O}\left(n^{2}\right)$ possible substrings, procedure ${\mathsf{NearPalindrome}}$ succeeds for all substrings with probability at least $1/n$, and the result follows. $\Box$

5 One-Pass Streaming Algorithm with Additive Error $E$

In this section, we prove Theorem 1.2, showing a one-pass streaming algorithm which uses $\mathcal{O}\left(\frac{dn\log^{6}n}{E}\right)$ bits of space. The initialization of the algorithm is the same as that in Section 4.1 for the one-pass streaming algorithm with multiplicative error $(1+\epsilon)$.

Maintenance:

(1)

Read $S[x]$. Update the sliding window to $S[x-2d,x]$.

(2)

Update the Master Fingerprints to be $\mathcal{F}^{F}(1,x)$ and $\mathcal{F}^{R}(1,x)$:

(a)

For the first-level fingerprints: for every $p\in\mathcal{P}$, let $r\equiv x\mod p$, and increment $\phi_{r,p}^{F}(S)$ by $S[x]\cdot B^{{\left\lceil x/p\right\rceil}}\bmod{P}$ and increment $\phi_{r,p}^{R}(S)$ by $S[x]\cdot B^{-{\left\lceil x/p\right\rceil}}\bmod{P}$.

(b)

For the second-level fingerprints: for every $p\in\mathcal{P}$ and $q\in\mathcal{Q}$, let $r^{\prime}\equiv x\mod pq$, and increment $\phi_{r^{\prime},pq}^{F}(S)$ by $S[x]\cdot B^{{\left\lceil x/(pq)\right\rceil}}\bmod{P}$ and increment $\phi^{R}_{r^{\prime},pq}(S)$ by

$S[x]\cdot B^{-{\left\lceil x/(pq)\right\rceil}}\bmod{P}$.

(3)

If $x$ is a multiple of ${\left\lfloor\frac{E}{2}\right\rfloor}$, then add the checkpoint $c=x$ to $\mathcal{C}$. Set $\mathit{fingerprints}\left(c\right)={\cal F}^{F}(1,x)\cup{\cal F}^{R}(1,x)$.

(4)

For every checkpoint $c\in\mathcal{C}$ such that $x-c>\tilde{\ell}$, we call procedure ${\mathsf{NearPalindrome}}$ to see if $S[c,x]$ is a near-palindrome. If $S[c,x]$ is a near-palindrome, then set $c_{start}=c$, $\tilde{\ell}=x-c$ and $\mathcal{M}$ to be the indices returned by ${\mathsf{NearPalindrome}}$.

(5)

If $x=n$, then report $c_{start}$, $\tilde{\ell}$, and $\mathcal{M}$.

Corollary 5.1

The algorithm uses $\mathcal{O}\left(\frac{dn\log^{6}n}{E}\right)$ bits of space and $\mathcal{O}\left(\frac{dn\log^{5}n}{E}\right)$ time per arriving symbol.

**Proof :  ** Each of the Karp-Rabin fingerprints consist of $\mathcal{O}\left(d\log^{5}n\right)$ integers modulo $P$. Since $P\in[n^{5},n^{6}]$, then $\mathcal{O}\left(d\log^{6}n\right)$ bits of space are necessary for each fingerprint. Each checkpoint is spaced ${\left\lfloor\frac{E}{2}\right\rfloor}$ positions apart, so there are at most $\frac{2n}{E}+1$ checkpoints, and the total space required is $\mathcal{O}\left(\frac{dn\log^{6}n}{E}\right)$ bits. For each arriving symbol $S[x]$, the algorithm checks each checkpoint and possibly the fingerprints of each checkpoint to check whether the substring is a near-palindrome. Since there are $\mathcal{O}\left(\frac{n}{E}\right)$ checkpoints, each containing fingerprints of size $\mathcal{O}\left(d\log^{5}n\right)$, and each subpattern of a fingerprint may be compared in constant time, then the overall update time per arriving symbol is $\mathcal{O}\left(\frac{dn\log^{5}n}{E}\right)$. $\Box$

The correctness of the algorithm follows immediately from the spacing of the checkpoints, and the correctness of procedure ${\mathsf{NearPalindrome}}$.

**Proof of Theorem 1.2:  ** For each $x,y$, procedure ${\mathsf{NearPalindrome}}$ returns, with probability at least $1-\frac{1}{n^{3}}$, whether $S[c_{i},x]$ is a $d$-near-palindrome. Thus by a simple union bound over all possible substrings of the stream, ${\mathsf{NearPalindrome}}$ succeeds with probability at least $1-\frac{1}{n}$. Because the checkpoints are separated by distance ${\left\lfloor\frac{E}{2}\right\rfloor}$, the longest $d$-near-palindrome can begin at most ${\left\lfloor\frac{E}{2}\right\rfloor}-1$ characters before a checkpoint. Hence, the algorithm outputs some $\tilde{\ell}$ such that $\tilde{\ell}\geq\ell_{max}-E$. $\Box$

6 Two-Pass Exact Streaming Algorithm

In this section, we prove Theorem 1.3. Namely, we present a two-pass streaming algorithm which returns the longest $d$-near-palindrome with space $\mathcal{O}\left(d^{2}\sqrt{n}\log^{6}n\right)$.

Recall that we assume the lengths of $d$-near-palindromes are even. Thus, for any substring $S[x,y]$ of even length, we define its midpoint $m={\left\lfloor\frac{x+y}{2}\right\rfloor}$. Upon reading $x$, we say that $x-\sqrt{n}$ is a candidate midpoint if the sliding window $S[x-2\sqrt{n},x]$ is a $d$-near-palindrome.

First, we modify the one-pass streaming algorithm with additive error in Section 5 so that it returns a list $\mathcal{L}$ of candidate midpoints of $d$-near-palindromes with length at least $\ell-\frac{\sqrt{n}}{2}$, where $\ell$ is an estimate of the maximum length output by the algorithm. However, we show in Lemma 6.2 that the string has a periodic structure which allows us to keep only $\mathcal{O}\left(d\right)$ fingerprints in order to recover the fingerprint for any substring between two midpoints.

In the second pass, we explicitly keep the $\frac{\sqrt{n}}{2}$ characters before the starting positions and candidate midpoints of “long” $d$-near-palindromes identified in the first pass. We use a procedure ${\mathsf{Recover}}$ to exactly identify the number and locations of mismatches within the $d$-near-palindromes identified in the first pass. We then use the $\frac{\sqrt{n}}{2}$ characters to extend the near-palindromes until the number of mismatches exceed $d+1$.

For an example, see Figure 6.

We first describe a structural property of a series of overlapping $d$-near-palindromes, showing that they are “almost” periodic.

Definition 6.1

A string $S$ is said to have period $\pi$ if $S[j]=S[j+\pi]$ for all $j=1,\ldots,|S|-\pi$.

The following structural result is a generalization of a structural result about palindromes from [BEMS14] and demonstrates two properties. The first property shows that the midpoints of long near-palindromes are equally spaced, and thus the entire set can be represented succinctly after the first pass, even if it is linear in size. The second property shows a repetitive nature of the string that allows the fingerprint reconstruction of many substrings just by storing a small number of fingerprints.

Lemma 6.2

Let $m_{1}<m_{2}<\ldots<m_{h}$ be indices in $S$ that are consecutive midpoints of $d$-near-palindromes of length $\ell^{*}$, for some integer $\ell^{*}>0$. If $m_{h}-m_{1}\leq\ell^{*}$, then

1. (1)

$m_{1},m_{2},\ldots,m_{h}$* are equally spaced in $S$, so that $|m_{2}-m_{1}|=|m_{i}-m_{i+1}|$ for all $i\in[h-1]$.* 2. (2)

For each $1\leq i\leq h$, there exists string $E_{i}$ with at most $d$ nonzero entries such that $E_{i}+S[m_{1}+1,m_{i}]$ is a prefix of $ww^{R}ww^{R}\ldots$ of length at least $\ell^{*}$, for some string $w$ of length $|w|=m_{2}-m_{1}$.

**Proof :  ** Note that $m_{2}$ is a midpoint of a $d$-near-palindrome of length at least $\ell^{*}$, so there exists a string $E_{2}$ with at most $d$ nonzero entries such that $E_{2}+S[m_{1}+1,2m_{2}-m_{1}]$ is a palindrome of length at least $\ell^{*}$.

Inductively, we assume that 1 and 2 hold up to $m_{j-1}$. First, we argue that $|m_{j}-m_{1}|$ is a multiple of $|m_{2}-m_{1}|=|w|$. Suppose, by way of contradiction, that $m_{j}=m_{1}+|w|\cdot q+r$ for some integers $q\geq 0$ and $0<r<|w|$. Since $m_{h}-m_{1}\leq\ell^{*}$, then $[m_{1}+1,m_{j-1}+\ell^{*}]$ contains $m_{j}$. From our inductive hypothesis, $m_{j}-r$ is an index where either $w$ or $w^{R}$ begins. This implies that the prefix of $ww^{R}$ or $w^{R}w$ of size $2r$ is a palindrome. By assumption, there exists $E_{j-1}$ with at most $d$ nonzero entries such that $E_{j-1}+S[m_{1}+1,m_{h}]$ is a prefix of $ww^{R}ww^{R}\ldots$ of length at least $\ell^{*}$. Thus, the interval $[m_{1}+1,m_{1}+r]$ contains a midpoint of a $d$-near-palindrome with length at least $\ell^{*}$. However, there is no such midpoint in the interval $[m_{1}+1,m_{2}-1]$, an interval with length greater than $r$, which is a contradiction.

Thus, $m_{j}=m_{j-1}+|w|\cdot q$. Since $m_{j}$ is a midpoint of a $d$-near-palindrome, then 2 follows. But then $m_{j-1}+|w|$ is the midpoint of a $d$-near-palindrome of length at least $\ell^{*}$. Specifically, $S[m_{j-1}+|w|-\ell^{*}+1,m{j-1}+|w|+\ell^{*}]$ is the desired $d$-near-palindrome. Hence, $m_{j}=m_{j-1}+|w|$, satisfying 1, and the induction is complete. $\Box$

In the first pass, we specify that the algorithm has sliding window size $2\sqrt{n}$. Thus, if the longest $d$-near-palindrome has length less than $2\sqrt{n}$, the algorithm can identify it. Otherwise, if the longest $d$-near-palindrome has length at least $2\sqrt{n}$, then the algorithm finds at most $\frac{\sqrt{n}}{2}$ non-overlapping $d$-near-palindromes of length at least $\ell-\epsilon\sqrt{n}$. Hence, $\mathcal{O}\left(d^{2}\sqrt{n}\log^{6}n\right)$ is enough space to store the fingerprints for the substrings between any two candidate midpoints, as well as between checkpoints $s_{i}\in\mathcal{L}$ and midpoints. The first pass of the algorithm appears below, omitting the details for when the longest $d$-near-palindrome has length at most $2\sqrt{n}$ and is therefore recognized by the sliding window.

First pass:

(1)

Read $S[x]$. Set $m=x-\sqrt{n}$. Update the sliding window to $S[x-2\sqrt{n},x]$.

(2)

Update the Master Fingerprints to be $\mathcal{F}^{F}(1,x)$ and $\mathcal{F}^{R}(1,x)$:

(a)

For the first-level fingerprints: for every $p\in\mathcal{P}$, let $r\equiv x\mod p$, and increment $\phi_{r,p}^{F}(S)$ by $S[x]\cdot B^{{\left\lceil x/p\right\rceil}}\bmod{P}$ and increment $\phi_{r,p}^{R}(S)$ by $S[x]\cdot B^{-{\left\lceil x/p\right\rceil}}\bmod{P}$.

(b)

For the second-level fingerprints: for every $p\in\mathcal{P}$ and $q\in\mathcal{Q}$, let $r^{\prime}\equiv x\mod pq$, and increment $\phi_{r^{\prime},pq}^{F}(S)$ by $S[x]\cdot B^{{\left\lceil x/(pq)\right\rceil}}\bmod{P}$ and increment $\phi^{R}_{r^{\prime},pq}(S)$ by

$S[x]\cdot B^{-{\left\lceil x/(pq)\right\rceil}}\bmod{P}$.

(3)

If $x$ is a multiple of ${\left\lfloor\frac{\epsilon\sqrt{n}}{2}\right\rfloor}$, then add the checkpoint $c=x$ to $\mathcal{C}$. Set $\mathit{fingerprints}\left(c\right)=\mathcal{F}^{F}(1,x)\cup\mathcal{F}^{R}(1,x)$, $longest(c)=0$.

(4)

For every checkpoint $c\in\mathcal{C}$ such that $x-c\geq\tilde{\ell}-\frac{\sqrt{n}}{2}$, we call procedure ${\mathsf{NearPalindrome}}$ to see if $S[c,x]$ is a near-palindrome. If $S[c,x]$ is a near-palindrome, then set $longest(c)=x-c$. If $x-c>\tilde{\ell}$, set $\tilde{\ell}=x-c$.

(5)

If $[x-2\sqrt{n},x]$ is a $d$-near-palindrome:

(a)

Add $m$ to $L_{c^{\prime}}$, the list of candidate midpoints for the most recent checkpoint $c^{\prime}$.

(b)

If $|L_{c^{\prime}}|=0$, store the first-level and second-level fingerprints of $S[c^{\prime}+1,x]$.

(c)

Else, let $m_{i}$ be the largest index in $L_{c^{\prime}}$.

i.

If the first-level and second-level fingerprints of $S[m_{i},m]$ match those of some other entry $S[m_{j},m_{j+1}]$ stored in $L_{c^{\prime}}$ and the set of indices for $m_{j}$ is less than $d$, add $m$ to the set of indices for $m_{j}$.

ii.

Else, if the first-level and second-level fingerprints of $S[m_{i},m]$ do not match those of any other entry stored in $L_{c^{\prime}}$, then add the first-level and second-level fingerprints of $S[m_{i},m]$ into $L_{c^{\prime}}$, along with the index $m$.

(6)

If $x=n$, then remove all $c\in\mathcal{C}$ such that $longest(c)<\tilde{\ell}-\sqrt{n}$. Report $\tilde{\ell}$, $\mathcal{C}$, and $\{L_{c}\}$.

Before we proceed to the second pass, we describe procedure ${\mathsf{Recover}}(m_{i},m_{j},L_{c})$ which either outputs that $S[m_{i},m_{j}]$ is not a $d$-near-palindrome, or returns the number of mismatches, as well as their indices. The procedure crucially relies on structural result from Lemma 6.2 to reconstruct the fingerprints of $S[m_{i},m_{j}]$ from fingerprints stored by the first pass. From the reconstructed fingerprints, the subroutine can then determine whether $S[m_{i},m_{j}]$ is a $d$-near-palindrome, and identify the location of the mismatches, if necessary. The details of procedure ${\mathsf{Recover}}$ in full is below:

${\mathsf{Recover}}(m_{i},m_{j},L_{c})$: (determines whether $S[m_{i},m_{j}]$ is a $d$-near-palindrome and outputs the indices and hence, number, of mismatches if it is)

(1)

Construct the first-level and second-level fingerprints of $S[m_{i},m_{j}]$:

(a)

$\phi^{F}_{a,b}(S[m_{i},m_{j}])=\sum_{k=i}^{j-1}B^{t}\cdot\phi^{F}_{a,b}(S[m_{k},m_{k+1}])\bmod{P}$, where $t$ is the length of the subpattern $S_{a,b}[m_{k},m_{k+1}]$.

(b)

$\phi^{R}_{a,b}(S[m_{i},m_{j}])=\sum_{k=i}^{j-1}B^{-t}\cdot\phi^{R}_{a,b}(S[m_{k},m_{k+1}])\bmod{P}$, where $t$ is the length of the subpattern $S^{R}_{a,b}[m_{k},m_{k+1}]$.

(2)

Call procedure ${\mathsf{NearPalindrome}}(S[m_{i},m_{j}])$ to see whether $S[m_{i},m_{j}]$ is a $d$-near-palindrome:

(a)

If $S[m_{i},m_{j}]$ is not a $d$-near-palindrome, reject $S[m_{i},m_{j}]$.

(b)

Else, accept $S[m_{i},m_{j}]$. Output $\mathcal{I}$, the set of mismatches output by ${\mathsf{NearPalindrome}}$.

Before the second pass, we first prune the list of checkpoints $\mathcal{C}$ to greedily include only those who are the starting indices for $d$-near-palindromes of length at least $\tilde{\ell}-\frac{\sqrt{n}}{2}$ and do not overlap with other $d$-near-palindromes already included in the list. In the second pass, the algorithm keeps track of the $\frac{\sqrt{n}}{2}$ characters before $c$, for each starting index $c\in\mathcal{C}$. We call procedure ${\mathsf{Recover}}$ to fully recover the mismatches in a region following $c$. After reading the last symbol in the region, we compare each subsequent symbol with the corresponding symbol before $c$, counting the total number of mismatches. When the total number of mismatches reaches $d+1$ after seeing character $S[c+k+j+1]$, where $k$ is the size of the region, then the previous symbol is the end of the near-palindrome. Hence, the near-palindrome is $S[c-j,c+k+j]$, and if $k+2j>\tilde{l}$, then we update the information for $\tilde{\ell}$ accordingly. For an example, see Figure 7.

We describe the second algorithm below, again omitting the case for when the longest $d$-near-palindrome has length at most $2\sqrt{n}$ and is therefore immediately recognized by the sliding window in the first pass. Recall that $\mathcal{C}$ has already been pruned in the first pass to only include checkpoints serving as the start of $d$-near-palindromes of length at least $\tilde{\ell}-\sqrt{n}$. We further prune $\mathcal{C}$ by removing checkpoints causing overlapping $d$-near-palindromes.

Preprocessing:

For each $c\in\mathcal{C}$:

If $c^{\prime}<c<c^{\prime}+\tilde{\ell}-\sqrt{n}$ for some other $c^{\prime}\in\mathcal{C}$, then remove $c$.

Second pass:

(1)

Maintain a sliding window of size $2\sqrt{n}$ and set $\ell=\tilde{\ell}$ from the first pass.

(2)

Initialize $\mathcal{A}$ to be an empty array of size $\sqrt{n}$. It will dynamically contain the $\frac{\sqrt{n}}{2}$ characters before $c\in\mathcal{C}$ reported in the first pass.

(3)

Initialize $\mathcal{B}$ to be an empty array of size $d\sqrt{n}$. It will dynamically contain the $\frac{\sqrt{n}}{2}$ characters before each of the at most $d$ different substrings between midpoints in each $L_{c}$.

(4)

If $x=c-\frac{\sqrt{n}}{2}-j$ for some $c\in\mathcal{C}$ and $1\leq j\leq\frac{\sqrt{n}}{2}$, insert $S[x]$ into $\mathcal{A}$.

(5)

If $x=m_{i}-\frac{\sqrt{n}}{2}-j$ for some $c\in\mathcal{C},m_{i}\in L_{c}$ which has not been recorded, and $1\leq j\leq\frac{\sqrt{n}}{2}$, insert $S[x]$ into $\mathcal{B}$.

(6)

If there exists $m_{i},m_{j}\in L_{c}$ for some $c\in\mathcal{C}$ such that $x=m_{j}$ and $x-m_{i}\geq\ell-\sqrt{n}$, then call procedure ${\mathsf{Recover}}(m_{i},x,L_{c})$ is see whether $S[m_{i},x]$ is a $d$-near-palindrome.

If $S[m_{i},x]$ is a $d$-near-palindrome, allocate space for $mismatches((m_{i}+x)/2)$ and set it to be the number of mismatches in $S[m_{i},x]$. Also, keep the indices of the mismatches returned by procedure ${\mathsf{Recover}}(m_{i},x,L_{c})$.

(7)

If there exists $m\in L_{c}$ for some $c\in\mathcal{C}$ such that $m+\frac{\ell}{2}<x<m+\frac{\ell}{2}+\frac{\sqrt{n}}{2}$ and $S[x]\neq S[m-(x-m)+1]$ (which is stored in $\mathcal{A}$):

(a)

If $mismatches(m)<d$, then insert $x$ into the set of mismatches and increment $mismatches(m)$.

(b)

If $mismatches(m)=d$:

i.

If $2x-2m-2>\tilde{\ell}$, set $\tilde{\ell}=2x-2m-2$, $start=2m-x+1$, and $\mathcal{M}$ to be the set of mismatches.

ii.

Deallocate space for $mismatches(m)$.

iii.

If $m$ is the largest midpoint in $L_{c}$, remove the $\frac{\sqrt{n}}{2}$ characters in $\mathcal{A}$ before $c$.

(8)

If there exists $m_{i},m_{j}\in L_{c}$ for some $c\in\mathcal{C}$ such that $m_{j}-m_{i}\geq\ell-\sqrt{n}$, $0<x-m_{j}<\frac{\sqrt{n}}{2}$, $mismatches((m_{i}+m_{j})/2)\leq d$ and $S[x]\neq S[m_{i}-(x-m_{j})+1]$ (which is stored in $\mathcal{B}$):

(a)

If $mismatches((m_{i}+m_{j})/2)<d$, increment $mismatches((m_{i}+m_{j})/2)$, and insert $x$ into the set of mismatches.

(b)

If $mismatches((m_{i}+m_{j})/2)=d$:

i.

If $2x-m_{i}-m_{j}>\tilde{\ell}$, set $\tilde{\ell}=2x-m_{i}-m_{j}$, $start=m_{i}+m_{j}-x$, and $\mathcal{M}$ to be the set of mismatches.

ii.

Deallocate space for $mismatches((m_{i}+m_{j})/2)$.

iii.

If $x>c+\ell+\sqrt{n}$, remove the characters in $\mathcal{B}$ associated with $c$.

(9)

If $x=n$, then output $\tilde{\ell}$, $start$, and $\mathcal{M}$.

The correctness of the algorithm follows from the first pass recognizing the longest $d$-near-palindrome, possibly with the exception of up to $\frac{\sqrt{n}}{2}$ characters before checkpoints (and the corresponding $\frac{\sqrt{n}}{2}$ characters at the end of the $d$-near-palindrome. The first pass can keep this information by storing at most $\mathcal{O}\left(d\right)$ fingerprints, by Lemma 6.2. Then, procedure ${\mathsf{Recover}}$ can fully recover the mismatches found in the first pass by reconstructing the fingerprints. Since the second pass dynamically keeps the $\frac{\sqrt{n}}{2}$ characters before checkpoints and candidate midpoints, then the remaining (at most $\sqrt{n}$) characters of the longest $d$-near-palindrome are explicitly checked and recognized. Therefore, the second pass returns exactly the longest $d$-near-palindrome.

Lemma 6.3

The total space used by the algorithm is $\mathcal{O}\left(d^{2}\sqrt{n}\log^{6}n\right)$ bits. The update time per arriving symbol is $\mathcal{O}\left(d^{2}\sqrt{n}\log^{5}n\right)$.

**Proof :  ** Each of the Karp-Rabin fingerprints consist of $\mathcal{O}\left(d\log^{5}n\right)$ integers modulo $P$. Since $P\in[n^{5},n^{6}]$, then $\mathcal{O}\left(d\log^{6}n\right)$ bits of space are necessary for each fingerprint. There are $\sqrt{n}$ checkpoints, each of which may require $d$ fingerprints due to the compression allowed by the structural result. Hence, the space used by the fingerprints across all checkpoints is $\mathcal{O}\left(d^{2}\sqrt{n}\log^{6}n\right)$ bits. Note that the algorithm also keeps $2\sqrt{n}$ characters in $\mathcal{A}$ and $d\sqrt{n}$ characters in $\mathcal{B}$, so the space usage by the algorithm follows.

For each arriving symbol $S[x]$, the algorithm checks each checkpoint and possibly the fingerprints of each checkpoint to check whether the substring is a $d$-near-palindrome. Since there are $\sqrt{n}$ checkpoints, each containing up to $d$ fingerprints of size $\mathcal{O}\left(d\log^{5}n\right)$, and each subpattern of a fingerprint may be compared in constant time, then the overall update time is $\mathcal{O}\left(d^{2}\sqrt{n}\log^{5}n\right)$. $\Box$

7 Lower Bounds

*Reminder of Theorem 1.4. *** Let $d=o(\sqrt{n})$. Any randomized streaming algorithm that returns an estimate $\hat{\ell}$ of the length of the longest $d$-near-palindrome, where $\hat{\ell}\leq\ell_{max}\leq(1+\epsilon)\hat{\ell}$, with probability at least $1-\frac{1}{n}$, must use $\Omega\left(d\log n\right)$ bits of space.

**Proof of Theorem 1.4:  ** By Yao’s Minimax Principle [Yao77], to show a $\Omega(d\log n)$ lower bound for randomized algorithms, it suffices to show a distribution over inputs such that every deterministic algorithm using less than $\frac{d\log n}{3}$ bits of memory fails with probability at least $\frac{1}{n}$.

We use an approach similar to [GMSU16] who showed lower bounds for palindromes. Let $X$ be the set of binary strings of length $\frac{n}{4}$ with $d$ many $1$’s. Given $x\in X$, let $Y_{x}$ be the set of binary strings of length $\frac{n}{4}$ with either ${\mathsf{HAM}}(x,y)=d$ or ${\mathsf{HAM}}(x,y)=d+1$. We pick $(x,y)$ uniformly at random from $(X,Y_{x})$.

Lemma 7.1

Given an input $x\circ y$, any deterministic algorithm $\mathcal{D}$ which uses less than $\frac{d\log n}{3}$ bits of memory cannot correctly output whether ${\mathsf{HAM}}(x,y)=d$ or ${\mathsf{HAM}}(x,y)=d+1$ with probability at least $1-\frac{1}{n}$.

**Proof :  ** Note that $|X|=\binom{n/4}{d}$. By Stirling’s approximation, $|X|\geq\left(\frac{n}{4d}\right)^{d}$. Since $d=o(\sqrt{n})$, then $|X|\geq\left(\frac{n}{16}\right)^{d/2}$.

Because $\mathcal{D}$ uses less than $\frac{d\log n}{3}$ bits of memory, then $\mathcal{D}$ has at most $2^{\frac{d\log n}{3}}=n^{d/3}$ unique memory configurations. Since $|X|\geq\left(\frac{n}{16}\right)^{d/2}$, then there are at least $\frac{1}{2}(|X|-n^{d/3})\geq\frac{|X|}{4}$ pairs $x,x^{\prime}$ such that $\mathcal{D}$ has the same configuration after reading $x$ and $x^{\prime}$. We show that $\mathcal{D}$ errs on a significant fraction of these pairs $x,x^{\prime}$.

Let $\mathcal{I}$ be the positions where either $x$ or $x^{\prime}$ take value $1$, so that $d+1\leq|\mathcal{I}|\leq 2d$. Observe that if ${\mathsf{HAM}}(x,y)=d$, but $x$ and $y$ do not differ in any positions of $\mathcal{I}$, then ${\mathsf{HAM}}(x^{\prime},y)>d$. Recall that $\mathcal{D}$ has the same configuration after reading $x$ and $x^{\prime}$, so then $\mathcal{D}$ has the same configuration after reading $s(x,y)$ and $s(x^{\prime},y)$. But since ${\mathsf{HAM}}(x,y)=d$ and ${\mathsf{HAM}}(x^{\prime},y)>d$, then the output of $\mathcal{D}$ is incorrect for either $s(x,y)$ or $s(x^{\prime},y)$.

For each pair $(x,x^{\prime})$, there are $\binom{n/4-|\mathcal{I}|}{d}\geq\binom{n/4-2d}{d}$ such $y$ with ${\mathsf{HAM}}(x,y)=d$, but $x$ and $y$ do not differ in any positions of $\mathcal{I}$. Hence, there are $\frac{|X|}{4}\binom{n/4-2d}{d}$ strings $s(x,y)$ for which $\mathcal{D}$ errs. We note that there is no overcounting because the output of $\mathcal{D}$ can be correct for at most one $s(x_{i},y)$ for all $x_{i}$ mapped to the same configuration. Recall that $y$ satisfies either ${\mathsf{HAM}}(x,y)=d$ or ${\mathsf{HAM}}(x,y)=d+1$ so that there are $|X|\left(\binom{n/4}{d}+\binom{n/4}{d+1}\right)$ strings $s(x,y)$ in total. Thus, the probability of error is at least

[TABLE]

where the last line holds for large $n$, from Bernoulli’s Inequality and $d=o(\sqrt{n})$. $\Box$

Define an infinite string $1^{1}0^{1}1^{2}0^{2}1^{3}0^{3}\ldots$, and let $\nu$ be the prefix of length $\frac{n}{4}$. Given $x$ and $y$ from the above distribution, define string $s(x,y)=\nu^{R}xy^{R}\nu$ so that $s(x,y)$ is a $d$-near-palindrome of length $n$ if ${\mathsf{HAM}}(x,y)\leq d$.

Lemma 7.2

If ${\mathsf{HAM}}(x,y)\geq d+1$, then the longest $d$-near-palindrome of $s(x,y)$ has length at most $200d^{2}+\frac{n}{2}$.

**Proof :  ** Suppose, by way of contradiction, that the longest $d$-near-palindrome of $s(x,y)$ has length at least $200d^{2}+\frac{n}{2}$. Since $\nu$ has length $\frac{n}{4}$ and ${\mathsf{HAM}}(x,y)>d$, then the midpoint $m$ of the longest $d$-near-palindrome of $s(x,y)$ lies within $x$ or $y$. Suppose that the midpoint is in $x$, so that $m<\frac{n}{2}$. We consider the cases where $m<\frac{n}{2}-8d$ and $m\geq\frac{n}{2}-8d$.

If $m<\frac{n}{2}-8d$, then at least $8d$ characters of $\nu$ coincide with characters of $xy$ in the reverse. However, the final $8d$ characters of $\nu$ contain at least $4d$ many $1$’s while the characters of $xy$ contain at most $2d+1$ many $1$’s, and so the Hamming distance is at least $2d-1$, which is a contradiction. On the other hand, if $m\geq\frac{n}{2}-8d$, then at least $200d^{2}$ characters of $\nu$ and $\nu^{R}$ coincide. But because $m<\frac{n}{2}$, then the midpoint is closer to the end of $\nu^{R}$ than the beginning of $\nu$. Hence, for $k>8d$, each consecutive run of $k$ many $1$’s in $\nu^{R}$ corresponds with a [math] in $\nu$. But then by the time $\nu^{R}$ has a consecutive run of $10d$ many $1$’s, the Hamming distance is at least $2d-1$, which is a contradiction. Since $\nu^{R}$ has a consecutive run of $10d$ many $1$’s by the index $(10d)^{2}=100d^{2}$, then the longest $d$-near-palindrome has length at most $200d^{2}+\frac{n}{2}$.

A similar argument follows if $m\geq\frac{n}{2}-8d$, so that the midpoint is in $y$. $\Box$

Since $d=o(\sqrt{n})$, then any algorithm with approximation factor $(1+\epsilon)$ can distinguish whether the longest $d$-near-palindrome in $s(x,y)$ has length $n$ or at most $200d^{2}+\frac{n}{2}$, for large $n$ and small and constant $\epsilon$. In turn, this algorithm can distinguish between ${\mathsf{HAM}}(x,y)=d$ and ${\mathsf{HAM}}(x,y)>d$ by Lemma 7.2. However, by Lemma 7.1, any algorithm using less than $\frac{d\log n}{3}$ bits of memory cannot distinguish between ${\mathsf{HAM}}(x,y)=d$ and ${\mathsf{HAM}}(x,y)>d$ with probability at least $1-1/n$.

Therefore, $\Omega(d\log n)$ bits of memory are necessary to $(1+\epsilon)$-approximate the length of the longest $d$-near-palindrome with probability at least $1-\frac{1}{n}$. $\Box$

*Reminder of Theorem 1.5. *** Let $d=o(\sqrt{n})$ and $E>d$ be an integer. Any randomized streaming algorithm that returns an estimate $\hat{\ell}$ of the length of the longest $d$-near-palindrome, where $\hat{\ell}\leq\ell_{max}\leq\hat{\ell}+E$, with probability at least $1-\frac{1}{n}$, must use $\Omega\left(\frac{dn}{E}\right)$ bits of space.

**Proof of Theorem 1.5:  ** We use a similar strategy as in Theorem 1.4 and analyze deterministic algorithms using less than $\frac{dn}{12E}$ memory, on a special hard distribution of inputs.

For $n^{\prime}>0$, which we pick shortly, let $X$ be the set of binary strings of length $\frac{n^{\prime}}{2}$. Given $x\in X$, let $Y_{x}$ be the set of binary strings of length $\frac{n^{\prime}}{2}$ with either ${\mathsf{HAM}}(x,y)=d$ or ${\mathsf{HAM}}(x,y)=d+1$. We pick $(x,y)$ uniformly at random from $(X,Y_{x})$.

Lemma 7.3

Given an input $x\circ y$, any deterministic algorithm $\mathcal{D}$ which uses less than $\frac{n^{\prime}}{4}$ bits of memory cannot correctly output whether ${\mathsf{HAM}}(x,y)\leq d$ or ${\mathsf{HAM}}(x,y)>d+1$ with probability at least $1-\frac{1}{n^{\prime}}$, for $d=o(\sqrt{n})$.

**Proof :  ** Because $\mathcal{D}$ uses less than $\frac{n^{\prime}}{4}$ bits of memory, then $\mathcal{D}$ has at most $2^{n^{\prime}/4}$ unique memory configurations. Since $|X|=2^{n^{\prime}/2}$, then there are at least $\frac{1}{2}(|X|-2^{n^{\prime}/4})\geq\frac{|X|}{4}$ pairs $x,x^{\prime}$ such that $\mathcal{D}$ has the same configuration after reading $x$ and $x^{\prime}$. We show that $\mathcal{D}$ errs on a significant fraction of these pairs $x,x^{\prime}$.

Let $\mathcal{I}$ be the positions where $x$ and $x^{\prime}$ differ, so that ${\mathsf{HAM}}(x,x^{\prime})=|\mathcal{I}|>0$. Consider $i\neq\frac{|\mathcal{I}|}{2}$, so that either $i>\frac{|\mathcal{I}|}{2}$ or $i<\frac{|\mathcal{I}|}{2}$. If $i<\frac{|\mathcal{I}|}{2}$, let $y$ differ from $x$ in $i$ positions (where $i\leq d$) of $\mathcal{I}$ and in $d-i$ positions outside of $\mathcal{I}$. Then ${\mathsf{HAM}}(x,y)=d$, but ${\mathsf{HAM}}(x^{\prime},y)>d$. Similarly, let $y^{\prime}$ differ from $x^{\prime}$ in the $i$ positions of $\mathcal{I}$ and $d-i$ positions outside of $\mathcal{I}$ so that ${\mathsf{HAM}}(x^{\prime},y^{\prime})=d$, but ${\mathsf{HAM}}(x,y^{\prime})>d$. Hence, $\mathcal{D}$ errs on either $(x,y)$ or $(x^{\prime},y^{\prime})$.

There are at least $\sum_{i=0}^{\frac{|\mathcal{I}|}{2}-1}\binom{|\mathcal{I}|}{i}\binom{n^{\prime}/2-|\mathcal{I}|}{d-i}$ such $y$ for each pair $(x,x^{\prime})$.

Similarly, if $i>\frac{|\mathcal{I}|}{2}$, let $y$ differ from $x$ in the $i$ positions (where $i\leq d$) of $\mathcal{I}$ and in $d+1-i$ positions outside of $\mathcal{I}$. Then ${\mathsf{HAM}}(x,y)=d+1$, but ${\mathsf{HAM}}(x^{\prime},y)\leq d$. Similarly, let $y^{\prime}$ differ from $x^{\prime}$ in the $i$ positions of $\mathcal{I}$ and $d+1-i$ positions outside of $\mathcal{I}$ so that ${\mathsf{HAM}}(x^{\prime},y^{\prime})=d$, but ${\mathsf{HAM}}(x^{\prime},y^{\prime})>d$. Hence, $\mathcal{D}$ errs on either $(x,y)$ or $(x^{\prime},y^{\prime})$. There are at least $\sum_{\frac{|\mathcal{I}|}{2}+1}^{d}\binom{|\mathcal{I}|}{i}\binom{n^{\prime}/2-|\mathcal{I}|}{d+1-i}$ such $y$ for each pair $(x,x^{\prime})$.

The total number of such $y$ is therefore at least:

[TABLE]

Applying Vandermonde’s identity, the total number of such $y$ is at least $\binom{n^{\prime}/2}{d}-\binom{|\mathcal{I}|}{|\mathcal{I}|/2}\binom{n^{\prime}/2-|\mathcal{I}|}{d-|\mathcal{I}|/2}$. Recall that $\mathcal{I}$ is the number of indices in which $x$ and $x^{\prime}$ differ, so $|\mathcal{I}|\geq 1$. Thus,

[TABLE]

where the last inequality comes from $d=o(\sqrt{n})$. Therefore, for each pair $(x,x^{\prime})$ the total number of errors by $\mathcal{D}$, as it cannot distinguish between $(x,y)$ and $(x^{\prime},y^{\prime})$, is at least $\frac{1}{2}\binom{n/2}{d}\left(1-\frac{1}{\sqrt{n}}\right)\geq\frac{1}{4}\binom{n/2}{d}$.

Since there are $\frac{|X|}{4}$ pairs of $(x,x^{\prime})$, then there are at least $\frac{|X|}{16}\binom{n/2}{d}$ pairs $(x,y)$ for which $\mathcal{D}$ errs. Recall that $y$ satisfies either ${\mathsf{HAM}}(x,y)=d$ or ${\mathsf{HAM}}(x,y)=d+1$ so that there are $|X|\left(\binom{n^{\prime}/2}{d}+\binom{n^{\prime}/2}{d+1}\right)$ pairs $(x,y)$ in total. Thus, the probability of error is at least

[TABLE]

Therefore for $d=o(\sqrt{n^{\prime}})$, $\mathcal{D}$ fails with probability at least $\frac{1}{n^{\prime}}$. $\Box$

Given strings $x$ and $y$ from the above distribution, define string

[TABLE]

where $x_{i}$ represents the i$${}^{\mbox{\tiny{{th}}}} character of $x$ and $\textbf{1}^{\ell}$ represents $\ell$ repetitions of $1$. Let $s(x,y)$ be an input to $\mathcal{D}$ so that $s(x,y)$ has length $n=\left(\frac{E}{d}+1\right)(n^{\prime}-2)+2E+2\leq\frac{3E}{d}n^{\prime}$. Note if ${\mathsf{HAM}}(x,y)\leq d$, then $s(x,y)$ is a $d$-near-palindrome of length $\left(\frac{E}{d}+1\right)(n^{\prime}-2)+2E+2$. However, if ${\mathsf{HAM}}(x,y)>d$, then the longest $d$-near-palindrome of $s(x,y)$ has length at most $\left(\frac{E}{d}+1\right)(n^{\prime}-2)$. Consequently, any algorithm with additive error $E$ can be run on $s(x,y)$ to distinguish between ${\mathsf{HAM}}(x,y)\leq d$ and ${\mathsf{HAM}}(x,y)\geq d+1$. However, by Lemma 7.3, any algorithm using less than $\frac{n^{\prime}}{4}$ bits of memory cannot distinguish between ${\mathsf{HAM}}(x,y)=d$ and ${\mathsf{HAM}}(x,y)>d$ with probability at least $1-\frac{1}{n^{\prime}}>1-\frac{1}{n}$. Since $\frac{dn}{12}\leq\frac{n^{\prime}}{4}$, then it follows that for $d=o(\sqrt{n})$ and $E>d$, any randomized streaming algorithm which returns an additive approximation of $E$ to the length of the longest $d$-near-palindrome, with probability at least $1-\frac{1}{n}$, uses $\Omega\left(\frac{dn}{E}\right)$ space. $\Box$

Acknowledgments

We would like to thank Funda Ergün, Tatiana Kuznetsova, and Qin Zhang for helpful discussions and pointers. We would also like to thank anonymous reviewers for pointing out an error in the proof of Theorem 1.5.