Fine-Grained Complexity of Analyzing Compressed Data: Quantifying Improvements over Decompress-And-Solve

TL;DR

This paper investigates the complexity of analyzing compressed data directly, establishing bounds and optimality results for various problems under common compression schemes, and introduces a framework for conditional lower bounds.

Contribution

It provides a unified framework for proving lower bounds on analyzing compressed data and determines the optimality of decompress-and-solve for several fundamental problems.

Findings

LCS and Pattern Matching with Wildcards bounds are optimal under ETH.

Decompress-and-solve is optimal for Grammar Parsing and RNA Folding under the k-Clique conjecture.

Decompress-and-solve is not optimal for Disjointness, with a new algorithm.

Abstract

Can we analyze data without decompressing it? As our data keeps growing, understanding the time complexity of problems on compressed inputs, rather than in convenient uncompressed forms, becomes more and more relevant. Suppose we are given a compression of size $n$ of data that originally has size $N$, and we want to solve a problem with time complexity $T(\cdot)$. The naive strategy of "decompress-and-solve" gives time $T(N)$, whereas "the gold standard" is time $T(n)$: to analyze the compression as efficiently as if the original data was small. We restrict our attention to data in the form of a string (text, files, genomes, etc.) and study the most ubiquitous tasks. While the challenge might seem to depend heavily on the specific compression scheme, most methods of practical relevance (Lempel-Ziv-family, dictionary methods, and others) can be unified under the elegant notion of Grammar Compressions. A vast literature, across many disciplines, established this as an influential notion for Algorithm design. We introduce a framework for proving (conditional) lower bounds in this field, allowing us to assess whether decompress-and-solve can be improved, and by how much. Our main results are: - The $O(nN\sqrt{\log{N/n}})$ bound for LCS and the $O(\min\{N \log N, nM\})$ bound for Pattern Matching with Wildcards are optimal up to $N^{o(1)}$ factors, under the Strong Exponential Time Hypothesis. (Here, $M$ denotes the uncompressed length of the compressed pattern.) - Decompress-and-solve is essentially optimal for Context-Free Grammar Parsing and RNA Folding, under the $k$-Clique conjecture. - We give an algorithm showing that decompress-and-solve is not optimal for Disjointness.

Equations

$$W$$

$$={◯}_{i\in [2^{\tilde{n}/2}]}{◯}_{\ell \in [\tilde{m}]}\text{sat}({\alpha }_i,C_{\ell }).$$

$$T$$

$$={◯}_{\ell \in [\tilde{m}]}{◯}_{i\in [2^{\tilde{n}/2}]}\text{sat}({\alpha }_i,C_{\ell }).$$

$$A_0$$

$$A_i$$

$$S_0$$

$$S_i$$

$$T={◯}_{\ell \in [\tilde{m}]}{◯}_{i\in [2^{\tilde{n}}]}\text{sat}({\alpha }_i,C_{\ell })P=1({\ast }^{2^{\tilde{n}}-1}1{)}^{\tilde{m}-1}.$$

$$T=\mathop{\bigcirc}_{\ell=1}^{d}\mathop{\bigcirc}_{a_{1}\in{\cal A}^{(1)}}\ldots\mathop{\bigcirc}_{a_{k}\in{\cal A}^{(k)}}\big{[}a_{1}[\ell]=\ldots=a_{k}[\ell]=1\big{]},$$

$$T:={\left(!\circ {◯}_{i=1}^A\#T(a_i)\right)}^B.$$

$$z_j^{(k-1)}\stackrel{1}{\to }\{\begin{array}{ll}{z}_j^{(k)}& \text{if }{b}_j[k]=0,\\ z_j^{\mathrm{fail}}& \text{otherwise.}\end{array}$$

$$g_0\stackrel{!}{\to }{g}_1$$

$$g_j\stackrel{\#}{\to }{z}_j^{(0)}$$

$$z_{j-1}^{\mathrm{fail}}\stackrel{!}{\to }{g}_j$$

$$z_j^{\mathrm{fail}}\stackrel{\#}{\to }{z}_j^{(0)}$$

$$z_j^{(d)}\stackrel{\{\#,!\}}{\to }{z}_j^{(d)}$$

$$r_i=\sum _{\sigma \in \Sigma }{w}_i^{\sigma }=\sum _{\sigma \in \Sigma }\sum _{j=1}^M{u}_j^{\sigma }{v}_{i+j}^{\sigma }=\sum _{\sigma \in \Sigma }\sum _{j=1}^M[P[j]=\sigma ]\cdot \mathrm{cost}(\sigma ,T[i+j])=\sum _{j=1}^M\mathrm{cost}(P[j],T[i+j]),$$

$$\text{Match}(i):=\underset{0\le d\le |T_i|-M}{\min }\sum _{j=1}^M\mathrm{cost}(P[j],T_i[j+d]),$$

$$\text{FixMatch}(i,d):=\sum _{j:\begin{array}{c}1\le j+d\le |T_i|,\\ 1\le j\le M\end{array}}\mathrm{cost}(P[j],T_i[j+d]).$$

$$\text{FixMatch}(i,d)=\text{FixMatch}(\ell ,d)+\text{FixMatch}(r,d-|T_{\ell }|),$$

$$\textup{Match}(i)=\min\Big{\{}\textup{Match}(\ell),\,\textup{Match}(r),\,\min_{-M<d<0}\textup{FixMatch}(r,d)+\textup{FixMatch}(\ell,|T_{\ell}|+d)\Big{\}}.$$

$$V$$

$$\displaystyle:=\mathop{\bigcirc}_{\ell=1}^{d}\mathop{\bigcirc}_{i_{1},\dots,i_{k}\in[A]}S\Big{(}b[\ell]\cdot a_{i_{1}}[\ell]\cdots a_{i_{k}}[\ell]\Big{)},$$

$$V_{\Delta }=V_{\Delta +A^k}=\dots =V_{\Delta +(d-1)A^k}=S(0).$$

$$b[\ell ]\cdot a_{i_1}[\ell ]\cdots a_{i_k}[\ell ]=0\text{for all }\ell \in [d].$$

$$S_0$$

$${\mathrm{Test}}_{\ell }$$

$${\mathrm{List}}_{\ell }^{(j)}$$

$${\mathrm{List}}_{\ell }^{(k+1)}$$

$$T=\mathrm{tuplify}(\mathcal{A},k,b,S(0),S(1)).$$

$$P=0({\ast }^{A^k-1}0{)}^{d-1}.$$

$$T={◯}_{(j_1,\dots ,j_{k_2})\in [A{]}^{k_2}}{1}^{A^{k_1}}\circ \mathrm{tuplify}(\mathcal{A},k_1,\min (a_{j_1},\dots ,a_{j_{k_2}}),0,1),$$

$$P=0({\ast }^{A^{k_1}-1}0{)}^{d-1}.$$

$$N$$

$$M$$

$$n$$

Full text

We restrict our attention to data in the form of a string (text, files, genomes, etc.) and study the most ubiquitous tasks. While the challenge might seem to depend heavily on the specific compression scheme, most methods of practical relevance (Lempel-Ziv-family, dictionary methods, and others) can be unified under the elegant notion of Grammar-Compressions. A vast literature, across many disciplines, established this as an influential notion for Algorithm design.

We introduce a framework for proving (conditional) lower bounds in this field, allowing us to assess whether decompress-and-solve can be improved, and by how much. Our main results are:

• •

The $O(nN\sqrt{\log{N/n}})$ bound for LCS and the $O(\min\{N\log N,nM\})$ bound for Pattern Matching with Wildcards are optimal up to $N^{o(1)}$ factors, under the Strong Exponential Time Hypothesis. (Here, $M$ denotes the uncompressed length of the compressed pattern.)

• •

Decompress-and-solve is essentially optimal for Context-Free Grammar Parsing and RNA Folding, under the $k$-Clique conjecture.

• •

We give an algorithm showing that decompress-and-solve is not optimal for Disjointness.

1. 1 Introduction

2. 1.1 Previous Work

3. 1.2 Our Work

4. 1.3 Technical Overview

5. 2 Preliminaries

6. 2.1 Hardness Assumptions

7. 3 Tight Bounds Assuming SETH

8. 3.1 DFA Acceptance

9. 3.2 Approximate Pattern Matching and Substring Hamming Distance

10. 3.3 Longest Common Subsequence

11. 3.3.1 Alignment Gadget Framework

12. 3.3.2 General Lower Bound

13. 3.3.3 Extended Alignment Gadget for LCS

14. 4 Tight Bounds Assuming (Combinatorial) $k$-Clique

15. 4.1 NFA Acceptance

16. 4.2 Context-Free Grammar Parsing

17. 4.3 RNA Folding

18. 5 Disjointness, Hamming Distance, and Subsequence

19. 5.1 Algorithms

20. 5.2 Lower Bounds

21. 6 Conclusion

1 Introduction

Computer Science is often called the science of processing digital data. A central goal of theoretical CS is to understand the time complexity of the tasks we want to perform on data. Data compression has been one of the most important notions in CS and Information Theory for decades, and it is increasingly relevant in our current age of “Big Data” where it is hard to think of reasons why not to compress our data: smaller data can be stored more efficiently, transmitting it takes less resources such as energy and bandwidth, and perhaps it can even be processed faster. Since nowadays and for years to come nearly all of our data comes in compressed form, a central question becomes:

What is the time complexity of analyzing compressed data?

Say we have a piece of data of size $N$ given in a compressed form of size $n$. For a problem with time complexity $T(\cdot)$, the naïve strategy of “decompress and solve” takes $\Theta(T(N))$ time, while the “gold standard” is $O(T(n))$ time: we want to solve the problem on the compression as efficiently as if the original data was small. To provide meaningful statements we need to decide on three things: What type of data is it? What problem do we want to solve? Which compression scheme is being used?

For the first two questions, the focus of this paper will be on the most basic setting. We consider data that comes as strings, i.e. sequences of symbols such as text, computer code, genomes, and so on. And we study natural and basic questions one could ask about strings such as Pattern Matching, Language Membership, Longest Common Subsequence, Parsing, and Disjointness.

For the third question, we restrict our attention to lossless compression and, even then, there are multiple natural settings that we do not find to be the most relevant. We could consider Kolmogorov complexity, giving us the best possible compression of our data: assume that a string $T$ is given by a short bitstring $K(T)$ which is a pair of Turing machine $M$ and input $x$ such that running $M$ on $x$ outputs $T$, i.e. $K(T)=\langle M,x\rangle$ such that $M(x)=T$. The issue with Kolmogorov-compressions is that none of our data comes in this form, for two good reasons: First, it is computationally intractable to compute $K(T)$ given $T$, not even approximately. And second, analyzing arbitrary Turing machines without just running them is an infamously hopeless task. Thus, while studying the time complexity of analyzing Kolmogorov-compressed strings is natural, it might not be the most relevant for computer science applications. Another option is to consider the mathematically simplest forms of compression such as Run-Length Encoding (RLE): we compress $x$ consecutive letters $\sigma$ into $\sigma^{x}$, so the compression has the form $0^{x_{1}}1^{x_{2}}0^{x_{3}}\cdots 1^{x_{\ell}}$, and we only need $n=O(\ell\cdot\log{N})$ bits to describe the potentially exponentially longer string of length $N$. This compression is at the other extreme of the spectrum: it is trivial to compute and easy to analyze, but it is far less “compressing” than popular schemes like Lempel-Ziv-compressions.

Instead, we consider what has proven to be one of the most influential kinds of compression for Algorithm design, namely Grammar-Compressions, a notion that has all the right properties. First, it is mathematically elegant and quite fun to reason about for theoreticians (as evidenced by the many pages of our paper). Second, it is equivalent [64] up to low order terms (moderate constants and log factors) to popular schemes like the Lempel-Ziv-family (LZ77, LZ78, LZW, etc.) [48, 81, 75], Byte-Pair Encoding [68], dictionary methods, and others [57, 50]. These compressions are used in ubiquitous applications such as the built-in Unix utility compress, zip, GIF, PNG, and even in PDF. Third, it is generic and likely to capture compression schemes that will be engineered in the future (after all, there is a whole industry on the topic and the quest might never be over). Fourth, we can compute the optimal such compression (up to log factors) in linear time [64, 23, 44]. And last but not least, ingenious algorithmic techniques have shown that it is possible to computationally analyze grammar-compressed data, beating the “decompress and solve” bound for many important problems.

A grammar compression of a string $X$ is simply a context-free grammar, whose language is exactly $\{X\}$, that is, the only string the grammar can produce is $X$. For the purposes of this paper, it is enough to focus on a restricted form of grammars, known as Straight Line Programs (SLP). An SLP is defined over some alphabet $\Sigma$, say $\{0,1\}$, and it is a set of replacement rules (or productions) of a very simple form: a rule is either a symbol in $\Sigma$ or it is the concatenation of two previous rules (under some fixed ordering of the rules). The last replacement rule is the sequence defined by the SLP. For example, we can compress the sequence $01011$ with the rules S_{1}\to 0;\ \ S_{2}\to 1;\ \ S_{3}\to S_{1}\,S_{2};\ \ S_{4}\to S_{3}\,S_{3};\ \ S_{5}\to S_{4}\,S_{2}\ and $S_{5}$ corresponds to the sequence $01011$. For some strings this can give an exponential compression. A more formal definition and a figure are given in Section 2.

To learn more about the remarkable success of grammar-compressions, we refer the reader to the surveys [77, 47, 34, 67, 36, 63, 65, 53, 66]. As a side remark, one of the exciting developments in this context was the surprising observation that a “compress and solve” strategy could actually lead to theoretically new algorithms for some problems, e.g. [60, 45].

Thus, we focus on what we find the most important interpretation of the central question above:

What is the time complexity of basic problems on grammar-compressed strings?

1.1 Previous Work

As a motivating example, consider the Longest Common Subsequence (LCS) problem. Given two uncompressed strings of length $N$ we can find the length of the longest common (not necessarily contiguous) subsequence in $O(N^{2})$ time using dynamic programming, and there are almost-matching $N^{2-o(1)}$ conditional lower bounds [2, 17, 3]. Throughout the paper we mostly ignore log factors, and so we think of LCS as a problem with $\tilde{\Theta}(N^{2})$ time complexity (on uncompressed data). Now, assume our sequences are given in compressed form of size $n$. A natural setting to keep in mind is where $n\approx N^{1/2}$. How much time do we need to solve LCS on these compressed strings? The naïve upper bound gives $O(N^{2})$ and the gold standard is $O(n^{2})\approx O(N)$, so which is it?

Besides being a very basic question, LCS and the closely related Edit Distance are a popular theoretical modeling of sequence alignment problems that are of great importance in Bioinformatics111The heuristic algorithm BLAST for a generalized version of the problem has received sixty-thousand citations.. Thus, this is a relatively faithful modeling of the question whether “compress-and-solve” can speed up genome analysis tasks, a question which has received extensive attention throughout the years [39, 57, 50, 38, 36].

A long line of work [18, 54, 7, 8, 27, 69, 70, 40] has shown that we can do much better than $O(N^{2})$. The current best algorithm has the curious runtime $O(nN\sqrt{\log{N/n}})$ [35] which is tantalizingly close to a conjectured bound of $O(nN)$ from the seminal paper of Lifshits [49]. In our candidate setting of $n\approx N^{1/2}$, this is $\tilde{O}(N^{1.5})$. This is major speedup over the $\Omega(N^{2})$ decompress-and-solve bound, but is still far away from the gold standard of $O(n^{2})$ which in this case would be $O(N)$. Can we do better? For example, an $O(n^{2}\cdot N^{0.1})$ bound could lead to major real-world improvements.

While there is a huge literature on the topic, both from the Algorithms community and from applied areas, in addition to the potential for real-world impact, studying these questions has not become a mainstream topic in the top algorithms conferences. In one of the only STOC/FOCS papers on the topic, Charikar et al. [23] write “In short, the smallest grammar problem has been considered by many authors in many disciplines for many reasons over a span of decades. Given this level of interest, it is remarkable that the problem has not attracted greater attention in the general algorithms community.”

We believe that one key reason for this is the lack of a relevant complexity theory and tools for proving lower bounds, leaving a confusing state of the art in which it is hard to distinguish algorithms providing fundamental new insights from ad hoc solutions. Most importantly, previous work has not given us the tools to know, when we encounter a data analysis problem in the real-world, what kind of upper bound we should expect. Instead, researchers have been proving P vs. NP-hard results, classifying problems into ones solvable in $\textup{poly}(n,\log{N})$ time and ones that probably require time $N^{\Omega(1)}$. In fact, even LCS is NP-hard [49]. This means that even if we have a compression of very small size $n=O(\log{N})$ then we cannot solve LCS in $\textup{poly}(n)$ time, unless $\textup{P}=\textup{NP}$. Dozens of such negative results have been proven (see [53]), and it has long been clear that almost any task of interest is “NP-hard”, including the basic $\textup{poly}(N)$ time solvable problems we discuss in this paper. However, this is hardly relevant to the questions we ask in this paper since it does not address the possibility of highly desirable bounds such as $n^{2}\cdot N^{0.1}$. What we would really like to know is whether the bound should be $\textup{poly}(n)\cdot N^{\varepsilon}$, or $\textup{poly}(n)\cdot N$, or even higher: could it be that decompress-and-solve is impossible to beat for some problems?

1.2 Our Work

In this work, we introduce a framework for showing lower bounds on the time complexity of problems on grammar-compressed strings. Our lower bounds are based on popular conjectures from Hardness in P and Fine-Grained Complexity. This is perhaps surprising since the problems we consider are technically NP-hard. Our new complexity theoretic study of this field leads to three exciting developments: First, we resolve the exact time complexity up to $N^{o(1)}$ factors of some of the most classical problems such as LCS on compressed data. Second, we discover problems that cannot be solved faster than the decompress-and-solve bound by any $N^{\varepsilon}$ factor. Third, we fail at proving tight lower bounds for some classical problems, which hints to us that known algorithms might be suboptimal. Indeed, in this paper we also find new algorithms for fundamental problems. We hope that our work will inspire increased interest in this important topic.

Longest Common Subsequence

Our first result is a resolution of the time complexity of LCS on compressed data, up to $N^{o(1)}$ factors, under the Strong Exponential Time Hypothesis222SETH is the pessimistic version of $\textup{P}\neq\textup{NP}$, stating that we cannot solve $k$-SAT in $O((2-\varepsilon)^{n})$ time, for some $\varepsilon>0$ independent of and for all constant $k$ [42, 19]. (SETH). We complement the $O(nN\sqrt{\log{N/n}})$ upper bound of Gawrychowski [35] with an $(nN)^{1-o(1)}$ lower bound. Thus, in the natural setting $n\approx N^{1/2}$ from above, we should indeed be content with the $\tilde{O}(N^{1.5})$ upper bound since we will not be able to get much closer to the gold standard, unless SETH fails. Assuming SETH, our result confirms the conjecture of Lifshits, up to $N^{o(1)}$ factors. See Theorem 3.12 in Section 3.3 for the formal statement.

One way to view this result is as an Instance Optimality result for LCS. The exact complexity of LCS on two strings is precisely proportional to the product of the decompressed size $N$ and the instance-inherent measure $n$ of how compressible they are.

RNA Folding and CFG Parsing

Next, we turn our attention to two other fundamental problems: Context-Free Grammar Recognition (aka Parsing) and RNA Folding. Parsing is the core computer science problem in which we want to decide whether a given string (e.g. computer code) can be derived from a given grammar (e.g. the grammar of a programming language). Having the ability to efficiently parse a compressed file is certainly desirable. In RNA Folding we are given a string over some alphabet (e.g. $\{A,C,G,T\}$) with a fixed pairing between its symbols (e.g. $A-T$ match and $C-G$ match), and the goal is to compute the maximum number of non-crossing arcs between matching letters that one can draw above the string (which corresponds to the minimum energy folding in two dimensions). RNA Folding is one of the most central problems in bioinformatics, and as we have discussed above, the ability to analyze compressed data is important in this field. How fast can we solve these problems?

Given an uncompressed string of size $N$, classical dynamic programming algorithms, such as the CYK parser [25, 80, 46], solve RNA Folding in $O(N^{3})$ time and Parsing in $O(N^{3}\cdot g)$ time if the grammar has size $g$. Wikipedia lists twenty-four parsing algorithms designed throughout the years, all of which take cubic time in the worst case. A theoretical breakthrough of Leslie Valiant [72] in 1975 showed that there are truly sub-cubic $O(gN^{\omega})$ parsing algorithms, where $\omega<2.38$ is the fast matrix multiplication (FMM) exponent. However, Valiant’s algorithm has not been used in practice due the inefficiency of FMM algorithms, and obtaining a combinatorial333For the purposes of this paper, “combinatorial” should be interpreted as any practically efficient algorithm that does not suffer from the issues of FMM such as large constants and inefficient memory usage. sub-cubic time algorithm would be of major interest. Alas, it was recently proved [1] that any improvement over these bounds implies breakthrough $k$-Clique algorithms: either finding such a combinatorial subcubic algorithm or getting any $O(N^{\omega-\varepsilon})$ time algorithm, for any $\varepsilon>0$, would refute the $k$-Clique Conjecture444Given a graph on $n$ nodes, the $k$-Clique conjecture [1] is in fact two independent conjectures: The first one states that we cannot solve $k$-clique in $O(n^{(1-\varepsilon)\cdot\omega k/3})$, for any $\varepsilon>0$. The second one states that we cannot solve $k$-Clique combinatorially in $O(n^{(1-\varepsilon)k})$ time, for any $\varepsilon>0$.. The situation for RNA is even more interesting since Valiant’s sub-cubic algorithm does not generalize to this case. Under the $k$-Clique conjecture, the same lower bounds still apply [1, 22], implying that any improvement will have to use FMM. Indeed, an $O(N^{2.82})$ algorithm using FMM was recently achieved [15].

Cubic time is a real bottleneck when analyzing large genomic data. One would hope that if we are able to compress the data down to size $n$ we could solve problems like RNA Folding and Parsing in time that is much faster than the $N^{3}$ lower bounds (to simplify the discussion we focus on combinatorial algorithms), such as $n^{3}\cdot N^{o(1)}$ or at least $n^{1.5}N^{1.5}$, in certain analogy the LCS case. No such algorithms were found to date, and we provide an explanation: Decompress-and-solve cannot be beaten for Parsing and (essentially) for RNA Folding, under the $k$-Clique Conjecture. For both problems we prove a conditional lower bound of $N^{\omega-o(1)}$ for any kind of algorithm, and $N^{3-o(1)}$ for combinatorial algorithms, even restricted to $n=O(N^{\varepsilon})$ for any $\varepsilon>0$. See Theorem 4.4 in Section 4.2 for CFG Parsing and Theorem 4.10 in Section 4.3 for RNA Folding.

Approximate Pattern Matching

We continue our quest for quantifying the possible improvements over decompress-and-solve for basic problems. Consider the following compressed versions of important primitives in text analysis known as Approximate Pattern Matching problems. In all these problems we assume that we are given a compressed text $T$ of size $n$ (and decompressed size $N$), and a compressed pattern $P$ of size $m$ (and decompressed size $M$), both over some constant size alphabet.

• •

Pattern Matching with Wildcards: In this problem, the strings contain wildcard symbols that can be replaced by any letter, and our goal is to decide if $P$ appears in $T$.

• •

Substring Hamming Distance: Compute the smallest Hamming distance of any substring of $T$ to $P$.

And a problem that generalizes both is:

• •

Generalized Pattern Matching: Given some cost function on pairs of alphabet symbols, find the length-$M$ substring $T^{\prime}$ of $T$ minimizing the total cost of all pairs $(T^{\prime}[i],P[i])$.

The above problems have been extensively studied both in the uncompressed (see [24]) and in the compressed [49, 13, 32] settings. All three problems can be solved in time $O(\min\{N\log N,nM\})$ (see Section 3.2). Note that this bound beats the decompress-and-solve bound when the pattern is small, but can we avoid decompressing the pattern? We show a completely tight SETH-based lower bound of $\min\{N,nM\}^{1-o(1)}$ for all three problems, even for constant size alphabets and in all settings where the parameters are polynomially related. See Theorems 3.9 and 3.10 in Section 3.2.

Language Membership

Consider the compressed version of the most basic language membership problems. Assume we are given a compressed string $T$ (again, from size $N$ to $n$).

• •

DFA Acceptance: Given $T$ and a DFA $F$ with $q$ states, decide whether $F$ accepts $T$.

• •

NFA Acceptance: Given $T$ and a NFA $F$ with $q$ states, decide whether $F$ accepts $T$.

Classic algorithms solve the DFA Acceptance problem in time $O(\min\{nq,N+q\})$ [61, 41], and we prove a matching SETH-based lower bound of $\min\{nq,N+q\}^{1-o(1)}$. See Theorem 3.2 in Section 3.1.

For the NFA problem, the classic algorithms give $O(\min\{nq^{\omega},Nq^{2}\})$ [55, 61, 41]. For combinatorial algorithms, we prove a matching lower bound of $\min\{nq^{3},Nq^{2}\}^{1-o(1)}$, under the (combinatorial) $k$-Clique conjecture. See Theorem 4.2 in Section 4.1. Our lower bounds hold for constant size alphabets, and in all settings of $n,N,q$, even restricted to instances with $N=\Theta(n^{\alpha_{N}})$ and $q=\Theta(n^{\alpha_{q}})$ for any $\alpha_{N}>1$ and $\alpha_{q}>0$.

Disjointness, Hamming Distance, and Subsequence

Could it be that for other, even more basic problems the decompress-and-solve bound cannot be beaten? One candidate might be Disjointness, the canonical hard problem in Communication Complexity.

• •

Disjointness: Given two equal-length bit-strings, is there a coordinate in which both are $1$?

The following two natural problems are at least as hard as Disjointness (see Section 5).

• •

Hamming Distance: Compute the Hamming Distance of two strings.

• •

Subsequence: Decide if a pattern of length $M$ is a subsequence of a text of length $N$.

Note that all these problems can be solved trivially in $O(N)$ time if our strings are uncompressed. Could it be that we cannot solve them without decompressing our data? We are not aware of any known algorithms solving any of these problems in $O(N^{1-\varepsilon})$ time, for any $\varepsilon>0$, even when our strings are compressed into size $n=O(N^{\alpha})$ for some small constant $\alpha>0$. The only exceptions are the known $\tilde{O}(M)$ time algorithms [28, 20, 69, 79, 71, 12] for the Subsequence problem, which beat the decompress-and-solve bound when the pattern is significantly smaller than the text. However, in the case $M=\Theta(N)$ no improvements seem to be known.

In Section 5 we present our attempts at proving a matching lower bound. We prove the following: $N^{1-o(1)}$ for Subsequence in the setting $N=\Theta(M)=\Theta(n^{2})=\Theta(m^{2})$ and $|\Sigma|=O(N^{\varepsilon})$ (Theorem 5.9). $N^{1/4-o(1)}$ for Disjointness (and thus also for the other two problems) in the setting $N=M$ and $n,m=O(N^{\varepsilon})$ for any $\varepsilon>0$, and constant alphabet size, assuming the $k$-SUM conjecture (Theorem 5.10). Similarly: $N^{1/3-o(1)}$ for Disjointness under Strong $k$-SUM conjecture (Theorem 5.11).

Motivated by our inability to prove tight lower bounds for these basic problems, despite seemingly having the right framework, we have turned our attention to upper bounds. In Section 5 we obtain the first improvement over the decompress-and-solve bound for Disjointness, Hamming Distance, and Subsequence. In particular, we obtain the first improvement over the decompress-and-solve bound for Disjointness, Hamming Distance, and Subsequence. Our algorithms solve all these problems in $O(n^{1.410}\cdot N^{0.593})$ time. As a side result, we also design a very simple algorithm for the Subsequence problem with $O((n|\Sigma|+M)\log N)$ runtime (Theorem 5.4), which is comparable to the known but more involved algorithms [12].

One of the biggest benefits of having complexity theoretic results is that algorithm designers know what to focus on. We believe that these upper bounds can be improved further and suggest it as an interesting open question: What is the time complexity of computing Disjointness on two grammar-compressed strings?

1.3 Technical Overview

From a technical perspective, our paper is most related to the conditional lower bounds for sequence similarity measures on strings and curves that have been shown in recent years, specifically, the SETH-based lower bounds for edit distance [10], longest common subsequence [2, 17], Fréchet distance [14], and others [4, 11, 16, 62].

These results all proceed as follows. Let $\phi$ be a given $k$-SAT instance on $\tilde{n}$ variables and clauses $C_{1},\ldots,C_{\tilde{m}}$. We can assume that $\tilde{m}=O(\tilde{n})$ by the Sparsification Lemma [43]. Split the $\tilde{n}$ variables into two halves $X_{1}$ and $X_{2}$ of size $\tilde{n}/2$. Enumerate all assignments $\alpha_{1},\ldots,\alpha_{2^{\tilde{n}/2}}$ of the variables in $X_{1}$. For any assignment $\alpha_{i}$ and any clause $C_{\ell}$, denote by $\textup{sat}(\alpha_{i},C_{\ell})$ whether $\alpha_{i}$ satisfies $C_{\ell}$, i.e., whether some variable in $X_{1}$ appears in $C_{\ell}$ (negated or unnegated) and is set by $\alpha_{i}$ so that $C_{\ell}$ is satisfied. Similarly, consider the assignments $\beta_{1},\ldots,\beta_{2^{\tilde{n}/2}}$ of $X_{2}$. By construction, we can solve the $k$-SAT instance $\phi$ by testing whether there are $\alpha_{i},\beta_{j}$ such that $\textup{sat}(\alpha_{i},C_{\ell})\vee\textup{sat}(\beta_{j},C_{\ell})$ holds for all $\ell\in[\tilde{m}]$. Making use of this fact, all previous conditional lower bounds for sequence similarity measures essentially construct the following natural sequence:

[TABLE]

One typical variation of this string is to replace the bits $\{0,1\}$, indicating whether $\textup{sat}(\alpha_{i},C_{\ell})$ holds, by two short strings $\{B(0),B(1)\}$. Other typical variations are to add appropriate padding strings around the substrings $\mathop{\bigcirc}_{\ell\in[\tilde{m}]}\textup{sat}(\alpha_{i},C_{\ell})$ or around the whole sequence $W$. These paddings typically only depend on $\tilde{n}$ and $\tilde{m}$. Constructing a second sequence $W^{\prime}$ with $\alpha_{i}$ replaced by $\beta_{i}$, one can then try to emulate the search for the half-assignments $\alpha_{i},\beta_{j}$ by a similarity measure on $W,W^{\prime}$. All previous reductions follow this recipe, and thus construct a sequence like $W$.

Is $W$ compressible?

For our purposes we need to construct compressible strings. Considering the entropy, the string $W$ is very well compressible, since it only depends on the $\tilde{{O}}(\tilde{n})$ input bits of the sparse $k$-SAT instance $\phi$. This entropy $\tilde{{O}}(\tilde{n})$ is extremely small compared to the length $O(\tilde{n}2^{\tilde{n}/2})$ of $W$. However, considering grammar-compression, the sequence $W$ is a bad representation, since $W$ is not generated by any SLP of size $o(2^{\tilde{n}/2}/\tilde{n})$ in general! To see this, first observe that all substrings $\mathop{\bigcirc}_{\ell\in[\tilde{m}]}\textup{sat}(\alpha_{i},C_{\ell})$ of $W$ can potentially be different, meaning that $W$ can have $2^{\tilde{n}/2}$ different substrings of length $\tilde{m}$. This happens e.g. if for each variable $x_{i}\in X$ there is a clause $C_{i}$ consisting only of $x_{i}$ (which makes the $k$-SAT instance trivial, but shows that $W$ may have many different substrings in general). Second, observe that for any SLP $\mathcal{T}$ consisting of $n$ non-terminals $S_{1}\ldots S_{n}$ and for any length $L\geq 1$ the generated string $\textup{eval}(\mathcal{T})$ has at most $n\cdot L$ different substrings of length $L$. Indeed, a rule $S_{i}\to S_{\ell}S_{r}$ can only create a new substring, that is not already contained in $\textup{eval}(S_{\ell})$ or $\textup{eval}(S_{r})$, if this substring overlaps the boundary between $\textup{eval}(S_{\ell})$ and $\textup{eval}(S_{r})$ in $\textup{eval}(S_{i})$. Hence, the rule $S_{i}\to S_{\ell}S_{r}$ can contribute at most $L$ new substrings of length $L$, amounting to at most $nL$ different substrings overall. Combining these two facts, with $L=\tilde{m}=O(\tilde{n})$, we see that $W$ in general has no SLP of size $o(2^{\tilde{n}/2}/\tilde{n})$.

Hence, the standard approach to conditional lower bounds for sequence similarity measures fails in the compressed setting, and it might seem like (SETH-based) conditional lower bounds are not applicable here.

A compressible sequence $T$

On the contrary, we show that by simply inverting the ordering we obtain a very well compressible string:

[TABLE]

The difference between $W$ and $T$ might seem negligible, but it greatly changes the game of emulating $k$-SAT by a sequence similarity measure: In $W$ we are looking for a local structure (a small substring) that “fits together” with a local structure in a different string $W^{\prime}$. In $T$ we have to ensure the choice of a consistent offset $\Delta\in[n]$ and “read” the symbols $T[\Delta],T[\Delta+2^{\tilde{n}/2}],\ldots,T[\Delta+(\tilde{m}-1)2^{\tilde{n}/2}]$, which seems much more complicated.

$T$ is compressible to an SLP $\mathcal{T}$ of size $O(\tilde{n}^{2})$, which is much smaller than the $\Omega(2^{\tilde{n}/2}/\tilde{n})$ bound for $W$. Indeed, consider a substring $\mathop{\bigcirc}_{i\in[2^{\tilde{n}/2}]}\textup{sat}(\alpha_{i},C_{\ell})$. We may assume that no variable appears more than once in $C_{\ell}$. Consider the following SLP rules, for $1\leq i\leq\tilde{n}/2$,

[TABLE]

[TABLE]

We clearly have $\textup{eval}(A_{i})=1^{2^{i}}$. Moreover, if $\neg x_{i}$ appears in $C_{\ell}$, then for $x_{i}=0$, no matter what we choose for $x_{1},\ldots,x_{i-1}$, we have $\textup{sat}(\alpha_{j},C_{\ell})=1$, and thus we may write $A_{i-1}$. For $x_{i}=1$ we note that the value $\textup{sat}(\alpha_{j},C_{\ell})$ only depends on the remaining variables $x_{1},\ldots,x_{i-1}$, and thus we may write $S_{i-1}$. Along these lines, one can check that $\textup{eval}(S_{\tilde{n}/2})=\mathop{\bigcirc}_{i\in[2^{\tilde{n}/2}]}\textup{sat}(\alpha_{i},C_{\ell})$. Creating such an SLP for each $\ell\in[\tilde{m}]$ and constructing their concatenation, we obtain an SLP of size $O(\tilde{m}\tilde{n})=O(\tilde{n}^{2})$ generating $T$.

Example Lower Bound: Pattern Matching with Wildcards

In the remainder of this section, we present an easy example for a conditional lower bound on compressed strings, namely for the problem Pattern Matching with Wildcards. Here we consider an alphabet $\Sigma$ and we say that symbols $\sigma,\sigma^{\prime}\in\Sigma\cup\{*\}$ match if $\sigma=*$ or $\sigma^{\prime}=*$ or $\sigma=\sigma^{\prime}$. We say that two equal-length strings $X,Y$ (over alphabet $\Sigma\cup\{*\}$) match if $X[i]$ and $Y[i]$ match for all $i$. Given a text $T$ of length $N$ and a pattern $P$ of length $M\leq N$, the task is to decide whether $P$ matches some length-$M$ substring of $T$.

Let $\phi$ be a $k$-SAT instance as above, but this time let $\alpha_{1},\ldots,\alpha_{2^{\tilde{n}}}$ be all the assignments of the $\tilde{n}$ variables in $\phi$. We define the text $T$ and pattern $P$ by

[TABLE]

Note that $P$ matches some substring of $T$ if and only if there is an offset $\Delta\in[2^{\tilde{n}}]$ such that $T[\Delta]=T[\Delta+2^{\tilde{n}}]=\ldots=T[\Delta+(\tilde{m}-1)2^{\tilde{n}}]=1$, which happens if and only if $\alpha_{\Delta}$ is a satisfying assignment of $\phi$. Hence, we constructed an equivalent instance of Pattern Matching with Wildcards.

Analogously to above, one can show that $T$ is generated by an SLP $\mathcal{T}$ of size $n=O(\tilde{n}^{2})$ that can be computed in time $O(\tilde{n}^{2})$. Similarly, it is easy to see that $P$ is generated by an SLP $\mathcal{P}$ of size $O(\tilde{n})$ that can be computed in time $O(\tilde{n})$. Hence, the reduction runs in time $O(\tilde{n}^{2})$. We stress that we define strings $T,P$ of exponential length in $\tilde{n}$, but in the reduction we never explicitly write down any such string, but we simply construct compressed representations. Since the resulting strings have length $O(2^{\tilde{n}}\tilde{n})$, any $O(N^{1-\varepsilon})$ time algorithm for Pattern Matching with Wildcards would imply an algorithm for $k$-SAT in time $O(2^{(1-\varepsilon)\tilde{n}}\textup{poly}(\tilde{n}))$, contradicting the Strong Exponential Time Hypothesis (SETH). Note that this conditional lower bound of $N^{1-o(1)}$ holds even for strings compressible to size $\textup{polylog}(N)$.

In Section 3.2 we analyze Pattern Matching with Wildcards in more detail and show that the optimal running time, conditional on SETH, is $\min\{N,nM\}^{1\pm o(1)}$, and this holds for all settings of the text length $N$, the compressed text size $n$, the pattern length $M$, and the compressed pattern size $m$.

In Pattern Matching with Wildcards, we got a consistent choice of an offset $\Delta$ for free. It is much more complicated to achieve this for other problems such as Longest Common Subsequence, CFG Parsing, or RNA Folding. This overview summarized the main technical contributions of this paper, but left out many problem-specific tricks that can be found in the subsequent proofs, and that we think will find more applications for analyzing problems on compressed strings.

2 Preliminaries

Here we give general preliminaries on strings, straight-line programs, and hardness assumptions. Problem definitions and additional problem specific preliminaries will be given in the corresponding sections. For a positive integer $n$ we let $[n]=\{1,\ldots,n\}$, while for a proposition $A$ we let $[A]$ be 1 if $A$ is true and 0 otherwise.

Strings

Let $\Sigma$ be a finite alphabet. In most parts of this paper we assume that $|\Sigma|=O(1)$, but in exceptional cases we allow the alphabet to grow with the input size. For a string $T$ over alphabet $\Sigma$, we write $|T|$ for its length, $T[i]$ for its $i$-th symbol, and $T[i..j]$ for the substring from position $i$ to position $j$. For two strings $T,T^{\prime}$ we write $T\mathop{\circ}T^{\prime}$, or simply $T\,T^{\prime}$, for their concatenation. For $k\geq 1$ we let $T^{k}:=\mathop{\bigcirc}_{i=1}^{k}T$.

Straight-Line Programs (SLPs)

An SLP $\mathcal{T}$ is a set of non-terminals $S_{1},\ldots,S_{n}$, each equipped with a rule of the form (1) $S_{i}\to\sigma$ for some $\sigma\in\Sigma$ or (2) $S_{i}\to S_{\ell(i)},S_{r(i)}$ with $\ell(i),r(i)<i$. The string $T$ generated by SLP $\mathcal{T}$ is recursively defined as follows. For a rule $S_{i}\to\sigma$ we let $\textup{eval}(S_{i}):=\sigma$, and for a rule $S_{i}\to S_{\ell(i)},S_{r(i)}$ we let $\textup{eval}(S_{i}):=\textup{eval}(S_{\ell(i)})\mathop{\circ}\textup{eval}(S_{r(i)})$. Then $T=\textup{eval}(\mathcal{T}):=\textup{eval}(S_{n})$ is the string generated by SLP $\mathcal{T}$. Note that an SLP is a context-free grammar describing a unique string; so $\mathcal{T}$ is a grammar-compressed representation of $T$. We call $|\mathcal{T}|=n$ the size of $\mathcal{T}$. See Figure 1 for the depiction of an SLP; in particular note the difference between the directed acyclic graph that is the compressed representation $\mathcal{T}$ and the parse tree that we obtain by decompressing $\mathcal{T}$ to a tree whose leaves spell the decompressed text $T$.

For an SLP $\mathcal{T}$ with non-terminals $S_{1},\ldots,S_{n}$, we recursively define the depth $0pt(S_{i})$ as follows. For a rule $S_{i}\to\sigma$ we set $0pt(S_{i}):=0$. For a rule $S_{i}\to S_{\ell(i)},S_{r(i)}$ we set $0pt(S_{i})=\max\{0pt(S_{\ell(i)}),0pt(S_{r(i)})\}+1$. The depth of $\mathcal{T}$ is defined as $0pt(S_{n})$. The SLP $\mathcal{T}$ is called an AVL-grammar [64] if it is balanced: for any rule $S_{i}\to S_{\ell(i)},S_{r(i)}$ in the SLP we have $|0pt(S_{\ell(i)})-0pt(S_{r(i)})|\leq 1$. This implies that the depth of $\mathcal{T}$ is $O(\log N)$, where $N=|\textup{eval}(\mathcal{T})|$.

Theorem 2.1** ([64]).**

Given a text $T$ of length $N$ by an SLP $\mathcal{T}$ of size $n$, in $O(n\log N)$ time we can construct an AVL-grammar $\mathcal{T}^{\prime}$ for $T$ with size $O(n\log N)$ and depth $O(\log N)$.

Observation 2.2**.**

For any string $T$ and $k\geq 1$, there is an SLP of size $O(|T|+\log k)$ generating the string $T^{k}$.

In all problems considered in this paper, the input contains a text $T$ given by a grammar-compressed representation $\mathcal{T}$, such that $T=\textup{eval}(\mathcal{T})$. We always denote by $N=|T|$ the length of the text and by $n=|\mathcal{T}|$ the size of its representation. Sometimes we are additionally given a pattern $P$ by a grammar-compressed representation $\mathcal{P}$, and we denote the pattern length by $M=|P|$ and its representation size by $m=|\mathcal{P}|$.

2.1 Hardness Assumptions

SETH and OV

The Strong Exponential Time Hypothesis (SETH) was introduced by Impagliazzo, Paturi, and Zane [43] and asserts that the central NP-hard satisfiability problem has no algorithms that are much faster than exhaustive search.

Conjecture 2.3** (SETH).**

There is no $\varepsilon>0$ such that for all $k\geq 3$, $k$-SAT on $n$ variables can be solved in time $O(2^{(1-\varepsilon)n})$.

Effectively all known SETH-based lower bounds for polynomial-time problems use reductions via the Orthogonal Vectors problem (OV): Given sets ${\cal A}$, ${\cal B}\subseteq\{0,1\}^{d}$ of size $|{\cal A}|=A$, $|{\cal B}|=B$, determine whether there exist vectors $a\in{\cal A}$, $b\in{\cal B}$ with $\sum_{i=1}^{d}a[i]\cdot b[i]=0$. Simple algorithms solve OV in time ${O}(2^{d}(A+B))$ and ${O}(dAB)$. For $A=B$ and $d=c(A)\log A$ the fastest known algorithm runs in time $A^{2-1/{O}(\log c(A))}$ [5], which is only slightly subquadratic for $d\gg\log A$. This has led to the following conjecture, which follows from SETH [76].

Conjecture 2.4** (OV).**

For any $\varepsilon>0$ and $\beta>0$, on instances with $B=\Theta(A^{\beta})$ OV has no $O(A^{1+\beta-\varepsilon}\textup{poly}(d))$ time algorithm.

It is known that if this conjecture holds for some $\beta>0$ then it holds for all $\beta>0$, see e.g. [17].

More generally, for $k\geq 2$ we say that a tuple $(a_{1},\ldots,a_{k})$ with $a_{i}\in\{0,1\}^{d}$ is orthogonal if for all $\ell\in[d]$ there exists an $i\in[k]$ such that $a_{i}[\ell]=0$. In the $k$-OV problem we are given a set ${\cal A}\subseteq\{0,1\}^{d}$ of size $A$ and want to determine whether there is an orthogonal tuple $(a_{1},\ldots,a_{k})$ with $a_{i}\in{\cal A}$. The fastest known algorithm for $k$-OV is to run an easy reduction to OV and then solve OV. The following conjecture follows from SETH.

Conjecture 2.5** ($k$-OV).**

For any $\varepsilon>0$ and $k\geq 2$, $k$-OV is not in time $O(A^{k-\varepsilon}\textup{poly}(d))$.

$k$-Clique

The fundamental $k$-Clique problem asks whether a given (undirected, unweighted) graph $G=(V,E)$ contains $k$ nodes that are pairwise adjacent. $k$-Clique is among the most well-studied problems in theoretical computer science, and it is the canonical intractable (W[1]-complete) problem in parameterized complexity. With slight abuse of notation, we will denote the number of vertices and edges of $G$ by $V$ and $E$, respectively. The naive algorithm for $k$-Clique takes time $O(V^{k})$. If $k$ is divisible by 3, the fastest known algorithm runs in time $O(V^{\omega k/3})$, where $\omega<2.373$ is the exponent of matrix multiplication [58]. See [31] for the case that $k$ is not divisible by 3. To improve this bound is a longstanding open problem [78, 56]. Since fast matrix multiplication is considered impractical, researchers also studied combinatorial algorithms, that avoid fast matrix multiplication555Combinatorial algorithms are a notion without agreed upon definition; finding a formal definition is considered an open problem.. The fastest combinatorial algorithm runs in time $O(V^{k}/\log^{k}{V})$ [73]. The following conjectures assert that these bounds are close to optimal, and have been used e.g. in [1, 16].

Conjecture 2.6** ($k$-Clique).**

For any $\varepsilon>0$ and $k\geq 3$, $k$-Clique has no $O(V^{(1-\varepsilon)\omega k/3})$ algorithm.

Conjecture 2.7** (Combinatorial $k$-Clique).**

For any $\varepsilon>0$ and $k\geq 3$, $k$-Clique has no combinatorial $O(V^{(1-\varepsilon)k})$ algorithm.

$k$-SUM

In the $k$-SUM problem, we are given integers $R,t\geq 0$ and a set $Z\subseteq\{0,1,\ldots,R\}$ of $|Z|=r$ integers, and the task is to decide whether there are $k$ (not necessarily distinct) integers $z_{1},\ldots,z_{k}\in Z$ that sum to $t$, i.e., $z_{1}+\ldots+z_{k}=t$. This problem has well-known algorithms in time $O(r^{\lceil k/2\rceil})$ and $O(r+R\log R)$, and it is conjectured that no much faster algorithms exist. The following conjectures, which generalize the more popular 3-SUM conjecture [33, 59] and Strong 3-SUM conjecture [6], remain believable despite recent algorithmic progress [9, 21, 37, 74].

Conjecture 2.8** ($k$-SUM).**

For any $k\geq 3$ and $R=r^{k}$, the $k$-SUM problem is not in time $O(r^{\lceil k/2\rceil-\varepsilon})$ for any $\varepsilon>0$.

Conjecture 2.9** (Strong $k$-SUM).**

For any $k\geq 3$ and $R=r^{\lceil k/2\rceil}$, the $k$-SUM problem is not in time $O(r^{\lceil k/2\rceil-\varepsilon})$ for any $\varepsilon>0$.

3 Tight Bounds Assuming SETH

In this section we prove matching conditional lower bounds based on the Strong Exponential Time Hypothesis (SETH, see Conjecture 2.3) for the following problems:

• •

DFA Acceptance, i.e., deciding whether a given deterministic finite automaton accepts a given string,

• •

Substring Hamming Distance, i.e., determining the minimum Hamming distance that can be achieved by aligning a given pattern sequence with a substring of a given text sequence,

• •

Pattern Matching with Wildcards, i.e., deciding whether the given pattern sequence (containing wildcards that match any symbol) matches a substring of the given text,

• •

Longest Common Subsequence, i.e., computing the length of the longest common subsequence of two given strings.

See the respective subsections for precise problem definitions. In all our proofs, instead of using SETH directly, we use the more convenient OV conjecture (Conjecture 2.4) or $k$-OV conjecture (Conjecture 2.5), which are implied by SETH.

For DFA Acceptance, the compression used in our reduction from the given OV instance is extremely simple, in that we only rely on the fact that any repetition $T^{\ell}$ can be generated by an SLP of size $O(|T|+\log\ell)$ (Observation 2.2).

For Substring Hamming Distance, Pattern Matching with Wildcards and Longest Common Subsequence, however, our construction are more subtle. We crucially use the following idea: consider a $k$-OV instance ${\cal A}$ on $A$ vectors in $d$ dimensions. There is a length-${O}(dA^{k})$ text $T$ representing this instance so that (1) $T$ is succinctly described by an SLP $\mathcal{T}$ of size ${O}(dA)$ and (2) testing whether the $k$-OV instance has a solution corresponds to determining whether there is some $i=1,\ldots,A^{k}$ such that all bits $T[i],T[i+A^{k}],\dots,T[i+(d-1)A^{k}]$ are equal to zero. Intuitively, $i\in\{1,\ldots,A^{k}\}$ denotes the $i$-th $k$-tuple of vectors in ${\cal A}^{k}$, and $T[i]=0$ holds if and only if the $k$ vectors in the $i$-th $k$-tuple are orthogonal in the $1$st coordinate. In general, for $1\leq\ell\leq d$, $T[i+(\ell-1)A^{k}]=0$ holds if and only if the $k$ vectors are orthogonal in the $\ell$-th coordinate. More formally, we set $T$ to be

[TABLE]

where $[.]$ is the Kronecker symbol, i.e., $[\textup{true}]=1$ and $[\textup{false}]=0$. For any $\ell$, the sequence $\mathop{\bigcirc}_{a_{1}\in{\cal A}^{(1)}}\ldots\mathop{\bigcirc}_{a_{k}\in{\cal A}^{(k)}}[a_{1}[\ell]=\ldots=a_{k}[\ell]=1]$ is generated by an SLP of size $O(dA)$: if $a_{1}[\ell]=0$, then for all $a_{2},\ldots,a_{k}$, the vectors will be orthogonal in this coordinate and we can write $0^{A^{k-1}}$, which is well compressible by Observation 2.2. Otherwise, if $a_{1}[\ell]=1$, we recurse on $a_{2},\ldots,a_{k}$ and the following $A^{k-1}$ symbols do not depend on $a_{1}[\ell]$ anymore.

A modification of the above construction of $T$ gives SETH hardness for Substring Hamming Distance and Pattern Matching with Wildcards. Showing hardness for Longest Common Subsequence requires more ideas. In particular, to be able to show tight hardness we extend the framework of [17].

We stress that if the sequence $T$ would enumerate all $k$-tuples one after another (instead of iterating over the coordinates in the outer loop over $\ell$), then it would not be compressible using SLPs, see Section 1.3. This makes our reductions quite different from all previously known hardness results where the sequences are concatenations of vector gadgets one after another.

Known Lower Bounds from Classic Complexity Theory

We observe that the Substring Hamming Distance problem is a generalization of the Hamming Distance problem which asks to output the Hamming distance between a compressed text and a compressed pattern of equal length. The latter problem is known to be $\#{\sf P}$-complete and thus the Substring Hamming Distance problem is $\#{\sf P}$-hard (see the discussion at the beginning of Section 5.2). Similarly, Longest Common Subsequence is a generalization of the problem of deciding whether a given pattern is a subsequence of a given text. The latter problem is known to be ${\sf PP}$-hard (see the aforementioned discussion) and this yields ${\sf PP}$-hardness for Longest Common Subsequence.

The DFA Acceptance problem can be solved in polynomial time (see Section 3.1) and no conditional lower bounds were known for this problem. Finally, our reduction in Theorem 3.9 below shows that Pattern Matching with Wildcards is ${\sf NP}$-hard.

3.1 DFA Acceptance

Recall that a finite-state automaton $F$ over an alphabet $\Sigma$ consists of a set of states $Z$ of size $q$, a starting state $z_{0}\in Z$, a set of accepting states $Z^{\prime}\subseteq Z$, and a set of transitions $z\stackrel{{\scriptstyle\sigma}}{{\to}}z^{\prime}$ with $z,z^{\prime}\in Z$ and $\sigma\in\Sigma$. We lift this notation to strings $T=T[1..N]$ by writing $z\stackrel{{\scriptstyle T}}{{\to}}z^{\prime}$ whenever there are states $z_{1},\dots,z_{\ell-1}$ and transitions $z\stackrel{{\scriptstyle T[1]}}{{\to}}z_{1},z_{1}\stackrel{{\scriptstyle T[2]}}{{\to}}z_{2},\dots,z_{\ell-1}\stackrel{{\scriptstyle T[N]}}{{\to}}z^{\prime}$. Furthermore, for a set $S\subseteq\Sigma$ we write $z\stackrel{{\scriptstyle S}}{{\to}}z^{\prime}$ whenever $z\stackrel{{\scriptstyle\sigma}}{{\to}}z^{\prime}$ for all $\sigma\in S$. The automaton $F$ is deterministic if for any $z\in Z$ and $\sigma\in\Sigma$ there is at most one $z^{\prime}\in Z$ with transition $z\stackrel{{\scriptstyle\sigma}}{{\to}}z^{\prime}$, and $F$ is non-determinisitic otherwise. The automaton $F$ accepts a given string $T$ if $z_{0}\stackrel{{\scriptstyle T}}{{\to}}z^{\prime}$ holds for some accepting state $z^{\prime}$.

Throughout this section, we assume the alphabet size to be constant. If $F$ is a deterministic finite-state automaton (DFA), we may assume without loss of generality that for every state $z$ and symbol $\sigma\in\Sigma$, there always exists a (uniquely defined) state $z^{\prime}$ with $z\stackrel{{\scriptstyle\sigma}}{{\to}}z^{\prime}$.666Note that we can always define an absorbing non-accepting state $z^{\mathrm{fail}}$ with $z^{\mathrm{fail}}\stackrel{{\scriptstyle\Sigma}}{{\to}}z^{\mathrm{fail}}$ and set, for any undefined transition from $z$ under $\sigma$, $z\stackrel{{\scriptstyle\sigma}}{{\to}}z^{\mathrm{fail}}$, which increases the number of states only by one. We fix the input description of $F$ to a list of transitions of $F$ as well as a list of accepting states. Observe that any DFA $F$ on constant alphabet $\Sigma$ has an input size of ${O}(q)$.

Consider the compressed variant of the acceptance problem of DFAs.

Problem 3.1** (DFA Acceptance).**

Given a text $T$ of length $N$ by a grammar-compressed representation $\mathcal{T}$ of size $n$ as well as a DFA $F$ with $q$ states, decide whether $T$ is accepted by $F$.

The naive solution decompresses $\mathcal{T}$ to obtain $T$ and runs the obvious acceptance algorithm for DFAs, which takes time ${O}(|T|+q)={O}(N+q)$. Exploiting the compressed setting, one can obtain an ${O}(nq)$-time algorithm [61]: Recall that $\mathcal{T}$ is a set of rules of the form $S_{i}\to S_{\ell(i)}S_{r(i)}$ or $S_{i}\to\sigma_{i}$, with $\ell(i),r(i)<i$ and $\sigma_{i}\in\Sigma$, for $1\leq i\leq n$. We compute, for increasing $i$, the state transition function $f_{i}\colon[q]\to[q]$ (we denote states using integers $1,\ldots,q$) that satisfies $z\stackrel{{\scriptstyle\textup{eval}(S_{i})}}{{\to}}f_{i}(z)$. For $S_{i}\to S_{\ell(i)}S_{r(i)}$ we can compute $f_{i}$ as $f_{r(i)}\circ f_{\ell(i)}$, where $\circ$ is function composition. For $S_{i}\to\sigma_{i}$ we simply have $f_{i}(z)=z^{\prime}$ for the unique transition $z\stackrel{{\scriptstyle\sigma_{i}}}{{\to}}z^{\prime}$. Hence, $f_{i}$ can be computed in time ${O}(q)$ for every $i$. The text $T$ is then accepted by $F$ if and only if $f_{n}(z_{0})$ is an accepting state, where $z_{0}$ is the starting state of $F$. Hence, the best-known algorithm takes time $O(\min\{nq,N+q\})$.

We prove that DFA Acceptance takes time $\min\{nq,N+q\}^{1-o(1)}$ assuming SETH, thus providing a conditional lower bound matching the known algorithmic results. It is straightforward to see that any algorithm must read the complete input description of $F$ to always correctly decide the problem, yielding a lower bound of $\Omega(q)$. In the remainder, we provide the remaining conditional lower bound of $\min\{nq,N\}^{1-o(1)}$.

Theorem 3.2**.**

Assuming the OV conjecture, for no $\varepsilon>0$ there is an ${O}(\min\{nq,N\}^{1-\varepsilon})$-time algorithm for DFA Acceptance. This holds even restricted to instances with $N=\Theta(n^{\alpha_{N}})$ and $q=\Theta(n^{\alpha_{q}})$ for any $\alpha_{N}>1$ and $\alpha_{q}>0$.

Proof.

Let ${\cal A}=\{a_{1},\dots,a_{A}\},{\cal B}=\{b_{1},\dots,b_{B}\}$ be a given OV instance in $d$ dimensions. We construct a string $T$ of length $N=O(dAB)$ with a representation $\mathcal{T}$ of size $n=O(dA)$ and a DFA $F$ with $q=O(dB)$. An $O(\min\{nq,N\}^{1-\varepsilon})$-time algorithm for DFA Acceptance would then imply an algorithm for OV in time $O((d^{2}AB)^{1-\varepsilon})=O((AB)^{1-\varepsilon}\textup{poly}(d))$, contradicting the OV conjecture. At the end of this proof we show that this also holds for all restrictions $N=\Theta(n^{\alpha_{N}})$ and $q=\Theta(n^{\alpha_{q}})$ with $\alpha_{N}>1$ and $\alpha_{q}>0$.

Constructing the Text $T$

We cast any vector $a\in\{0,1\}^{d}$ to a string $T(a):=\mathop{\bigcirc}_{k=1}^{d}a[k]$ by simply concatenating its coordinates. We define the text $T$ over the alphabet $\Sigma=\{0,1,\#,!\}$ as

[TABLE]

Here, we think of $!$ and $\#$ as “new group” and “new vector within group” indicators, respectively. Intuitively, the $j$-th repetition of $T(a_{i})$ is supposed to lead to an accepting state of $F$ if $a_{i}$ and $b_{j}$ are an orthogonal pair.

Constructing the DFA $F$

For an illustration of the DFA construction see Figure 2. We start by defining “vector gadgets”: For any vector $b_{j}\in{\cal B}$ we construct a DFA $F_{j}$ over alphabet $\{0,1\}$ with states $z^{(0)}_{j},z_{j}^{(1)},\dots,z_{j}^{(d)}$ and $z_{j}^{\mathrm{fail}}$. The initial state is $z^{(0)}_{j}$. For any $k\in[d]$ we have the transitions $z^{(k-1)}_{j}\stackrel{{\scriptstyle 0}}{{\to}}z^{(k)}_{j}$ and

[TABLE]

We let $z^{(d)}_{j}$ be an accepting state with transition $z^{(d)}_{j}\stackrel{{\scriptstyle\{0,1\}}}{{\to}}z^{(d)}_{j}$. Furthermore, we have $z^{\mathrm{fail}}_{j}\stackrel{{\scriptstyle\{0,1\}}}{{\to}}z^{\mathrm{fail}}_{j}$. It is easy to see that after reading a string $a\in\{0,1\}^{d}$, $F_{j}$ is in either $z^{(d)}_{j}$ or $z^{\mathrm{fail}}_{j}$, and it is in $z^{(d)}_{j}$ if and only if $a$ and $b_{j}$ are orthogonal.

We combine these smaller DFAs to our final DFA $F$ over the slightly larger alphabet $\Sigma=\{0,1,\#,!\}$ as follows. We define additional states $g_{0},g_{1},\dots,g_{B}$ and let $g_{0}$ be the initial state of $F$. We define the following additional transitions:

[TABLE]

In this way, each $z_{j}^{(d)}$ is an absorbing accepting state, and the symbols $\#$ and $!$ satisfy the semantics of jumping to the next vector in ${\cal A}$ and ${\cal B}$, respectively. This finishes the definition of the reduction.

Correctness

We claim that the constructed DFA $F$ accepts $T$ if and only if ${\cal A},{\cal B}$ contains an orthogonal pair. By structure of $F$ and $T$, as well as the properties argued for $F_{j},j\in[B]$, it is straightforward to show that after reading any prefix $T^{\prime}$ of $T$ ending on $\#$, $F$ is in the initial state of $F_{j}$, where $j$ is the number of !’s in $T^{\prime}$ – this holds until $F$ has encountered an accepting state for the first and final time. Thus, if $T$ is accepted by $F$, then some prefix $T^{\prime}$ of $T$ that ends on $\#\,a_{i}$ for some $i\in[A]$ has led an accepting state of $F$. This can only happen if $a_{i}$ is orthogonal to $b_{j}$, where $j$ is the number of !’s in $T^{\prime}$, i.e., ${\cal A},{\cal B}$ contain an orthogonal pair. Conversely, if ${\cal A},{\cal B}$ contains an orthogonal pair, let $a_{i},b_{j}$ the smallest such pair in terms of the lexicographic order on $(j,i)$. Then the prefix $T^{\prime}$ that ends on $\#a_{i}$ and contains $j$ !’s leads to the accepting state $z^{(d)}_{j}$.

Size Bounds

We count that $|T|=B((d+1)A+1)=O(dAB)$. Since $T$ consists of $B$ repetitions of a string of length $(d+1)A+1$, we can compute an SLP $\mathcal{T}$ of size $|\mathcal{T}|\leq{O}(\log(B)+dA)={O}(dA)$ by Observation 2.2. The number of states of $F$ is $O(dB)$. This satisfies the claimed size bounds. Note that the reduction can be implemented in linear time in the output size.

Strengthening the Statement

In the remainder, we verify that our construction proves the desired lower bound even restricted to instances with $N=\Theta(n^{\alpha_{N}})$ and $q=\Theta(n^{\alpha_{q}})$ for any $\alpha_{N}>1$ and $\alpha_{q}>0$. Note that the number of states, the size of the SLP, and the text length can all three be increased by easy padding. E.g., to increase the text length we introduce a garbage symbol “$\natural$” that can be read at any state of the automaton, not changing the current state, and add a suitable number of copies of “$\natural$” to the text.

We now set $\beta:=\min\{\alpha_{q},\alpha_{N}-1\}$ and only consider OV instances with $B=\Theta(A^{\beta})$. Note that the OV conjecture asserts a lower bound of $A^{1+\beta-o(1)}$ in this setting. Note that the above construction yields $n=O(dA)=O(d^{\max\{1,1/\beta\}}A)$, and we can pad to equality. Moreover, we have $q=O(dB)=O(d^{\max\{\beta,1\}}B)=O(d^{\max\{\beta,1\}}A^{\beta})=O(n^{\beta})=O(n^{\alpha_{q}})$, since $\beta\leq\alpha_{q}$, and we can pad to equality to obtain $q=\Theta(n^{\alpha_{q}})$. Similarly, we have $N=O(dAB)=O(d^{1+\max\{\beta,1\}}A^{1+\beta})=O(n^{1+\beta})=O(n^{\alpha_{N}})$, since $\beta\leq\alpha_{N}-1$, and we can pad to equality to obtain $N=\Theta(n^{\alpha_{N}})$. Finally, an $O(\min\{nq,N\}^{1-\varepsilon})$-time algorithm for DFA Acceptance restricted to $N=\Theta(n^{\alpha_{N}})$ and $q=\Theta(n^{\alpha_{q}})$ would imply an algorithm for OV in time $O(\min\{n^{1+\alpha_{q}},n^{\alpha_{N}}\}^{1-\varepsilon})=O(n^{(1+\beta)(1-\varepsilon)})=O((d^{\max\{1,1/\beta\}}A)^{(1+\beta)(1-\varepsilon)})=O(d^{\max\{1+\beta,1+1/\beta\}}A^{(1+\beta)(1-\varepsilon)})=O(A^{1+\beta-\varepsilon}\textup{poly}(d))$, contradicting the OV conjecture. This finishes the proof. ∎

3.2 Approximate Pattern Matching and Substring Hamming Distance

We study the following generalization of pattern matching.

Problem 3.3** (Generalized Pattern Matching).**

Given a text $T$ of length $N$ by an SLP $\mathcal{T}$ of size $n$, a pattern $P$ of length $M$ by an SLP $\mathcal{P}$ of size $m$, both over some alphabet $\Sigma$, and given a cost function $\mathrm{cost}\colon\Sigma\times\Sigma\to\mathbb{N}$, compute $\min_{0\leq i\leq N-M}\sum_{j=1}^{M}\mathrm{cost}(P[j],T[i+j])$, i.e., the minimum total cost of any alignment.

In other words, we want to compute the length-$M$ substring $T^{\prime}$ of $T$ minimizing the total cost of aligned symbols in $P$ and $T^{\prime}$. This problem has two important special cases: (1) We obtain Substring Hamming Distance when $\mathrm{cost}(\sigma,\sigma^{\prime})=[\sigma\neq\sigma^{\prime}]$ for any $\sigma,\sigma^{\prime}\in\Sigma$. (2) We obtain Pattern Matching with Wildcards when $T$ is over alphabet $\Sigma$ and $P$ is over alphabet $\Sigma\cup\{*\}$, we have $\mathrm{cost}(*,\sigma)=0$ for any $\sigma\in\Sigma$ and $\mathrm{cost}(\sigma,\sigma^{\prime})=[\sigma\neq\sigma^{\prime}]$ for any $\sigma,\sigma^{\prime}\in\Sigma$, and the task is to decide whether the minimum total cost of any alignment is 0.

Problem 3.4** (Substring Hamming Distance).**

Given a text $T$ of length $N$ by an SLP $\mathcal{T}$ of size $n$ and a pattern $P$ of length $M$ by an SLP $\mathcal{P}$ of size $m$, both over some alphabet $\Sigma$, compute \min_{0\leq i\leq N-M}\sum_{j=1}^{M}\big{[}P[j]\neq T[i+j]\big{]}, i.e., the minimum Hamming distance of any alignment.

Problem 3.5** (Pattern Matching with Wildcards).**

For some alphabet $\Sigma$, we are given a text $T$ of length $N$ by an SLP $\mathcal{T}$ of size $n$ over alphabet $\Sigma$ and a pattern $P$ of length $M$ by an SLP $\mathcal{P}$ of size $m$ over alphabet $\Sigma\cup\{*\}$. We say that $\sigma^{\prime}\in\Sigma\cup\{*\}$ and $\sigma\in\Sigma$ match if $\sigma^{\prime}=*$ or $\sigma^{\prime}=\sigma$. Decide whether for some offset $0\leq i\leq N-M$ all pairs $P[j],T[i+j]$ match for $1\leq j\leq M$.

In this section, for all three problems we show an upper bound of $O(\min\{|\Sigma|N\log N,nM\})$ and a SETH-based lower bound of $\min\{N,nM\}^{1-o(1)}$. This yields a tight bound in case of constant alphabet size, as the lower bound constructs constant-alphabet strings. We leave it as an open problem to get tight bounds for larger alphabet size.

Note that it suffices to prove the upper bound for Generalized Pattern Matching and the lower bound for the special cases Substring Hamming Distance and Pattern Matching with Wildcards. We start with the following two upper bounds, which follow standard arguments.

Lemma 3.6**.**

Generalized Pattern Matching can be solved in time $O(|\Sigma|N\log N)$.

Proof.

Decompress both the text $T$ and the pattern $P$. For each symbol $\sigma\in\Sigma$, build the vector $v^{\sigma}\in\mathbb{R}^{N}$ with $v_{i}^{\sigma}:=\mathrm{cost}(\sigma,T[i])$ and the vector $u^{\sigma}\in\{0,1\}^{M}$ with $u_{j}^{\sigma}:=[P[j]=\sigma]$. Compute their convolution $w^{\sigma}\in\mathbb{R}^{N-M+1}$ with $w_{i}^{\sigma}=\sum_{j=1}^{M}u_{j}^{\sigma}v_{i+j}^{\sigma}$. Using FFT, $w^{\sigma}$ can be computed in time $O(N\log N)$. Finally, compute the vector $r\in\mathbb{R}^{N-M+1}$ with $r_{i}=\sum_{\sigma\in\Sigma}w_{i}^{\sigma}$ and return the minimal entry of $r$. Note that

[TABLE]

which proves correctness. ∎

Lemma 3.7**.**

Generalized Pattern Matching can be solved in time $O(nM)$.

Proof Sketch.

Let $S_{1},\ldots,S_{n}$ be the non-terminals of the SLP $\mathcal{T}$ that generates the text $T$. In this proof, for simplicity we write $T_{i}:=\textup{eval}(S_{i})$. We decompress the pattern $P$. For $1\leq i\leq n$ we define

[TABLE]

or $\infty$, if $|T_{i}|<M$. This solves the Generalized Pattern Matching problem restricted to the substring $T_{i}$ of $T$. Clearly, we can solve the given Generalized Pattern Matching instance $(T,P)$ by calling $\textup{Match}(n)$. Moreover, for any offset $d$ and any $i\in[n]$ we define

[TABLE]

In other words, $\textup{FixMatch}(i,d)$ is equal to the total cost between $T_{i}$ and a shifted pattern $P$ (by $d$ symbols to the right, or $-d$ symbols to the left), where we consider only the symbols that have an aligned counterpart.

In the remainder we show how to compute these functions by simple recursive algorithms. We precompute all lengths $|T_{i}|$ in time $O(n)$. For $\textup{FixMatch}(.,.)$, observe that for a rule $S_{i}\to S_{\ell}S_{r}$ we have

[TABLE]

since the offset with respect to the first symbol of $T_{r}$ differs to the offset with respect to the first symbol of $T_{i}$ by $|T_{\ell}|$. Moreover, for a rule $S_{i}\to\sigma\in\Sigma$ we can compute $\textup{FixMatch}(i,d)$ in constant time. Note that whenever the offset $d$ is such that no symbols get aligned, we can immediately return [math]. This completes our algorithm for $\textup{FixMatch}(.,.)$.

Now consider $\textup{Match}(i)$. For a rule $S_{i}\to S_{\ell}S_{r}$, the optimal alignment of the pattern in $T_{i}$ is either completely contained in $T_{\ell}$ or completely contained in $T_{r}$ or it has a non-empty intersection with both of them, in which case it has an offset $-M<d<0$ with respect to the starting symbol of $T_{r}$, or equivalently an offset $|T_{\ell}|+d$ with respect to the starting symbol of $T_{\ell}$. Hence, we have

[TABLE]

Again, for a rule $S_{i}\to\sigma\in\Sigma$ we can compute $\textup{Match}(i)$ in constant time. This completes the algorithm for $\textup{Match}(.)$.

To obtain the claimed running time, we use memoization to ensure that each argument is called at most once. Clearly, there are $n$ possible arguments for $\textup{Match}(.)$, and each call takes time $O(M)$, resulting in time $O(nM)$. Note that $\textup{Match}(.)$ only calls $\textup{FixMatch}(i,d)$ for offsets $d$ such that the pattern crosses the left or right boundary of $T_{i}$. This property also holds as an invariant in the recursive subproblems of $\textup{FixMatch}(i,d)$. Hence, there are less than $2M$ possible offsets $d$ (i.e., less than $M$ offsets for the left and right boundary). As there are $n$ possible values for $i$, and each call to $\textup{FixMatch}(.,.)$ takes time $O(1)$, we obtain the claimed total running time of $O(nM)$. ∎

This completes the upper bound $O(\min\{|\Sigma|N\log N,nM\})$ for Generalized Pattern Matching. It remains to prove the SETH-based lower bound of $\min\{N,nM\}^{1-o(1)}$ for Substring Hamming Distance and Pattern Matching with Wildcards.

We now make the intuition given at the beginning of Section 3 formal, by designing a text $T$ that enumerates all combinations of $k$ vectors in a given $k$-OV instance, while still being well compressible. We give a slightly more general construction that will also be useful later for our SETH-based lower bounds for LCS, see Section 3.3. As usual, we consider $k$ as a constant.

Lemma 3.8**.**

Consider a $k$-OV instance ${\cal A}=\{a_{1},\ldots,a_{A}\}\subseteq\{0,1\}^{d}$. Let $b\in\{0,1\}^{d}$ be an additional vector, and let $S(0),S(1)$ be strings of length $\gamma$ ($S(i)$ is a sequence that represents an entry that is equal to $i$). We define the tuplified representation as follows:

[TABLE]

where the second $\mathop{\bigcirc}$ goes over all tuples $(i_{1},\ldots,i_{k})\in[A]^{k}$ in lexicographic order. This representation satisfies the following properties.

We can compute, in linear time in the output size, an SLP $\mathcal{V}$ generating $V$ of size ${O}(dA+\gamma)$ or, when given SLPs ${{\mathcal{S}}(0)},{{\mathcal{S}}(1)}$ generating $S(0),S(1)$, of size ${O}(dA+|{{\mathcal{S}}(0)}|+|{{\mathcal{S}}(1)}|)$. 2. 2.

Write $V=\mathop{\bigcirc}_{i=1}^{dA^{k}}V_{i}$ with $V_{i}\in\{S(0),S(1)\}$. Then there exist $i_{1},\ldots,i_{k}\in[A]$ such that $(b,a_{i_{1}},\dots,a_{i_{k}})$ is orthogonal if and only if there is an offset $1\leq\Delta\leq A^{k}$ such that

[TABLE]

Proof.

For the second property, note that by definition $V_{\Delta},V_{\Delta+A^{k}},\dots,V_{\Delta+(d-1)A^{k}}$ are all equal to $S(0)$ for $\Delta\in[A^{k}]$ if and only if the $\Delta$-th tuple $(i_{1},\dots,i_{k})\in[A]^{k}$ in the lexicographic ordering of $[A]^{k}$ satisfies

[TABLE]

This condition is equivalent to $(b,a_{i_{1}},\dots,a_{i_{k}})$ being an orthogonal pair, so the claim follows.

It remains to construct a short SLP $\mathcal{V}$ generating $V$. We construct non-terminals $P_{S(0)},P_{S(1)}$ with $\textup{eval}(P_{S(i)})=S(i)$ by an SLP of size $\gamma_{S}=O(\gamma)$ as in Observation 2.2, or of size $\gamma_{S}=O(|{{\mathcal{S}}(0)}|+|{{\mathcal{S}}(1)}|)$ by using given SLPs ${{\mathcal{S}}(0)},{{\mathcal{S}}(1)}$. We can extend this, using Observation 2.2, to a slightly larger SLP of size ${O}(\log A+\gamma_{S})$ that includes, for every $1\leq j\leq k$, a non-terminal $P_{S(0)}^{j}$ with $\textup{eval}(P_{S(0)}^{j})=S(0)^{A^{j}}$.

The crucial observation is the following: for any tuple $(i_{1},\dots,i_{k})\in[A]^{k}$, let $p_{\ell}(i_{1},\dots,i_{k})=a_{i_{1}}[\ell]\cdots a_{i_{k}}[\ell]$. Then for any $\ell\in[d],j\in[k]$ and $(i_{1},\dots,i_{j})\in[A]^{j}$, we have that $a_{i_{j}}[\ell]=0$ implies $p_{\ell}(i_{1},\dots,i_{j},i^{\prime}_{j+1},\dots,i^{\prime}_{k})=0$ for all $(i^{\prime}_{j+1},\dots,i^{\prime}_{k})\in[A]^{k-j}$. We now define the final SLP using the starting non-terminal $S_{0}$ and the following productions

[TABLE]

It is straight-forward to verify that $\textup{eval}(S_{0})=V$. Note that the size of this SLP, i.e., the total number of non-terminals on the right hand side of the above rules, is bounded by ${O}(\gamma_{S}+dA)$. Moreover, the SLP can be constructed in linear time in its size. ∎

After this preparation, we can prove our conditional lower bounds.

Theorem 3.9**.**

Assuming the $k$-OV conjecture, Pattern Matching with Wildcards over alphabet $\{0,1\}$ (plus wildcards $*$) takes time $\min\{N,nM\}^{1-o(1)}$. This holds even restricted to instances with $n=\Theta(N^{\alpha_{n}})$, $M=\Theta(N^{\alpha_{M}})$ and $m=\Theta(N^{\alpha_{m}})$ for any $0<\alpha_{n}<1$ and $0<\alpha_{m}\leq\alpha_{M}\leq 1$.

Before we prove Theorem 3.9, let us sketch the main idea by providing a simple $N^{1-o(1)}$-time conditional lower bound in the setting $n,m=O(N^{\varepsilon})$ and $N=\Theta(M)$. Let ${\cal A}\subseteq\{0,1\}^{d}$ of size $A$ be an arbitrary $k$-OV instance with $k>1/\varepsilon$, and assume for simplicity $d\leq A^{o(1)}$. Using Lemma 3.8 on ${\cal A}$, $k$, $S(0)=0,S(1)=1$ and $b=(1,\dots,1)\in\{0,1\}^{d}$, we compute an SLP $\mathcal{T}$ for

[TABLE]

We define the pattern $P$ as

[TABLE]

Note that Pattern Matching with Wildcards on instance $T,P$ checks whether for some offset $\Delta$ we have $T[\Delta]=T[\Delta+A^{k}]=\ldots=T[\Delta+(d-1)A^{k}]=0$. Hence, by Lemma 3.8, pattern $P$ matches $T$ if and only if there is an orthogonal tuple $(a_{1},\dots,a_{k})\in{\cal A}^{k}$, showing correctness of the reduction.

Note that we have $N=\Theta(M)=\Theta(dA^{k})$. By Lemma 3.8, $T$ has an SLP of size $O(dA)$, and by Observation 2.2, $P$ has an SLP of size $O(d\log A)$. By $d\leq A^{o(1)}$ and $k>1/\varepsilon$, we are indeed in the setting $n,m=O(N^{\varepsilon})$ and $N=\Theta(M)$. An $O(N^{1-\varepsilon})$ algorithm for Pattern Matching with Wildcards would now imply an $O(A^{k(1-\varepsilon)}\textup{poly}(d))$ for $k$-OV, contradicting the $k$-OV conjecture.

We now give the slightly more involved general construction.

Proof of Theorem 3.9.

For $k\geq 2$, let ${\cal A}=\{a_{1},\ldots,a_{A}\}$ be a $k$-OV instance in $d$ dimensions, and let $k_{1},k_{2}\geq 1$ with $k_{1}+k_{2}=k$. We will construct an equivalent instance of Pattern Matching with Wildcards with $N=O(dA^{k})$, $M=O(dA^{k_{1}})$, $n=O(dA^{k_{2}+1})$, and $m=O(d\log A)$. Any $O(\min\{N,nM\}^{1-\varepsilon})$ algorithm for Pattern Matching with Wildcards would then imply an algorithm for $k$-OV in time $O(A^{(k+1)(1-\varepsilon)}\textup{poly}(d))=O(A^{k(1-\varepsilon/2)}\textup{poly}(d))$ for $k\geq 2/\varepsilon$, contradicting the $k$-OV conjecture. Below we strengthen this statement to hold restricted to instances with $n=\Theta(N^{\alpha_{n}})$, $M=\Theta(N^{\alpha_{M}})$ and $m=\Theta(N^{\alpha_{m}})$ for any $0<\alpha_{n}<1$ and $0<\alpha_{m}\leq\alpha_{M}\leq 1$.

To give such a reduction, we define the text as

[TABLE]

where $\min(b_{1},\dots,b_{\ell})$ denotes the component-wise minimum of $b_{1},\dots,b_{\ell}$.

We define the pattern $P$ as

[TABLE]

Correctness

Observe that $P$ cannot overlap any $1^{A^{k_{1}}}$-block, since never more than $A^{k_{1}}-1$ wildcards are followed by a 0 in $P$. Thus, $P$ matches $T$ if and only if there is a tuple $(j_{1},\dots,j_{k_{2}})\in[A]^{k_{2}}$ such that $P$ matches $T((j_{1},\dots,j_{k_{2}})):=\mathrm{tuplify}({\cal A},k_{1},\min(a_{j_{1}},\dots,a_{j_{k_{2}}}),0,1)$. By the structure of the pattern, $P$ matches any string $S$ if and only if there is an offset $\Delta$ such that $S[\Delta]=S[\Delta+A^{k_{1}}]=\dots=S[\Delta+(d-1)A^{k_{1}}]=0$. Thus, by Lemma 3.8, $P$ matches $T((j_{1},\dots,j_{k_{2}}))$ if and only if there are vectors $a_{1},\ldots,a_{k_{1}}\in{\cal A}$ for which $(a_{1},\dots,a_{k_{1}},\min(a_{j_{1}},\dots,a_{j_{k_{2}}}))$ is an orthogonal tuple. The latter condition is equivalent to $(a_{1},\dots,a_{k_{1}},a_{j_{1}},\dots,a_{j_{k_{2}}})$ being an orthogonal tuple. Since $k_{1}+k_{2}=k$ and $T$ contains $T((j_{1},\dots,j_{k_{2}}))$ for all $(j_{1},\dots,j_{k_{2}})\in[A]^{k_{2}}$, this proves that $P$ matches $T$ if and only if there is an orthogonal $k$-tuple in the instance ${\cal A}$.

Size Bounds

Note that $N=|T|=O(dA^{k})$. By Lemma 3.8 and Observation 2.2, we can compute an SLP $\mathcal{T}$ of size $n={O}(dA^{k_{2}+1})$ generating $T$, in linear time. Similarly, note that $M=|P|=O(dA^{k_{1}})$. By Observation 2.2, we can compute an SLP $\mathcal{P}$ of length $m={O}(d\log A)$ generating $P$, in linear time. This proves the claimed bounds.

Strengthening the Statement

We now prove the lower bound restricted to instances with $n=\Theta(N^{\alpha_{n}})$, $M=\Theta(N^{\alpha_{M}})$ and $m=\Theta(N^{\alpha_{m}})$ for any $0<\alpha_{n}<1$ and $0<\alpha_{m}\leq\alpha_{M}\leq 1$. Let $\varepsilon>0$ and set $\beta:=\min\{1,\alpha_{M}+\alpha_{n}\}$. We choose $k_{1},k_{2}\geq 1$ such that $k_{1}+k_{2}=k$ and $k_{1}\approx\min\{\alpha_{M},1-\alpha_{n}\}k/\beta$ and $k_{2}\approx\alpha_{n}k/\beta$. Note that $k_{1},k_{2}$ are restricted to be integers, however, for sufficiently large $k$ depending only on $\varepsilon,\alpha_{M},\alpha_{n}$, we can ensure $k_{1}\leq(1+\varepsilon/4)\min\{\alpha_{M},1-\alpha_{n}\}k/\beta$ and $k_{2}+1\leq(1+\varepsilon/4)\alpha_{n}k/\beta$. Note that for the dimension $d$ we can assume $d\leq A$, since otherwise an $O(A^{k-\varepsilon}\textup{poly}(d))$ algorithm clearly exists. In particular, for sufficiently large $k$ we have $d\leq A^{(\varepsilon/4)\cdot\min\{\alpha_{M},\alpha_{m},\alpha_{n},1-\alpha_{n}\}k/\beta}$. This yields

[TABLE]

Standard padding777Add a prefix of wildcards to the pattern and a prefix of 1’s to the text, and partially decompress the SLPs. of these four parameters allows us to achieve equality, up to constant factors, in the above inequalities, which yields the desired $n=\Theta(N^{\alpha_{n}})$, $M=\Theta(N^{\alpha_{M}})$ and $m=\Theta(N^{\alpha_{m}})$. Any $O(\min\{N,nM\}^{1-\varepsilon})$ algorithm for Pattern Matching with Wildcards in this setting would now imply an algorithm for $k$-OV in time $O(\min\{A^{(1+\varepsilon/2)k/\beta},A^{(1+\varepsilon/2)(\alpha_{M}+\alpha_{n})k/\beta}\}^{1-\varepsilon})=O(A^{(1+\varepsilon/2)(1-\varepsilon)\min\{1,\alpha_{M}+\alpha_{n}\}k/\beta})=O(A^{(1-\varepsilon/2)k})$, where we used the definition of $\beta$ and $(1+\varepsilon/2)(1-\varepsilon)\leq 1-\varepsilon/2$. This contradicts the $k$-OV conjecture, finishing the proof. ∎

We next prove a lower bound similar to Theorem 3.9 for another special case of generalized pattern matching, namely Substring Hamming Distance. Instead of a direct reduction from $k$-OV, we present a linear-time reduction from Pattern Matching with Wildcards over alphabet $\{0,1\}$ to Substring Hamming Distance.

Theorem 3.10**.**

Assuming the $k$-OV conjecture, Substring Hamming Distance on constant-size alphabet takes time $\min\{N,nM\}^{1-o(1)}$. This holds even restricted to instances with $n=\Theta(N^{\alpha_{n}})$, $M=\Theta(N^{\alpha_{M}})$ and $m=\Theta(N^{\alpha_{m}})$ for any $0<\alpha_{n}<1$ and $0<\alpha_{m}\leq\alpha_{M}\leq 1$.

Proof.

For short, we write $d_{H}(X,Y)$ for the Hamming distance of strings $X,Y$. We prove the result by reducing any Pattern Matching with Wildcards instance $T_{\mathrm{PM}},P_{\mathrm{PM}}$ over alphabet $\Sigma=\{0,1\}$ to an instance $T_{\mathrm{HD}},P_{\mathrm{HD}}$ of Substring Hamming Distance. We first define coordinate strings

[TABLE]

Observe that these strings are defined in such a way that $d_{H}(s_{P}(*),s_{T}(y))=1$ for $y\in\{0,1\}$, $d_{H}(s_{P}(x),s_{T}(y))=1$ for $x=y\in\{0,1\}$, and $d_{H}(s_{P}(x),s_{T}(y))=3$ if $x\neq y$, $x,y\in\{0,1\}$.

We introduce the guarding $G(s):=s\mathop{\circ}2\,3\,4$ for length-3 strings $s\in\{0,1\}^{3}$. This allows us to reduce $T_{\mathrm{PM}}$, $P_{\mathrm{PM}}$ to the following instance, using alphabet $\Sigma=\{0,1,2,3,4\}$,

[TABLE]

Note that for any $0\leq i\leq N-M$,

[TABLE]

where $\mathrm{mismatch}(z,z^{\prime})=\#\{i\mid z^{\prime}[i]\neq*,z[i]\neq z^{\prime}[i]\}$ is the number of mismatches of $z$ and $z^{\prime}$.

We now observe that for all $i$ with $i\bmod 6\neq 0$, we have $d_{H}(T_{\mathrm{HD}}[i+1..i+6M],P_{\mathrm{HD}})\geq 3M$, as no two symbols $2,3,4$ in $P_{\mathrm{HD}}$ are aligned, so that each $G(s_{P}(P_{\mathrm{PM}}[j]))$ contributes at least 3 to the Hamming distance. Since $d_{H}(T_{\mathrm{HD}}[6i+1..6i+6M],P_{\mathrm{HD}})\leq 3M$ for all $i$, the substring with smallest Hamming distance has thus a Hamming distance of $M+2\cdot\min_{0\leq i\leq N-M}\mathrm{mismatch}(T_{\mathrm{PM}}[i+1..i+M],P_{\mathrm{PM}})$. This value is equal to $M$ if and only if $P_{\mathrm{PM}}$ matches $T_{\mathrm{PM}}$, proving correctness.

The corresponding reduction of the compressed problems is straightforward: We can augment the SLP $\mathcal{T}_{\mathrm{PM}}$ for $T_{\mathrm{PM}}$ by ${O}(1)$-sized productions to obtain an SLP $\mathcal{T}_{\mathrm{HD}}$ for $T_{\mathrm{HD}}$, by replacing each terminal $\sigma\in\{0,1,*\}$ by a non-terminal evaluating to $G(s_{T}(\sigma))$. Analogously, we can compute an SLP for $P_{\mathrm{HD}}$ of size $|\mathcal{P}_{\mathrm{HD}}|=|\mathcal{P}_{\mathrm{PM}}|+{O}(1)$ in linear time. Overall, since also $|T_{\mathrm{HD}}|={O}(|T_{\mathrm{PM}}|),|P_{\mathrm{HD}}|={O}(|P_{\mathrm{PM}}|)$, all parameters are preserved up to constant factors. By this linear-time parameter-preserving reduction, the lower bound of Theorem 3.9 translates to Substring Hamming Distance, yielding the claim. ∎

3.3 Longest Common Subsequence

In this section, we study the Longest Common Subsequence (LCS) problem. Recall that a string $S$ of length $\ell$ is a substring of a string $X$ if there are $1\leq i_{1}<\dots<i_{\ell}\leq|X|$ with $S[j]=X[i_{j}]$ for any $j\in[\ell]$. In the LCS problem, given two strings $X,Y$, the task is to determine the longest string $S$ that is a subsequence of both $X$ and $Y$. We denote the length of the LCS by $L(X,Y)=|S|$, and more precisely consider the problem of computing $L(X,Y)$. In the whole section, the alphabet $\Sigma$ has constant size.

Problem 3.11** (LCS).**

Given strings $X,Y$ of length at most $N$ by grammar-compressed representations ${\cal X},{\cal Y}$ of size at most $n$, compute the length of the LCS of $X$ and $Y$.

As discussed in the introduction, the $O(nN\sqrt{\log{N/n}})$ time algorithm by Gawrychowski [35] is the fastest known. Here we prove a matching lower bound of $(Nn)^{1-o(1)}$, assuming the $k$-OV conjecture.

Theorem 3.12**.**

Assuming the $k$-OV conjecture, there is no $(nN)^{1-\varepsilon}$-time algorithm for LCS for any $\varepsilon>0$. This even holds restricted to instances with $n=\Theta(N^{\alpha_{n}})$ for any $0<\alpha_{n}<1$, and an alphabet of constant size.

The general approach is very similar to the lower bound for Pattern Matching with Wildcards given in Section 3.2. In particular, we again use the tuplified representation $T=\mathop{\bigcirc}_{i=1}^{dA^{k}}T[i]$ of Lemma 3.8 for a $k$-OV instance ${\cal A}$. Recall that this allows us to decide the $k$-OV instance by testing whether there is a subsequence of $d$ substrings $T[\Delta],T[\Delta+A^{k}],\dots,T[\Delta+(d-1)A^{k}]$ all equal to a certain 0-coordinate string. Finding a pattern to test this was quite simple for Pattern Matching with Wildcards, yielding an $N^{1-o(1)}$ lower bound. For LCS, enforcing a coherent offset is much more complicated, since the “pattern” is matched as a subsequence not as a substring. Furthermore, the extension to a $(nN)^{1-o(1)}$ lower bound is more involved and relies on the quadratic-time nature of LCS. Fortunately, we can overcome the technical obstacles for LCS using (an extension of) alignment gadgets developed in [17]. We first redevelop and extend the corresponding alignment gadget tools in Section 3.3.1, then give the lower bound for compressed instances for general distance measures in Section 3.3.2 and then finish our LCS lower bound by designing an alignment gadget for LCS in Section 3.3.3.

3.3.1 Alignment Gadget Framework

We start by reviewing and adapting the definitions of [17]. In particular, we extend the alignment gadget definition for our purposes.

More generally than LCS, we consider an arbitrary similarity measure $\delta:{\cal I}\times{\cal I}\to\mathbb{N}$. For LCS, the set of inputs ${\cal I}$ is the set of all strings over some sufficiently large constant-sized alphabet $\Sigma$, and $\delta(X,Y):=|X|+|Y|-2L(X,Y)$, where $L(X,Y)$ is the length of the LCS of $X$ and $Y$.

Any sequence $X\in{\cal I}$ is assigned an (abstract) type $\mathrm{type}(X)$. For LCS, we use $\mathrm{type}(X):=(|X|,\Sigma)$, where $|X|$ is the length of $X$ and $\Sigma$ the alphabet over which $X$ is defined. We define ${\cal I}_{t}:=\{X\in{\cal I}\mid\mathrm{type}(X)=t\}$ as the set of all inputs of type $t$.

Alignments

Let $n\geq m$. An alignment is a set $\Lambda=\{(i_{1},j_{1}),\ldots,(i_{k},j_{k})\}$ with $0\leq k\leq m$ such that $1\leq i_{1}<\ldots<i_{k}\leq n$ and $1\leq j_{1}<\ldots<j_{k}\leq m$. We say that $(i,j)\in\Lambda$ are aligned. Any $i\in[n]$ or $j\in[m]$ that is not contained in any pair in $\Lambda$ is called unaligned. We denote the set of all alignments (with respect to $n,m$) by ${\mathbf{\Lambda}}_{n,m}$.

We call the alignment $\{(\Delta+1,1),\ldots,(\Delta+m,m)\}$, with $0\leq\Delta\leq n-m$, a structured alignment. We denote the set of all structured alignments by ${\cal S}_{n,m}$.

Defining the cost of an alignment $\Lambda\in{\mathbf{\Lambda}}_{n,m}$, we deviate from [17]: for any $X_{1},\ldots,X_{n}\in{\cal I}$ and $Y_{1},\ldots,Y_{m}\in{\cal I}$, we define the cost of $\Lambda=\{(i_{1},j_{1}),\dots,(i_{|\Lambda|},j_{|\Lambda|})\}$ as

[TABLE]

where we set $\gamma:=\max_{i,j}\delta(X_{i},Y_{j})$. In other words, (1) for any $j\in[m]$ which is aligned to some $i$, we “pay” the distance $\delta(X_{i},Y_{j})$, (2) if $\Lambda$ is unstructured because it contains an unaligned $j$, we “pay” a penalty of $\gamma$ for each such unaligned $j$ (note that there are $m-|\Lambda|$ unaligned $j\in[m]$) and (3) if $\Lambda$ is unstructured because it aligns all $j$ but leaves out some $i$ between the first and last aligned $i$, then for any unaligned $i$ that is between the first aligned $i_{1}$ and last aligned $i_{|\Lambda|}$, we also “pay” a penalty of $\gamma$ (note that $\sum_{k=1}^{|\Lambda|-1}(i_{k+1}-i_{k}-1)=i_{|\Lambda|}-i_{1}-|\Lambda|+1$). This means that we incur punishment for any deviation from a structured alignment.

In [17], the cost of an alignment was defined to be the smaller quantity $\sum_{k=1}^{|\Lambda|}\delta(X_{i_{k}},Y_{j_{k}})+(m-|\Lambda|)\gamma$, i.e., unstructured alignments (that still align all $j\in[m]$) were punished less. For structured alignments both definitions coincide. Hence, the following extended alignment gadget is more powerful than the alignment gadget defined in [17].

Definition 3.13** (Extended alignment gadget).**

*The similarity measure $\delta$ admits an *extended alignment gadget, if the following conditions hold: given instances $X_{1},\dots,X_{n}\in{\cal I}_{t_{{\textsc{x}}}}$, $Y_{1},\dots,Y_{m}\in{\cal I}_{t_{{\textsc{y}}}}$ with $m\leq n$ and types $t_{\textsc{x}}=(\ell_{\textsc{x}},\Sigma),t_{\textsc{y}}=(\ell_{\textsc{y}},\Sigma)$, we can construct new instances $X={\mathrm{GA}}_{{\textsc{x}}}^{m,t_{{\textsc{y}}}}(X_{1},\dots,X_{n})$ and $Y={\mathrm{GA}}_{{\textsc{y}}}^{n,t_{{\textsc{x}}}}(Y_{1},\dots,Y_{m})$ and $C\in\mathbb{Z}$ such that

[TABLE]

Moreover, $\mathrm{type}(X)$, $\mathrm{type}(Y)$ and $C$ only depend on $n,m,t_{\textsc{x}},t_{\textsc{y}}$. Finally, $|X|,|Y|=\Theta((n+m)(\ell_{\textsc{x}}+\ell_{\textsc{y}}))$.

Definition 3.14** (Compressible alignment gadget).**

We call an extended alignment gadget compressible, if $X$ and $Y$ are of the form $X=X_{L}\left(\mathop{\bigcirc}_{i=1}^{n}\mathrm{pad}_{\textsc{x}}(X_{i})\right)X_{R}$ and $Y=Y_{L}\left(\mathop{\bigcirc}_{j=1}^{m}\mathrm{pad}_{\textsc{y}}(Y_{j})\right)Y_{R}$ for some strings $X_{L},X_{R},Y_{L},Y_{R}$ and functions $\mathrm{pad}_{\textsc{x}}:{\cal I}_{t_{\textsc{x}}}\to{\cal I}$ and $\mathrm{pad}_{\textsc{y}}:{\cal I}_{t_{\textsc{y}}}\to{\cal I}$ that satisfy the following properties:

$X_{L},X_{R},Y_{L},Y_{R}$* have SLPs of size ${O}(\log n+\log(\ell_{\textsc{x}}+\ell_{\textsc{y}}))$, computable in linear time in the output.* 2. 2.

Given SLPs ${\cal X}_{i},{\cal Y}_{j}$ for $X_{i},Y_{j}$, we can compute SLPs for $\mathrm{pad}_{\textsc{x}}(X_{i}),\mathrm{pad}_{\textsc{y}}(Y_{j})$ of size ${O}(|{\cal X}_{i}|+\log(\ell_{\textsc{x}}+\ell_{\textsc{y}})),{O}(|{\cal Y}_{j}|+\log(\ell_{\textsc{x}}+\ell_{\textsc{y}}))$ in linear time in the output.

In Section 3.3.3, we provide a compressible extended alignment gadget for LCS.

At the lowest level of our construction, we need the following notion.

Definition 3.15**.**

The similarity measure $\delta$ admits coordinate values, if there exist $\mathbf{0}_{{\textsc{x}}},\mathbf{0}_{{\textsc{y}}},\mathbf{1}_{{\textsc{x}}},\mathbf{1}_{{\textsc{y}}}\in{\cal I}$ satisfying

[TABLE]

and, moreover, $\mathrm{type}(\mathbf{0}_{{\textsc{x}}})=\mathrm{type}(\mathbf{1}_{{\textsc{x}}})$ and $\mathrm{type}(\mathbf{0}_{{\textsc{y}}})=\mathrm{type}(\mathbf{1}_{{\textsc{y}}})$.

3.3.2 General Lower Bound

The following theorem proves a conditional lower bound of $(Nn)^{1-o(1)}$ for any similarity measure admitting a compressible extended alignment gadget and coordinate values.

Theorem 3.16**.**

Let $\delta$ be a similarity measure admitting a compressible extended alignment gadget and coordinate values. Then unless the $k$-OV conjecture fails, there is no $(nN)^{1-o(1)}$-time algorithm for computing the value $\delta(X,Y)$, given SLPs ${\cal X},{\cal Y}$ of size at most $n$ generating strings $X,Y$ of length at most $N$. This even holds restricted to instances with $n=\Theta(N^{\alpha_{n}})$ for any $0<\alpha_{n}<1$, and constant alphabet size.

We prove this theorem in the remainder of this section.

Let ${\cal A}=\{a_{1},\ldots,a_{A}\}$ be a $k$-OV instance in $d-1$ dimensions. We augment all vectors in ${\cal A}$ by another dimension where all vectors are 0 to obtain ${\cal A}_{0}$, or where all vectors are 1 to obtain ${\cal A}_{1}$. For any $k^{\prime}\geq 1$ we let ${\cal A}^{(k^{\prime})}:=\{\min(a_{i_{1}},\ldots,a_{i_{k^{\prime}}})\mid i_{1},\ldots,i_{k^{\prime}}\in[A]\}$, i.e., for each $k^{\prime}$-tuple of vectors in ${\cal A}$ the set ${\cal A}^{(k^{\prime})}$ contains the pointwise minimum of this $k^{\prime}$-tuple. Note that ${\cal A}^{(k^{\prime})}$ is in general a multiset, it has size $|{\cal A}^{(k^{\prime})}|=A^{k^{\prime}}$, and is naturally ordered by the lexicographic ordering on $k^{\prime}$-tuples $(i_{1},\ldots,i_{k^{\prime}})\in[A]^{k^{\prime}}$. Similarly, we define ${\cal A}_{0}^{(k^{\prime})}$ and ${\cal A}_{1}^{(k^{\prime})}$ for the augmented vectors. We split $k=k_{1}+2k_{2}$ for some $k_{1},k_{2}\geq 1$ and set

[TABLE]

Observe that deciding the given $k$-OV instance is equivalent to testing whether there are orthogonal vectors $({\boldsymbol{a}},{\boldsymbol{b}},{\boldsymbol{c}})$ with ${\boldsymbol{a}}\in{\boldsymbol{A}}$, ${\boldsymbol{b}}\in{\boldsymbol{B}}$ and ${\boldsymbol{c}}\in{\boldsymbol{C}}$. In particular, the additional dimension is irrelevant for orthogonality, since we choose at least one vector in ${\boldsymbol{A}}$ and any such vector has the last coordinate equal to 0. For any $\ell\in[A^{k_{1}}]$, we denote by ${\boldsymbol{a}}(\ell)$ the $\ell$-th vector in ${\boldsymbol{A}}$.

Tuple gadgets.

For any ${\boldsymbol{b}}\in{\boldsymbol{B}}$, ${\boldsymbol{c}}\in{\boldsymbol{C}}$, we define vectors $u_{\boldsymbol{b}}\in\{0,1\}^{dA^{k_{1}}}$ and $v_{\boldsymbol{c}}\in\{0,1\}^{(d-1)A^{k_{1}}+1}$:

[TABLE]

In other words, for $j\in[d]$ and $\ell\in[A^{k_{1}}]$ we have $(u_{\boldsymbol{b}})_{j\cdot d+\ell}={\boldsymbol{a}}(\ell)[j+1]\cdot{\boldsymbol{b}}[j+1]$ as well as $(v_{\boldsymbol{c}})_{j\cdot d+\ell}={\boldsymbol{c}}[j+1]$ if $\ell=1$ and $(v_{\boldsymbol{c}})_{j\cdot d+\ell}=0$ otherwise.

The key idea is as follows. Consider a structured alignment $\Lambda=\{(\Delta+1,1),\dots,(\Delta+m,m)\}\in{\cal S}_{n,m}$ for the above vectors, where $n=dA^{k_{1}}$ and $m=(d-1)A^{k_{1}}+1$. This chooses some tuple ${\boldsymbol{a}}(\Delta+1)\in{\boldsymbol{A}}$ and aligns the pairs $({\boldsymbol{a}}(\Delta+1)[\ell]\cdot{\boldsymbol{b}}[\ell],{\boldsymbol{c}}[\ell])$ for all $\ell\in[d]$, additional to some trivial pairs where the coordinate of $v_{\boldsymbol{c}}$ is 0. This allows us to determine whether $({\boldsymbol{a}}(\Delta+1),{\boldsymbol{b}},{\boldsymbol{c}})$ is orthogonal.

To formalize this, create $\tilde{u}_{\boldsymbol{b}}$ by replacing each 0- and 1-entry in $u_{\boldsymbol{b}}$ by $\mathbf{0}_{{\textsc{x}}}$ and $\mathbf{1}_{{\textsc{x}}}$ (from Definition 3.15), and create $\tilde{v}_{\boldsymbol{c}}$ by replacing each 0- and 1-entry in $v_{\boldsymbol{c}}$ by $\mathbf{0}_{{\textsc{y}}}$ and $\mathbf{1}_{{\textsc{y}}}$, respectively. Let $t_{\textsc{x}}$ and $t_{\textsc{y}}$ be the types of $\mathbf{0}_{{\textsc{x}}},\mathbf{1}_{{\textsc{x}}}$ and $\mathbf{0}_{{\textsc{y}}},\mathbf{1}_{{\textsc{y}}}$, respectively. Set $\delta_{0}:=\delta(\mathbf{0}_{{\textsc{x}}},\mathbf{0}_{{\textsc{y}}})=\delta(\mathbf{0}_{{\textsc{x}}},\mathbf{1}_{{\textsc{y}}})=\delta(\mathbf{1}_{{\textsc{x}}},\mathbf{0}_{{\textsc{y}}})$ and $\delta_{1}:=\delta(\mathbf{1}_{{\textsc{x}}},\mathbf{1}_{{\textsc{y}}})$. We define the tuple gadgets

[TABLE]

Let $t_{\textsc{x}}^{\prime},t_{\textsc{y}}^{\prime}$ denote the types of ${\mathrm{TG}}_{\textsc{x}}({\boldsymbol{b}})$, ${\mathrm{TG}}_{\textsc{y}}({\boldsymbol{c}})$, and let $C$ be the number obtained from Definition 3.13 when creating ${\mathrm{TG}}_{\textsc{x}}({\boldsymbol{b}})$, ${\mathrm{TG}}_{\textsc{y}}({\boldsymbol{c}})$. Note that $t_{\textsc{x}}^{\prime},t_{\textsc{y}}^{\prime}$, and $C$ do not depend on the choice of ${\boldsymbol{b}}\in{\boldsymbol{B}},{\boldsymbol{c}}\in{\boldsymbol{C}}$.

Claim 3.17**.**

Let ${\boldsymbol{b}}\in{\boldsymbol{B}},{\boldsymbol{c}}\in{\boldsymbol{C}}$ and set $n:=dA^{k_{1}}$ and $m:=(d-1)A^{k_{1}}+1$. If there exists ${\boldsymbol{a}}\in{\boldsymbol{A}}$ such that $({\boldsymbol{a}},{\boldsymbol{b}},{\boldsymbol{c}})$ are orthogonal, then $\delta({\mathrm{TG}}_{\textsc{x}}({\boldsymbol{b}}),{\mathrm{TG}}_{\textsc{y}}({\boldsymbol{c}}))=C+m\cdot\delta_{0}$. Otherwise $\delta({\mathrm{TG}}_{\textsc{x}}({\boldsymbol{b}}),{\mathrm{TG}}_{\textsc{y}}({\boldsymbol{c}}))\geq C+(m-1)\delta_{0}+\delta_{1}$.

Proof.

If there is an ${\boldsymbol{a}}\in{\boldsymbol{A}}$ for which $({\boldsymbol{a}},{\boldsymbol{b}},{\boldsymbol{c}})$ are orthogonal, let $\Delta$ be such that ${\boldsymbol{a}}={\boldsymbol{a}}(\Delta+1)$, where ${\boldsymbol{a}}(\ell)$ is the $\ell$-th tuple in the lexicographic ordering of ${\boldsymbol{A}}$. The structured alignment $\Lambda=\{(\Delta+1,1),\dots,(\Delta+m,m)\}$ satisfies

[TABLE]

Furthermore, for any $\Lambda\in{\mathbf{\Lambda}}_{n,m}$, we have $\mathrm{cost}(\Lambda)\geq m\cdot\delta_{0}$, since $\mathrm{cost}(\Lambda)$ contains at least $m$ summands of value at least $\min\{\gamma,\min_{i,j}\delta(X_{i},Y_{j})\}=\min_{i,j}\delta(X_{i},Y_{j})\geq\delta_{0}$. Thus $\delta({\mathrm{TG}}_{\textsc{x}}({\boldsymbol{b}}),{\mathrm{TG}}_{\textsc{y}}({\boldsymbol{c}}))=C+m\cdot\delta_{0}$ by Definition 3.13.

Otherwise, if no such ${\boldsymbol{a}}$ exists, let $\Lambda\in{\mathbf{\Lambda}}_{n,m}$ be arbitrary. If $\Lambda=\{(\Delta+1,1),\dots,(\Delta+m,m)\}$ is a structured alignment, then

[TABLE]

since there exists some $\ell\in[d]$ with ${\boldsymbol{a}}(\Delta+1)[\ell]={\boldsymbol{b}}[\ell]={\boldsymbol{c}}[\ell]=1$ which contributes a value of $\delta_{1}$.

If $\Lambda=\{(i_{1},j_{1}),\dots,(i_{|\Lambda|},j_{|\Lambda|})\}$ is unstructured, then either $|\Lambda|<m$, in which case we have

[TABLE]

or $|\Lambda|=m$ and $i_{m}-i_{1}>m-1$, and thus

[TABLE]

Thus by Definition 3.13, $\delta({\mathrm{TG}}_{\textsc{x}}({\boldsymbol{b}}),{\mathrm{TG}}_{\textsc{y}}({\boldsymbol{c}}))\geq C+(m-1)\delta_{0}+\delta_{1}$. ∎

Normalization.

As usual in these kinds of reductions, we need a normalization step. We define a normalization sequence as

[TABLE]

Claim 3.18**.**

For any ${\boldsymbol{c}}\in{\boldsymbol{C}}$, we have $\delta({\mathrm{TG}}_{\mathrm{norm}},{\mathrm{TG}}_{\textsc{y}}({\boldsymbol{c}}))=C+(m-1)\delta_{0}+\delta_{1}$.

Proof.

Let $n=dA^{k_{1}}$ and $m=(d-1)A^{k_{1}}+1$. Let $\Lambda=\{(\Delta+1,1),\dots,(\Delta+m,m)\}\in{\cal S}_{n,m}$ be a structured alignment. Then by construction of ${\mathrm{TG}}_{\mathrm{norm}}$ and ${\mathrm{TG}}_{\textsc{y}}({\boldsymbol{c}})$, the only pair corresponding to $\mathbf{1}_{{\textsc{x}}}$ and possibly $\mathbf{1}_{{\textsc{y}}}$ entries is the pair $(\Delta+m,m)$, since only the last $A^{k_{1}}$ entries of ${\mathrm{TG}}_{\mathrm{norm}}$ are $\mathbf{1}_{{\textsc{x}}}$, and the only possible $\mathbf{1}_{{\textsc{y}}}$-entry of ${\mathrm{TG}}_{\textsc{y}}({\boldsymbol{c}})$ that could be aligned with one of them is its final entry. Now we use that we constructed the vectors ${\boldsymbol{C}}$ as ${\cal A}_{1}^{(k_{2})}$, i.e., we augmented all vectors by a $d$-th coordinate 1, which implies that the $m$-th entry of ${\mathrm{TG}}_{\textsc{y}}({\boldsymbol{c}})$ is indeed $\mathbf{1}_{{\textsc{y}}}$. Hence, the pair $(\Delta+m,m)$ contributes a distance of $\delta_{1}$ while all others contribute $\delta_{0}$. This yields $\mathrm{cost}(\Lambda)=(m-1)\delta_{0}+\delta_{1}$.

Let $\Lambda\in{\mathbf{\Lambda}}_{n,m}\setminus{\cal S}_{n,m}$ be an unstructured alignment. Then its cost is at least $\mathrm{cost}(\Lambda)\geq(m-1)\delta_{0}+\gamma\geq(m-1)\delta_{0}+\delta_{1}$, since it contains at least $m-1$ summands of value $\min\{\gamma,\min_{i,j}\delta(X_{i},Y_{j})\}\geq\delta_{0}$ and at least one punishment term $\gamma\geq\delta_{1}$ for a deviation from a structured assignment. Thus by Definition 3.13, we have $\delta({\mathrm{TG}}_{\mathrm{norm}},{\mathrm{TG}}_{\textsc{y}}({\boldsymbol{c}}))=C+(m-1)\delta_{0}+\delta_{1}$. ∎

We now define for any ${\boldsymbol{b}}\in{\boldsymbol{B}}$, ${\boldsymbol{c}}\in{\boldsymbol{C}}$ the normalized tuple gadgets

[TABLE]

We let $t_{\textsc{x}}^{\prime\prime},t_{\textsc{y}}^{\prime\prime}$ denote the resulting types of ${\mathrm{NTG}}_{\textsc{x}}({\boldsymbol{b}}),{\mathrm{NTG}}_{\textsc{y}}({\boldsymbol{c}})$, and $C^{\prime}$ be the number obtained from Definition 3.13 when creating ${\mathrm{NTG}}_{\textsc{x}}({\boldsymbol{b}}),{\mathrm{NTG}}_{\textsc{y}}({\boldsymbol{c}})$. Note that $t_{\textsc{x}}^{\prime\prime},t_{\textsc{y}}^{\prime\prime}$, and $C^{\prime}$ do not depend on the choice of ${\boldsymbol{b}}\in{\boldsymbol{B}},{\boldsymbol{c}}\in{\boldsymbol{C}}$. This definitions satisfies the following properties.

Claim 3.19**.**

Let ${\boldsymbol{b}}\in{\boldsymbol{B}},{\boldsymbol{c}}\in{\boldsymbol{C}}$. If there exists ${\boldsymbol{a}}\in{\boldsymbol{A}}$ such that $({\boldsymbol{a}},{\boldsymbol{b}},{\boldsymbol{c}})$ are orthogonal, then $\delta({\mathrm{NTG}}_{\textsc{x}}({\boldsymbol{b}}),{\mathrm{NTG}}_{\textsc{y}}({\boldsymbol{c}}))=\delta_{\mathrm{orth}}$, otherwise we have $\delta({\mathrm{NTG}}_{\textsc{x}}({\boldsymbol{b}}),{\mathrm{NTG}}_{\textsc{y}}({\boldsymbol{c}}))=\delta_{\mathrm{non}}$, where

[TABLE]

Proof.

We check all possible alignments $\Lambda\in{\mathbf{\Lambda}}_{2,1}$: If $\Lambda=\{(1,1)\}$, then $\mathrm{cost}(\Lambda)=\delta({\mathrm{TG}}_{\textsc{x}}({\boldsymbol{b}}),{\mathrm{TG}}_{\textsc{y}}({\boldsymbol{c}}))$. If $\Lambda=\{(2,1)\}$, we have $\mathrm{cost}(\Lambda)=\delta({\mathrm{TG}}_{\mathrm{norm}},{\mathrm{TG}}_{\textsc{y}}({\boldsymbol{c}}))$. For the only unstructured alignment $\Lambda=\emptyset$, we have $\mathrm{cost}(\Lambda)=\gamma\geq\max\{\delta({\mathrm{TG}}_{\textsc{x}}({\boldsymbol{b}}),{\mathrm{TG}}_{\textsc{y}}({\boldsymbol{c}})),\delta({\mathrm{TG}}_{\mathrm{norm}},{\mathrm{TG}}_{\textsc{y}}({\boldsymbol{c}}))\}$. Thus by Definition 3.13, we have $\delta({\mathrm{NTG}}_{\textsc{x}}({\boldsymbol{b}}),{\mathrm{NTG}}_{\textsc{y}}({\boldsymbol{c}}))=C^{\prime}+\min\{\delta({\mathrm{TG}}_{\textsc{x}}({\boldsymbol{b}}),{\mathrm{TG}}_{\textsc{y}}({\boldsymbol{c}})),\delta({\mathrm{TG}}_{\textsc{x}}({\boldsymbol{b}}),{\mathrm{TG}}_{\mathrm{norm}})\}$. The claim now follows from Claims 3.17 and 3.18. ∎

Final construction.

To obtain our final instance, we enumerate all ${\boldsymbol{b}}(1),\ldots,{\boldsymbol{b}}(A^{k_{2}})\in{\boldsymbol{B}}$ and ${\boldsymbol{c}}(1),\ldots,{\boldsymbol{c}}(A^{k_{2}})\in{\boldsymbol{C}}$ in an arbitrary fashion. We finally combine their corresponding normalized tuple gadgets by defining

[TABLE]

Let $C^{\prime\prime}$ be the number obtained from Definition 3.13 when creating $X$, $Y$.

Claim 3.20**.**

We have $\delta(X,Y)\leq C^{\prime\prime}+(A^{k_{2}}-1)\delta_{\mathrm{non}}+\delta_{\mathrm{orth}}$ if and only if there are ${\boldsymbol{a}}\in{\boldsymbol{A}},{\boldsymbol{b}}\in{\boldsymbol{B}},{\boldsymbol{c}}\in{\boldsymbol{C}}$ such that $({\boldsymbol{a}},{\boldsymbol{b}},{\boldsymbol{c}})$ is orthogonal.

Proof.

Assume that there exists an orthogonal set of vectors and let ${\boldsymbol{a}}\in{\boldsymbol{A}}$, ${\boldsymbol{b}}(i)\in{\boldsymbol{B}}$, ${\boldsymbol{c}}(j)\in{\boldsymbol{C}}$ be the vectors representing them. Let $n=2A^{k_{2}}$ and $m=A^{k_{2}}$. If $i\geq j$, we consider the structured alignment $\Lambda=\{(i-j+1,1),\dots,(i-j+m,m)\}$. Then $\Lambda$ aligns ${\mathrm{NTG}}_{\textsc{x}}({\boldsymbol{b}}(i)),{\mathrm{NTG}}_{\textsc{y}}({\boldsymbol{c}}(j))$, yielding cost $\delta_{\mathrm{orth}}$ by Claim 3.19. Since $\delta({\mathrm{NTG}}_{\textsc{x}}({\boldsymbol{b}}),{\mathrm{NTG}}_{\textsc{y}}({\boldsymbol{c}}))\leq\delta_{\mathrm{non}}$