Optimal Joins using Compact Data Structures

TL;DR

This paper introduces a storage-efficient, worst-case optimal join algorithm using compact data structures like quadtrees and qdags, eliminating the need for additional indexes and enabling efficient evaluation of complex queries.

Contribution

It presents a novel approach that achieves worst-case optimal join processing directly from compact point-set representations without extra storage overhead.

Findings

The proposed method is worst-case optimal in data complexity.

It uses compact quadtrees and qdags for static and dynamic indexing.

The framework extends to evaluate expressive relational algebra queries.

Abstract

Worst-case optimal join algorithms have gained a lot of attention in the database literature. We now count with several algorithms that are optimal in the worst case, and many of them have been implemented and validated in practice. However, the implementation of these algorithms often requires an enhanced indexing structure: to achieve optimality we either need to build completely new indexes, or we must populate the database with several instantiations of indexes such as B$+$-trees. Either way, this means spending an extra amount of storage space that may be non-negligible. We show that optimal algorithms can be obtained directly from a representation that regards the relations as point sets in variable-dimensional grids, without the need of extra storage. Our representation is a compact quad tree for the static indexes, and a dynamic quadtree sharing subtrees (which we dub a qdag) for intermediate results. We develop a compositional algorithm to process full join queries under this representation, and show that the running time of this algorithm is worst-case optimal in data complexity. Remarkably, we can extend our framework to evaluate more expressive queries from relational algebra by introducing a lazy version of qdags (lqdags). Once again, we can show that the running time of our algorithms is worst-case optimal.

Figures

Equations

$$R(A,B)\bowtie S(B,C)\bowtie T(A,C)=(R(A,B)\times \text{All}(C))\cap (S(B,C)\times \text{All}(A))\cap (T(A,C)\times \text{All}(B)).$$

$$(\text{AND},(\text{AND},(\text{EXTEND},(\text{QTREE},Q_R),\mathcal{A}),(\text{EXTEND},(\text{QTREE},Q_S),\mathcal{A})),(\text{EXTEND},(\text{QTREE},Q_T),\mathcal{A})).$$

$$(\text{JOIN},L_1({\mathcal{A}}_1),L_2({\mathcal{A}}_2))$$

$$(\text{DIFF},L_1,L_2)$$

$$(\text{AND},(\text{AND},(\text{EXTEND},(\text{QTREE},Q_R),\mathcal{A}),(\text{EXTEND},(\text{QTREE},Q_S),\mathcal{A})),(\text{EXTEND},(\text{NOT},Q_T),\mathcal{A})).$$

$$(\text{AND},\text{scChildAt}((\text{EXTEND},(\text{QTREE},Q_R),\mathcal{A}),101),\text{scChildAt}((\text{EXTEND},(\text{QTREE},Q_S),\mathcal{A}),101)).$$

$$F=(\text{OR},(\text{AND},(\text{QTREE},Q_R),(\text{QTREE},Q_S)),(\text{AND},(\text{QTREE},Q_R),(\text{QTREE},Q_T)))$$

Full text

We show that optimal algorithms can be obtained directly from a representation that regards the relations as point sets in variable-dimensional grids, without the need of extra storage. Our representation is a compact quadtree for the static indexes, and a dynamic quadtree sharing subtrees (which we dub a qdag) for intermediate results. We develop a compositional algorithm to process full join queries under this representation, and show that the running time of this algorithm is worst-case optimal in data complexity. Remarkably, we can extend our framework to evaluate more expressive queries from relational algebra by introducing a lazy version of qdags (lqdags). Once again, we can show that the running time of our algorithms is worst-case optimal.

1 Introduction

The state of the art in query processing has recently been shaken by a new generation of join algorithms with strong optimality guarantees based on the AGM bound of queries: the maximum size of the output of the query over all possible relations with the same cardinalities [2]. One of the basic principles of these algorithms is to disregard the traditional notion of a query plan, favoring a strategy that can take further advantage of the structure of the query, while at the same time taking into account the actual size of the database [14, 16].

Several of these algorithms have been implemented and tested in practice with positive results [8, 18], especially when handling queries with several joins. Because they differ from what is considered standard in relational database systems, the implementation of these algorithms often requires additional data structures, a database that is heavily indexed, or heuristics to compute the best computation path given the indexes that are present. For example, algorithms such as Leapfrog [21], Minesweeper [15], or InsideOut [10] must select a global order on the attributes, and assume that relations are indexed in a way that is consistent with these attributes [18]. If one wants to use these algorithms with more flexibility in the way attributes are processed, then one would probably need to instantiate several combinations of B+ trees or other indexes [8]. On the other hand, more involved algorithms such as Tetris [9] or Panda [11] require heavier data structures that allow reasoning over potential tuples in the answer.

Our goal is to develop optimal join algorithms that minimize the storage for additional indexes while at the same time being independent of a particular ordering of attributes. We address this issue by resorting to compact data structures: indexes using a nearly-optimal amount of space while supporting all operations we need to answer join queries.

We show that worst-case optimal algorithms can be obtained when one assumes that the input data is represented as quadtrees, and stored under a compact representation for cardinal trees [4]. Quadtrees are geometric structures used to represent data points in grids of size $\ell\times\ell$ (which can be generalized to any dimension). Thus, a relation $R$ with attributes $A_{1},\dots,A_{d}$ can be naturally viewed as a set of points over grids of dimension $d$, one point per tuple of $R$: the value of each attribute $A_{i}$ is the $i$-th coordinate of the corresponding point.

To support queries under this representation, our main tool is a new dynamic version of quadtrees, which we denote qdags, where some nodes may share complete subtrees. Using qdags, we can reduce the computation of a full join query $J=R_{1}\bowtie\cdots\bowtie R_{n}$ with $d$ attributes, to an algorithm that first extends the quadtrees for $R_{1},\dots,R_{n}$ into qdags of dimension $d$, and then intersects them to obtain a quadtree. Our first result shows that such algorithm is indeed worst-case optimal:

Theorem 1.1**.**

Let $R_{1}(\mathcal{A}_{1}),\ldots,R_{n}(\mathcal{A}_{n})$ be $n$ relations with attributes in $[0,\ell-1]$, and let $d=|\bigcup_{i}\mathcal{A}_{i}|$. We can represent the relations using only $\sum_{i}(|\mathcal{A}_{i}|+2+o(1))|R_{i}|\log\ell+O(n\log d)$ bits, so that the result of a join query $J=R_{1}\bowtie\cdots\bowtie R_{n}$ over a database instance $D$, can be computed in $\tilde{O}(\text{AGM})$ time111 $\tilde{O}$ hides poly-log $N$ factors, for $N$ the total input size, as well as factors that just depend on $d$ and $n$ (i.e., the query size), which are assumed to be constant. We provide a precise bound in Section 3.3. .

Note that just storing the tuples in any $R_{i}$ requires $|\mathcal{A}_{i}||R_{i}|\log\ell$ bits, thus our representation adds only a small extra space of $(2+o(1))|R_{i}|\log\ell+O(n\log d)$ bits (basically, two words per tuple, plus a negligible amount that only depends on the schema). Instead, any classical index on the raw data (such as hash tables or B$+$-trees) would pose a linear extra space, $O(|\mathcal{A}_{i}||R_{i}|\log\ell)$ bits, often multiplied by a non-negligible constant (especially if one needs to store multiple indexes on the data).

Our join algorithm works in a rather different way than the most popular worst-case algorithms. To illustrate this, consider the triangle query $J=R(A,B)\bowtie S(B,C)\bowtie T(A,C)$. The most common way of processing this query optimally is to follow what Ngo et al. [16] define as the generic algorithm: select one of the attributes of the query (say $A$), and iterate over all elements $a\in A$ that could be an answer to this query, that is, all $a\in\pi_{a}(R)\cap\pi_{a}(T)$. Then, for each of these elements, iterate over all $b\in B$ such that the tuple $(a,b)$ can be an answer: all $(a,b)$ in $(R\bowtie\pi_{B}(S))\bowtie\pi_{A}(T)$, and so on.

Instead, quadtrees divide the output space, which corresponds to a grid of size $\ell^{3}$, into 8 subgrids of size $(\ell/2)^{3}$, and for each of these grids it recursively evaluates the query. As it turns out, this strategy is as good as the generic strategy defined by Ngo et al. [16] to compute joins, and can even be extended to other relational operations, as we explain next.

Our join algorithm boils down to two simple operations on quadtrees: an Extend operation that lifts the quadtree representation of a grid to a higher-dimensional grid, and an And operation that intersects trees. But there are other operations that we can define and implement. For example, the synchronized Or of two quadtrees gives a compact representation of their union, and complementing the quadtree values can be done by a Not operation. We integrate all these operations in a single framework, and use it to answer more complex queries given by the combination of these expressions, as in relational algebra.

To support these operations we introduce lazy qdags, or lqdags for short, in which nodes may be additionally labeled with query expressions. The idea is to be able to delay the computation of an expression until we know such computation is needed to derive the output. To analyze our framework we extend the idea of a worst-case optimal algorithm to arbitrary queries: If a worst-case optimal algorithm to compute the output of a formula $F$ takes time $T$ over relations $R_{1},\ldots,R_{n}$ of sizes $s_{1},\ldots,s_{n}$, respectively, of a database $D$, then there exists a database $D^{\prime}$ with relations $R_{1}^{\prime},\ldots,R_{n}^{\prime}$ of sizes $O(s_{1}),\ldots,O(s_{n})$, respectively, where the output of $F$ over $R_{1}^{\prime},\ldots,R_{n}^{\prime}$ is of size $\Omega(T)$. We prove that lqdags allow us to maintain optimality in the presence of union and negation operators:

Theorem 1.2**.**

Let $Q$ be a relational algebra query built with joins, union and complement, and where no relation appears more than once in $Q$. Then there is an algorithm to evaluate $Q$ that is worst-case optimal in data complexity.

Consider, for example, the query $J^{\prime}=R(A,B)\bowtie S(B,C)\bowtie\overline{T}(A,C)$, which joins $R$ and $S$ with the complement $\overline{T}$ of $T$. One could think of two ways to compute this query. The first is just to join $R$ and $S$ and then see which of the resulting tuples are not in $T$. But if $T$ is dense ($\overline{T}$ is small), it may be more efficient to first compute $\overline{T}$ and then proceed as on the usual triangle query. Our algorithm is optimal because it can choose locally between both strategies: by dividing into quadrants one finds dense regions of $T$ in which computing $\overline{T}$ is cheaper, while in sparse regions the algorithm first computes the join of $R$ and $S$.

Our framework is the first in combining worst-case time optimality with the use of compact data structures. The latter can lead to improved performance in practice, because relations can be stored in faster memory, higher in the memory hierarchy [13]. This is especially relevant when the compact representation fits in main memory while a heavily indexed representation requires resorting to the disk, which is orders of magnitude slower. Under the recent trend of maintaining the database in the aggregate main memory of a distributed system, a compact representation leads to using fewer computers, thus reducing hardware, communication, and energy costs, while improving performance.

2 Quadtrees

A Region Quadtree [6, 19] is a structure used to store points in two-dimensional grids of $\ell\times\ell$. We focus on the variant called MX-Quadtree [22, 19], which can be described as follows. Assume for simplicity that $\ell$ is a power of 2. If $\ell=1$, then the grid has only one cell and the quadtree is an integer $1$ (if the cell has a point) or [math] (if not). For $\ell>1$, if the grid has no points, then the quadtree is a leaf. Otherwise, the quadtree is an internal node with four children, each of which is the quadtree of one of the four $\ell/2\times\ell/2$ quadrants of the grid. (The deepest internal nodes, whose children are $1\times 1$ grids, store instead four integers in $\{0,1\}$ to encode their cells.)

Assume each data point is described using the binary representation of each of its coordinates (i.e., as a pair of $\log\ell$-bit vectors). We order the grid quadrants so that the first contains all points with coordinates of the form $(0\cdot c_{x},0\cdot c_{y})$, for $\log\ell-1$ bit vectors $c_{x}$ and $c_{y}$, the second contains points $(0\cdot c_{x},1\cdot c_{y})$, the third $(1\cdot c_{x},0\cdot c_{y})$, and the last quadrant stores the points $(1\cdot c_{x},1\cdot c_{y})$. Fig. 1 shows a grid and its deployment as a quadtree.

Quadtrees can be generalized to higher dimensions. A quadtree of dimension $d$ is a tree used to represent data points in a $d$-dimensional grid $G$ of size $\ell^{d}$. Here, an empty grid is represented by a leaf and a nonempty grid corresponds to an internal node with $2^{d}$ children representing the $2^{d}$ subspaces spanning from combining the first bits of each dimension. Generalizing the case $d=1$, the children are ordered using the Morton [12] partitioning of the grid: a sequence of $2^{d}$ subgrids of size $(\ell/2)^{d}$ in which the $i$-th subgrid of the partition, represented by the binary encoding $b_{i}$ of $i$, is defined by all the points $(b_{c_{1}},\dots,b_{c_{d}})$ in which the word formed by concatenating the first bit of each string $b_{c_{j}}$ is precisely the string $b_{i}$.

A quadtree with $p$ points has at most $p\log\ell$ nodes (i.e., root-to-leaf paths). A refined analysis in two dimensions [7, Thm. 1] shows that quadtrees have fewer nodes when the points are clustered: if the points distribute along $c$ clusters, $p_{i}$ of them inside a subgrid of size $\ell_{i}\times\ell_{i}$, then there are in total $O(c\log\ell+\sum_{i}p_{i}\log\ell_{i})$ nodes in the quadtree. The result easily generalizes to $d$ dimensions: the cells are of size $\ell_{i}^{d}$ and the quadtree has $O(c\log\ell+\sum_{i}p_{i}\log\ell_{i})$ internal nodes, each of which stores $2^{d}$ pointers to children (or integers, in the last level).

Brisaboa et al. [4] introduced a compact quadtree representation called the $k^{d}$-tree. They represent each internal quadtree node as the $2^{d}$ bits telling which of its quadrants is empty (0) or nonempty (1). Leaves and single-cell nodes are not represented because their data is deduced from the corresponding bit of their parent. The $k^{d}$-tree is simply a bitvector $V$ obtained by concatenating the $2^{d}$ bits of every (internal) node in levelwise order. Each node is identified with its order in this deployment, the root being 1. Navigation on the quadtree (from a parent to its children, and vice-versa) is simulated in constant time using $o(|V|)$ additional bits on top of $V$. On a quadtree in dimension $d$ storing $p$ points, the length of the bitvector $V$ is $|V|\leq 2^{d}p\log\ell$, increasing exponentially with $d$. This bitvector is sparse, however, because it has at most $p\log\ell$ 1s, one per quadtree node. We then resort to a representation of high-arity cardinal trees introduced by Benoit et al. [3, Thm. 4.3], which requires only $(d+2)p\log\ell+o(p\log\ell)+O(\log d)$ bits, and performs the needed tree traversal operations in constant time.

Observation 2.1**.**

*(cf. Benoit et al. [3], Thm. 4.3)  

Let $Q$ be a quadtree storing $p$ points in $d$ dimensions with integer coordinates in the interval $[0,\log\ell-1]$. Then, there is a representation of $Q$ which uses $(d+2+o(1))p\log\ell+O(\log d)$ bits, can be constructed in linear expected time, and supports constant time parent-children navigation on the tree.*

From now on, by quadtree we refer to this compact representation. Next, we show how to represent relations using quadtrees and evaluate join queries over this representation.

3 Multi-way Joins using Qdags

We assume for simplicity that the domain $\mathcal{D}(A)$ of an attribute $A$ consists of all binary strings of length $\log\ell$, representing the integers in $[0,\ell-1]$, and that $\ell$ is a power of $2$.

A relation $R(\mathcal{A})$ with attributes $\mathcal{A}=\{A_{1},\dots,A_{d}\}$ can be naturally represented as a quadtree: simply interpret each tuple in $R(\mathcal{A})$ as a data point over a $d$-dimensional grid with $\ell^{d}$ cells, and store those points in a $d$-dimensional quadtree. Thus, using quadtrees one can represent the relations in a database using compact space. The convenience of this representation to handle restricted join queries with naive algorithms has been demonstrated practically on RDF stores [1]. In order to obtain a general algorithm with provable performance, we introduce qdags, an enhanced version of quadtrees, together with a new algorithm to efficiently evaluate join queries over the compressed representations of the relations.

We start with an example to introduce the basics behind our algorithms and argue for the need of qdags. We then formally define qdags and explore their relation with quadtrees. Finally, we provide a complete description of the join algorithm and analyze its running time.

3.1 The triangle query: quadtrees vs qdags

Let $R(A,B)$, $S(B,C)$, $T(A,C)$ be relations over the attributes $\{A,B,C\}$ denote the domains of $A,B$ and $C$ respectively, and consider the triangle query $R(A,B)\bowtie S(B,C)\bowtie T(A,C)$. The basic idea of the algorithm is as follows: we first compute a quadtree $Q_{R}^{*}$ that represents the cross product $R(A,B)\times\text{All}(C)$, where $\text{All}(C)$ is a relation with an attribute $C$ storing all elements in the domain $[0,\ell-1]$. Likewise, we compute $Q_{S}^{*}$ representing $S(B,C)\times\text{All}(A)$, and $Q_{T}^{*}$ representing $T(A,C)\times\text{All}(B)$. Note that these quadtrees represent points in the three-dimensional grid with a cell for every possible value in $\mathcal{D}(A)\times\mathcal{D}(B)\times\mathcal{D}(C)$, where we assume that the domains $\mathcal{D}(\cdot)$ of the attributes are all $[0,\ell-1]$. Finally, we traverse the three quadtrees in synchronization building a new quadtree that represents the intersection of $Q_{R}^{*}$, $Q_{S}^{*}$ and $Q_{T}^{*}$. This quadtree represents the desired output because

[TABLE]

Though this algorithm is correct, it can perform poorly in terms of space and running time. The size of $Q_{R}^{*}$, for instance, can be considerably bigger than that of $R$, and even than the size of the output of the query. If, for example, the three relations have $n$ elements each, the size of the output is bounded by $n^{3/2}$ [2], while building $Q_{R}^{*}$ costs $\Omega(n\ell)$ time and space. This inefficiency stems from the fact that quadtrees are not smart to represent relations of the form $R^{*}(\mathcal{A})=R(\mathcal{A}^{\prime})\times\text{All}(\mathcal{A}\setminus\mathcal{A}^{\prime})$, where $\mathcal{A}^{\prime}\subset\mathcal{A}$, with respect to the size of a quadtree representing $R(\mathcal{A}^{\prime})$. Due to its tree nature, a quadtree does not benefit from the regularities that appear in the grid representing $R^{*}(\mathcal{A})$. To remedy this shortcoming, we introduce qdags, quadtree-based data structures that represent sets of the form $R(\mathcal{A}^{\prime})\times\text{All}(\mathcal{A}\setminus\mathcal{A}^{\prime})$ by adding only constant additional space to the quadtree representing $R(\mathcal{A}^{\prime})$, for any $\mathcal{A}^{\prime}\subseteq\mathcal{A}$.

A qdag is an implicit representation of a $d$-dimensional quadtree $Q^{d}$ (that has certain regularities) using only a reference to a $d^{\prime}$-dimensional quadtree $Q^{d^{\prime}}$, with $d^{\prime}\leq d$, and an auxiliary mapping function that defines how to use $Q^{d^{\prime}}$ to simulate navigation over $Q^{d}$. Qdags can then represent relations of the form $R(\mathcal{A}^{\prime})\times\text{All}(\mathcal{A}\setminus\mathcal{A}^{\prime})$ using only a reference to a quadtree representing $R(\mathcal{A}^{\prime})$, and a constant-space mapping function.

To illustrate how a qdag works, consider a relation $S(B,C)$, and let $Q_{S}^{*}$ be a quadtree representing $S^{*}(A,B,C)=\text{All}(A)\times S(B,C)$. Since $Q_{S}^{*}$ stores points in the $\ell^{3}$ cube, each node in $Q_{S}^{*}$ has $8$ children. As $\text{All}(A)$ contains all $\ell$ elements, for each original point $(b,c)$ in $S$, $S^{*}$ contains $\ell$ points corresponding to elements $(0,b,c),\dots,(\ell-1,b,c)$. We can think of this as extending each point in $S$ to a box of dimension $\ell\times 1\times 1$. With respect to $Q_{S}^{*}$, this implies that, among the $8$ children of a node, the last $4$ children will always be identical to the first $4$, and their values will in turn be identical to those of the corresponding nodes in the quadtree $Q_{S}$ representing $S$. In other words, each of the four subgrids $1a_{1}a_{2}$ is identical to the subgrid $0a_{1}a_{2}$, and these in turn are identical to the subgrid $a_{1}a_{2}$ in $S$ (see Fig. 2 for an example). Thus, we can implicitly represent $Q_{S}^{*}$ by the pair $(Q_{S},M=[0,1,2,3,0,1,2,3])$: the root of $Q_{S}^{*}$ is the root of $Q_{S}$, and the $i$-child of the root of $Q_{S}^{*}$ is represented by the pair $(C,M)$, where $C$ is the $M[i]$-th child of the root of $Q_{S}$.

3.2 Qdags for relational data

We now introduce a formal definition of the qdags, and describe the algorithms which allow the evaluation of multijoin queries in worst-case optimal time.

Definition 3.1** (qdag).**

Let $Q^{d^{\prime}}$ be a quadtree representing a relation with $d^{\prime}$ attributes. A qdag $Q^{d}$, for $d\geq d^{\prime}$, is a pair $(Q^{d^{\prime}},M)$, with $M:[0,2^{d}-1]\rightarrow[0,2^{d^{\prime}}-1]$. This qdag represents a quadtree $Q$, which is called the completion of $Q^{d}$, as follows:

If $Q^{d^{\prime}}$ represents a single cell, then $Q$ represents a single cell with the same value. 2. 2.

If $Q^{d^{\prime}}$ represents a $d^{\prime}$-dimensional grid empty of points, then $Q$ represents a $d$-dimensional grid empty of points. 3. 3.

Otherwise, the roots of both $Q^{d^{\prime}}$ and $Q$ are internal nodes, and for all $0\leq i<2^{d}$, the $i$-th child of $Q$ is the quadtree represented by the qdag $(C(Q^{d^{\prime}},M[i]),M)$, where $C(Q^{d^{\prime}},j)$ denotes the $j$-th child of the root node of quadtree $Q^{d^{\prime}}$.

We say that a qdag represents the same relation $R$ represented by its completion. Note that, for any $d$-dimensional quadtree $Q$, one can generate a qdag whose completion is $Q$ simply as the pair $(Q,M)$, where $M$ is the identity mapping $M[i]=i$, for all $0\leq i<2^{d}$. We can then describe all our operations over qdags. Note, in particular, that we can use mappings to represent any reordering of the attributes.

In terms of representation, the references to quadtree nodes consist of the identifier of the quadtree and the index of the node in level-wise order. This suffices to access the node in constant time from its compact representation. For a qdag $Q^{\prime}=(Q,M)$, we denote by $|Q^{\prime}|$ the number of internal nodes in the base quadtree $Q$, and by $||Q^{\prime}||$ the number of internal nodes in the completion of $Q^{\prime}$.

Algorithms 1 and 2, based on Definition 3.1, will be useful for the navigation of qdags. Operation Value yields a 0 iff the subgrid represented by the qdag is empty (thus the qdag is a leaf); a $1$ if the qdag is a full single cell, and ½ if it is an internal node. Operation ChildAt lets us descend by a given child from internal nodes representing nonempty grids. The operations “integer $Q$”, “$Q$ is a leaf”, and “$C(Q,j)$” are implemented in constant time on the compact representation of $Q$.

Operation Extend. We introduce an operation to obtain, from the qdag representing a relation $R$, a new qdag representing the relation $R$ extended with new attributes.

Definition 3.2**.**

Let $\mathcal{A}^{\prime}\subseteq\mathcal{A}$ be sets of attributes, let $R(\mathcal{A}^{\prime})$ be a relation over $\mathcal{A}^{\prime}$, and let $Q_{R}=(Q,M)$ be a qdag that represents $R(\mathcal{A}^{\prime})$. The operation $\text{\sc Extend}(Q_{R},\mathcal{A})$ returns a qdag $Q_{R}^{*}=(Q,M^{\prime})$ that represents the relation $R\times\text{All}(\mathcal{A}\setminus\mathcal{A}^{\prime})$.

To provide intuition on its implementation, let $\mathcal{A}^{\prime}$ be the set of attributes $\{A,B,D\}$ and let $\mathcal{A}=\{A,B,C,D\}$, and consider $R(\mathcal{A}^{\prime})$, $Q_{R}$ and $Q_{R}^{*}$ from Definition 3.2. Each node of $Q_{R}$ has 8 children, while each node of $Q_{R}^{*}$ has 16 children. Consider the child at position $i=12$ of $Q_{R}^{*}$. This node represents the grid with Morton code $m_{4}$=‘1100’ (i.e., 12 in binary), and contains the tuples whose coordinates in binary start with 1 in attributes $A,B$ and with 0 in attributes $C,D$. This child has elements if and only if the child with Morton code $m_{3}$=‘110’ of $Q_{R}$ (i.e., its child at position $j=6$) has elements; this child is in turn the $M[6]$-th child of $Q$. Note that $m_{3}$ results from projecting $m_{4}$ to the positions 0,1,3 in which the attributes $A,B,D$ appear in $\{A,B,C,D\}$. Since the Morton code 1110’ (i.e., 14 in binary) also projects to $m_{3}$, it holds that $M^{\prime}[12]=M^{\prime}[14]=M[6]$. We provide an implementation of the Extend operation for the general case in Algorithm 3. The following lemma states the time and space complexity of our implementation of Extend. For simplicity, we count the space in terms of computer words used to store references to the quadtrees and values of the mapping function $M$.

Lemma 3.3**.**

Let $|\mathcal{A}|=d$ in Definition 3.2. Then, the operation $\text{\sc Extend}(Q_{R},\mathcal{A})$ can be supported in time $O(2^{d})$ and its output takes $O(2^{d})$ words of space.

Proof 3.4**.**

We show that Algorithm 3 meets the conditions of the lemma. The computations of $m_{d}$ and $i^{\prime}$ are immaterial (they just interpret a bitvector as a number or vice versa). The computation of $m_{d}^{\prime}$ is done with a constant table (that depends only on the database dimension $d$) of size $O(2^{3d})$:222They can be reduced to two tables of size $O(2^{2d})$, but we omit the details for simplicity. The argument $\mathcal{A}$ is given as a bitvector of size $d$ telling which attributes are in $\mathcal{A}$, the qdag on $\mathcal{A}^{\prime}$ stores a bitvector of size $d$ telling which attributes are in $\mathcal{A}^{\prime}$, and the table receives both bitvectors and $m_{d}$ and returns $m_{d}^{\prime}$.

3.3 Join algorithm

Now that we can efficiently represent relations of the form $R(\mathcal{A}^{\prime})\times\text{All}(\mathcal{A}\setminus\mathcal{A}^{\prime})$, for $\mathcal{A}^{\prime}\subseteq\mathcal{A}$, we describe a worst-case optimal implementation of joins over the qdag representations of the relations. Our algorithm follows the idea discussed for the triangle query: we first extend every qdag to all the attributes that appear in the query, so that they all have the same dimension and attributes. Then we compute their intersection, building a quadtree representing the output of the query. The implementation of this algorithm is surprisingly simple (see Algorithms 4 and 5), yet worst-case optimal, as we prove later on. Using qdags is key for this result; this algorithm would not be at all optimal if computed over relational instances stored using standard representations such as B+ trees. First, we describe how to compute the intersection of several qdags, and then analyze the running time of the join.

Operation And. We introduce an operation And, which computes the intersection of several relations represented as qdags.

Definition 3.5**.**

Let $Q_{1},\dots,Q_{n}$ be qdags representing relations $R_{1},\ldots,R_{n}$, all over the attribute set $\mathcal{A}$. Operation $\text{\sc And}(Q_{1},\dots,Q_{n})$ returns a quadtree $Q$ that represents the relation $R_{1}\cap\ldots\cap R_{n}$.

We implement this operation by simulating a synchronized traversal among the completions $C_{1},\ldots,C_{n}$ of $Q_{1},\ldots,Q_{n}$, respectively, obtaining the quadtree $Q$ that stores the cells that are present in all the quadtrees $C_{i}$ (see Algorithm 5).We proceed as follows. If $\ell=1$, then all $C_{i}$ are integers with values [math] or $1$, and $Q$ is an integer equal to the minimum of the $n$ values. Otherwise, if any $Q_{i}$ represents an empty subgrid, then $Q$ is also a leaf representing an empty subgrid. Otherwise, every $C_{i}$ is rooted by a node $v_{i}$ with $2^{d}$ children, and so is $Q$, where the $j$-th child of its root $v$ is the result of the And operation of the $j$-th children of the nodes $v_{1},\dots,v_{n}$. However, we need a final pruning step to restore the quadtree invariants (line 6 of Algorithm 5): if $\text{\sc Value}(v_{i})=0$ for all the resulting children of $v$, then $v$ must become a leaf and the children be discarded. Note that once the quadtree is computed, we can represent it succinctly in linear expected time so that, for instance, it can be cached for future queries involving the output represented by $Q$333 This consumes linear expected time due to the use of perfect hashing in the compact representation [3]. .

Analysis of the algorithm. We compute the output $Q$ of $\text{\sc And}(Q_{1},\dots,Q_{n})$ in time $O(2^{d}\cdot(||Q_{1}||+\cdots+||Q_{n}||))$. More precisely, the time is bounded by $O(2^{d}\cdot|Q^{+}|)$, where $Q^{+}$ is the quadtree that would result from Algorithm 5 if we remove the pruning step of line 6. We name this quadtree $Q^{+}$ as the non-pruned version of $Q$. Although the size of the actual output $Q$ can be much smaller than that of $Q^{+}$,we can still prove that our time is optimal in the worst case. We start with a technical result.

Lemma 3.6**.**

The And operation can be supported in time $O(M\cdot 2^{d}n\log\ell)$, where $M$ is the maximum number of nodes in a level of $Q^{+}$.

Proof 3.7**.**

We show that Algorithm 5 meets the conditions of the lemma. Let $m_{j}$ be the number of nodes of depth $j$ in $Q^{+}$, and then $M=\max_{0\leq j<\log\ell}m_{j}$. The number of steps performed by Algorithm 5 is bounded by $n\cdot(\sum_{0\leq j<\log\ell}m_{j}\cdot 2^{d})\leq n\cdot M\cdot\log\ell\cdot 2^{d}$: In each depth we continue traversing all qdags $Q_{1},\dots,Q_{n}$ as long as they are all nonempty, and we generate the corresponding nodes in $Q^{+}$ (even if at the end some nodes will disappear in $Q$).

All we need to prove (data) optimality is to show that $|Q^{+}|$ is bounded by the size of the real output of the query. Recall that, for a join query $J$ on a database $D$, we use $2^{\rho^{*}(J,D)}$ to denote the AGM bound [2] of the query $J$ over $D$, that is, the maximum size of the output of $J$ over any relational database having the same number of tuples as $D$ in each relation.

Theorem 3.8**.**

Let $J=R_{1}\bowtie\dots\bowtie R_{n}$ be a full join query, and $D$ a database over schema $\{R_{1},\dots,R_{n}\}$, with $d$ attributes in total, and where the domains of the relations are in $[0,\ell-1]$. Let $\mathcal{A}_{i}$ be the set of attributes of $R_{i}$, for all $1\leq i\leq n$ and $N=\sum_{i}{|R_{i}|}$ be the total amount of tuples in the database. The relations $R_{1},\ldots,R_{n}$ can then be stored within $\sum_{i}(|\mathcal{A}_{i}|+2+o(1))|R_{i}|\log\ell+O(n\log d)$ bits, so that the output for $J$ can be computed in time $O(2^{\rho^{*}(J,D)}\cdot 2^{d}n\log\min(\ell,N))=\tilde{O}(2^{\rho^{*}(J,D)})$.

Proof 3.9**.**

The space usage is a simple consequence of Observation 2.1. As for the time, to solve the join query $J$ we simply encapsulate the quadtrees representing $R_{1},\ldots,R_{n}$ in qdags $Q_{1},\ldots,Q_{n}$, and use Algorithm 4 to compute the result of the query. We now show that Algorithm 4 runs in time within the bound of the theorem. First, assume that $\log\ell$ is $O(\log N)$. Let each relation $R_{i}$ be over attributes $\mathcal{A}_{i}$, and $\mathcal{A}=\bigcup\mathcal{A}_{i}$ with $d=|\mathcal{A}|$. Let $Q_{i}^{*}=\text{\sc Extend}(Q_{i},\mathcal{A})$, $Q=\text{\sc And}(Q_{1}^{*},\dots,Q_{n}^{*})$, and $Q^{+}$ be the non-pruned version of $Q$. The cost of the Extend operations is only $O(2^{d}n)$, according to Lemma 3.3, so the main cost owes to the And operation.

If the maximum $M$ of Lemma 3.6 is reached at the lowest level of the decomposition, where we store integers [math] or $1$, then we are done: each $1$ at a leaf of $Q^{+}$ exists in $Q$ as well because that single tuple is present in all the relations $R_{1},\ldots,R_{n}$. Therefore, $M$ is bounded by the AGM bound of $J$ and the time of the And operation is bounded by $O(2^{\rho^{*}(J,D)}\cdot 2^{d}n\log\ell)$.

Assume instead that $M$ is the number of internal nodes at depth $0<j<\log\ell$ of $Q^{+}$ (if $M$ is reached at depth [math] then $M=1$). Intuitively, we will take the relations at the granularity of level $j$, and show that there exists a database $D^{\prime}$ where such a $(2^{j})^{d}$ relation arises in the last level and thus the answer has those $M$ tuples.

We then construct the following database $D^{\prime}$ with relations $R_{i}^{\prime}$: For a binary string $c$, let $(c,j)$ denote the first $j$ bits of $c$. Then, for each relation $R_{i}$ and each tuple $(c_{1},\dots,c_{d_{i}})$ in $R_{i}$, where $d_{i}=|\mathcal{A}_{i}|$, let $R_{i}^{\prime}$ contain the tuples (0^{\log\ell-j}(c_{1},j),0^{\log\ell-j}(c_{2},j)\dots,$$0^{\log\ell-j}(c_{d_{i}},j)), corresponding to taking the first $j$ bits of each coordinate and prepending them with a string of $\log\ell-j$ [math]s. While this operation may send two tuples in a relation in $D$ to a single tuple in $D^{\prime}$, we still have that each relation $R_{i}^{\prime}$ in $D^{\prime}$ contains at most as many tuples as relation $R_{i}$ in $D$. Moreover, if we again store every $R_{i}^{\prime}$ as a qdag and process their join as in Algorithm 4, then by construction we have in this case that the leaves of the tree resulting from the And operation contain exactly $M$ nodes with $1$, and that this is the maximum number of nodes in a level of this tree. Since the leaves represent tuples in the answer, we have that $M\leq 2^{\rho^{*}(J,D^{\prime})}\leq 2^{\rho^{*}(J,D)}$, which completes the proof for the case when $\log\ell$ is $O(\log N)$.

Finally, when $\log N$ is $o(\log\ell)$, we can convert $O(\log\ell)$ to $O(\log N)$ in the time complexities by storing $R_{1},\dots,R_{n}$ using quadtrees, with a slight variation. We store the values of the attributes appearing in any relation in an auxiliary data structure (e.g., an array), and associate an $O(\log N)$-bits identifier to each different value in $[0,\ell-1]$ that appears in $D$ (e.g., the index of the corresponding value in the array). In this case, we represent the relations in quadtrees, but using the identifiers of the attribute values instead of the values themselves. This representation requires at most $dN\log\ell$ bits for the representation of the distinct attribute values and $O(dN\log N)$ bits for the representation of the quadtrees. Thus, in total it requires $dN\log\ell+O(dN\log N)=dN\log\ell(1+O(\log N/\log\ell))=dN\log\ell(1+o(1))$, which is within the stated bound. Note that in both cases, the height of the quadtrees representing the relations is $O(\log N)$, and this is the multiplying factor in the time complexities.

4 Extending Worst-Case Optimality to More General Queries

In this section we turn to design worst-case optimal algorithms for more expressive queries. At this point it should be clear that we can deal with set operations: we already studied the intersection (which corresponds to operation And over the qdags), and will show that union (operation Or) and complement (operation Not) can be solved optimally as well. What is most intriguing, however, is whether we can obtain worst-case optimality on combined relational formulas. We introduce a worst-case optimal algorithm to evaluate formulas expressed as combinations of join, union, and complement operations (which we refer to as JUC-queries; note that intersection is a particular case of join). We do not study other operations like selection and projection because these are easily solved in time essentially proportional to the size of the output, but refer to Appendix A for more details on how projection interplays with the rest of our framework.

The key ingredient of our algorithm is to deal with these operations in a lazy form, allowing unknown intermediate results so that all components of a formula are evaluated simultaneously. To do this we introduce lazy qdags (or lqdags), an alternative to qdags that can navigate over the quadtree representing the output of a formula without the need to entirely evaluate the formula. We then give a worst-case optimal algorithm to compute the completion of an lqdag, that is, the quadtree of the grid represented by the lqdag.

4.1 Lqdags for relational formulas

To support worst-case optimal evaluation of relational formulas we introduce two new ideas: we add “full leaves” to the quadtree representation to denote subgrids full of 1s, and we introduce lqdags to represent the result of a formula as an implicit quadtree that can be navigated without fully evaluating the formula.

While quadtree leaves representing a single cell store the cell value, [math] or $1$, quadtree leaves at higher levels always represent subgrids full of [math]s. We now generalize the representation, so that quadtree leaves at any level store an integer, [math] or $1$, which is the value of all the cells in the subgrid represented by the leaf. The generalization impacts on the way to compute Value, depicted in Algorithm 6. We will not use qdags in this section, however; the lqdags build directly on quadtrees. In terms of the compact representation, this generalization is implemented by resorting to an impossible quadtree configuration: an internal node with all zero children [5]. Note that replacing a full subgrid with this configuration can only decrease the size of the representation.

The second novelty, the lqdags, are defined as follows.

Definition 4.1** (lqdag).**

An lqdag $L$ is a pair $(f,o)$, where $f$ is a functor and $o$ is a list of operands. The completion of $L$ is the quadtree $Q_{R}=Q_{R}(\mathcal{A})$ representing relation $R(\mathcal{A})$ if $L$ is as follows:

$(\texttt{{QTREE}},Q_{R})$, where the lqdag just represents $Q_{R}$; 2. 2.

$(\texttt{{NOT}},Q_{\overline{R}})$, where $Q_{\overline{R}}$ is the quadtree representing the complement of $Q_{R}$; 3. 3.

$(\texttt{{AND}},L_{1},L_{2})$, where $L_{1}$ and $L_{2}$ are lqdags and $Q_{R}$ represents the intersection of their completions; 4. 4.

$(\texttt{{OR}},L_{1},L_{2})$, where $L_{1}$ and $L_{2}$ are lqdags and $Q_{R}$ represents the union of their completions; 5. 5.

$(\texttt{{EXTEND}},L_{1},\mathcal{A})$, where lqdag $L_{1}$ represents $R^{\prime}(\mathcal{A}^{\prime})$, $\mathcal{A}^{\prime}\subseteq\mathcal{A}$, and $Q_{R}$ represents $R(\mathcal{A})=R^{\prime}(\mathcal{A}^{\prime})\times\text{All}(\mathcal{A}\setminus\mathcal{A}^{\prime})$.

Note that, for a quadtree $Q_{R}$ representing a relation $R(\mathcal{A}^{\prime})$, and a set of attributes $\mathcal{A}$, the qdag $Q_{R}^{*}=(Q_{R},M_{\mathcal{A}})$ that represents the relation $R\times\text{All}(\mathcal{A}\setminus\mathcal{A}^{\prime})$ can be expressed as the lqdag $(\texttt{{EXTEND}},(\texttt{{QTREE}},Q_{R}),\mathcal{A})$. In this sense, lqdags are extensions of qdags. To further illustrate the definition of lqdags, consider the triangle query $R(A,B)\bowtie S(B,C)\bowtie T(A,C)$, with $\mathcal{A}=\{A,B,C\}$ and the relations represented by quadtrees $Q_{R}$, $Q_{S}$, and $Q_{T}$. This query can then be represented as the lqdag

[TABLE]

It is apparent that one can define other operations, like JOIN and DIFF, by combining the operations defined above:

[TABLE]

Note that in the definition of the lqdag for NOT, the operand is a quadtree instead of an lqdag, and then, for example, $L_{2}$ should be a quadtree in the definition of DIFF, in principle. However, we can easily get around that restriction by pushing down the NOT operators until the operand is a quadtree or the NOT is cancelled with another NOT. For instance, a NOT over an lqdag $(\texttt{{AND}},Q_{1},Q_{2})$ is equivalent to $(\texttt{{OR}}$, $(\texttt{{NOT}},Q_{1})$, $(\texttt{{NOT}},Q_{2}))$, and analogously with the other functors. The restriction, however, does limit the types of formulas for which we achieve worst-case optimality, as shown later.

To understand why we call lqdags lazy, consider the operation $Q_{1}\ \text{\sc And}\ Q_{2}$ over quadtrees $Q_{1},Q_{2}$. If any of the values at the roots of $Q_{1}$ or $Q_{2}$ is [math], then the result of the operation is for sure a leaf with value [math]. If any of the values is $1$, then the result of the operation is the other. However, if both values are ½, one cannot be sure of the value of the root until the And between the children of $Q_{1}$ and $Q_{2}$ has been evaluated. Solving this dependency eagerly would go against worst-case optimality: it forces us to fully evaluate parts of the formula without considering it as a whole. To avoid this, we allow the Value of a node represented by an lqdag to be, apart from [math], $1$, and ½, the special value $\Diamond$. This indicates that one cannot determine the value of the node without computing the values of its children.

As we did for qdags, in order to simulate the navigation over the completion $Q$ of an lqdag $L$ we need to describe how to obtain the value of the root of $Q$, and how to obtain an lqdag whose completion is the $i$-th child of $Q$, for any given $i$. We implement those operations in Algorithms 7–14, all constant-time. Note that ChildAt can only be invoked when $\text{\sc Value}=$ ½ or $\Diamond$. The base case is $\text{\sc Value}(\texttt{{QTREE}},Q)=\text{\sc Value}(Q)$ and $\text{\sc ChildAt}((\texttt{{QTREE}},Q),i)=\text{\sc ChildAt}(Q,i)$, where we enter the quadtree and resort to the algorithms based on the compact representation of $Q$. We will assume that $\text{\sc Value}(Q)$ returns ½ for internal nodes, and thus the implementation of Value for EXTEND is trivial (compare Algorithms 6 and 13 under this assumption).

Note that the recursive calls of Algorithms 7-14 traverse the nodes of the relational formula (fnodes, for short), and terminate immediately upon reaching an fnode of the form $(\texttt{{QTREE}},Q)$. Therefore, their time complexity depends only on the size of the formula represented by the lqdag. We show next how, using these implementations of Value and ChildAt, one can efficiently evaluate a relational formula using lqdags.

4.2 Evaluating JUC queries

To evaluate a formula $F$ represented as an lqdag $L_{F}$, we compute the completion $Q_{F}$ of $L_{F}$, that is, the quadtree $Q_{F}$ representing the output of $F$.

To implement this we introduce the idea of super-completion of an lqdag. The super-completion $Q_{F}^{+}$ of $L_{F}$ is the quadtree induced by navigating $L_{F}$, and interpreting the values $\Diamond$ as ½ (see Algorithm 15). Note that, by interpreting values $\Diamond$ as ½, we are disregarding the possibility of pruning resulting subgrids full of [math]s or $1$s and replacing them by single leaves with values [math] or $1$ in $Q_{F}$. Therefore, $Q_{F}^{+}$ is a non-pruned quadtree (just as $Q^{+}$ in Section 3.3) that nevertheless represents the same points of $Q_{F}$. Moreover, $Q_{F}^{+}$ shares with $Q_{F}$ a key property: all its nodes with value $1$, including the last-level leaves representing individual cells, correspond to actual tuples in the output of $F$.

To see how lqdags are evaluated, let us consider the query $F=R(A,B)\bowtie S(B,C)\bowtie\overline{T}(A,C)$. This corresponds to an lqdag $Q_{F}$:

[TABLE]

Assuming some of the trees involved have internal nodes, the super-completion $Q_{F}^{+}$ first produces $8$ children. Suppose the grid of $T$ is full of $1$s in the first quadrant ($00$). Then the first child ($00$) of $Q_{T}$ has value $1$, which becomes value [math] in $(\texttt{{NOT}},Q_{T})$. This implies that $(\texttt{{EXTEND}},(\texttt{{NOT}},Q_{T}))$ also yields value [math] in octants $000$ and $010$. Thus, when function ChildAt is called on child $000$ of $Q_{F}$, our [math] is immediately propagated and ChildAt returns [math], meaning that there are no answers for $F$ on this octant, without ever consulting the quadtrees $Q_{R}$ and $Q_{S}$ (see Figure 3 for an illustration). On the other hand, if the value of the child $11$ of $T$ is [math], then $(\texttt{{EXTEND}},(\texttt{{NOT}},Q_{T}))$ will return value $1$ in octants $101$ and $111$. This means that the result on this octant corresponds to the result of joining $R$ and $S$; indeed ChildAt towards $101$ in $Q_{F}$ returns

[TABLE]

If $\text{\sc ChildAt}((\texttt{{EXTEND}},(\texttt{{QTREE}},Q_{R}),\mathcal{A}),101)$ and $\text{\sc ChildAt}((\texttt{{EXTEND}},(\texttt{{QTREE}},Q_{S}),\mathcal{A}),101)$ are trees with internal nodes, the resulting AND can be either an internal node or a leaf with value [math] (if the intersection is empty), though not a leaf with value $1$. Thus, for now, the Value of this node is unknown, a $\Diamond$. See Figure 3 for an illustration.

Note that the running time of Algorithm 15 is $O(|Q_{F}^{+}|)$. One can then compact $Q_{F}^{+}$ to obtain $Q_{F}$, in time $O(|Q_{F}^{+}|)$ as well, with a simple bottom-up traversal. Thus, bounding $|Q_{F}^{+}|$ yields a bound for the running time of evaluating $F$. While $|Q_{F}^{+}|$ can be considerably larger than the actual size $|Q_{F}|$ of the output, we show that $|Q_{F}^{+}|$ is bounded by the worst-case output size of formula $F$ for a database with relations of approximately the same size. To prove this, the introduction of values $\Diamond$ plays a key role.444In an implementation, we could simply use ½ instead of $\Diamond$, without indicating that we are not yet sure that the value is ½: we build $Q_{F}^{+}$ assuming it is, and only make sure later, when we compact it into $Q_{F}$.

The power of the $\Diamond$ values. Consider again Algorithm 15. The lowest places in $L_{F}$ where $\Diamond$ values are introduced are the Value of AND and OR lqdags where both operands have $\text{\sc Value}=$ ½. We must then set the Value to $\Diamond$ instead of ½ because, depending on the evaluation of the children of the operands, the Value can turn out to be actually [math] for AND or $1$ for OR. Once produced, a value $\Diamond$ is inherited by the ancestors in the formula unless the other value is [math] (for AND) or $1$ (for OR).

Imagine that a formula $F$ involves $n$ relations $R_{1},\ldots,R_{n}$ represented as quadtrees in dimension $d$, including no negations. Suppose that we trim the quadtrees of $R_{1},\ldots,R_{n}$ by removing all the levels at depth higher than some $j$ (thus making the $j$-th level the last one) and assuming that the internal nodes at level $j$ become leaves with value $1$. We do not attempt to compact the resulting quadtrees, so their nodes at levels up to $j-1$ stay identical and with the same Value. If we now compute $Q_{F}^{+}$ over those (possibly non-pruned) quadtrees, the computation will be identical up to level $j-1$, and in level $j$ every internal node in the original $Q_{F}^{+}$, which had value ½, will now operate over all $1$s, and thus will evaluate to $1$ because And and Or are monotonic.

Thus, these nodes belong to the output of $F$ over the relations $R_{1}^{\prime},\ldots,R_{n}^{\prime}$ induced by the trimmed quadtrees (on smaller domains of size $\ell^{\prime}=2^{j}$), with sizes $|R_{1}|^{\prime}\leq|R_{1}|,\ldots,\!|R_{n}|^{\prime}\leq|R_{n}|$. This would imply, just as in the proof of Theorem 3.8, a bound on the maximum number of nodes in a level of $Q_{F}^{+}$, thus proving the worst-case optimality of the size of $Q_{F}^{+}$ (up to $\log min(\ell,N)$-factors, and factors depending on the query size), and thus the worst-case optimality of Algorithm 15 in data complexity.

However, this reasoning fails when one trims at the $j$-th level a quadtree $Q$ that appears in an lqdag $L=(\texttt{{NOT}},Q)$, because the value $1$ of the nodes at the $j$-th level of $Q$ after the trimming changes to [math] in $L$. So, to prove that our algorithm is worst-case optimal we cannot rely only on relations obtained by trimming those that appear in the formula. We need to generate new quadtrees for those relations under a NOT operation that preserve the values of the completion of NOT after the trimming. Next we formalize how to do this.

Analysis of the algorithm. Let $L_{F}$ be an lqdag for a formula $F$. The syntax tree of $F$ is the directed tree formed by the fnodes in $F$, with an edge from fnode $L$ to fnode $L^{\prime}$ if $L^{\prime}$ is an operand of $L$. The leaves of this tree are always atomic expressions, that is, the fnodes, with functors QTREE and NOT, that operate on one quadtree (see Figure 3 again). We say that two atomic expressions $L_{1}$ and $L_{2}$ are equal if both their functors and operands are equal. For example, in the formula

[TABLE]

there are three different atomic expressions, $(\texttt{{QTREE}},Q_{R})$, $(\texttt{{QTREE}},Q_{S})$, and $(\texttt{{QTREE}},Q_{T})$, while in $F^{\prime}=(\texttt{{AND}},(\texttt{{QTREE}},Q_{R}),(\texttt{{NOT}},Q_{R}))$ there are two atomic expressions. Notice that in formulas like $F^{\prime}$, where a relation appears both negated and not negated, the two occurrences are seen as different atomic expressions. We return later to the consequences of this definition.

The following lemma is key to bound the running time of Algorithm 15 while evaluating a formula $F$.

Lemma 4.2**.**

Let $F$ be a relational formula represented by an lqdag $L_{F}$ in dimension $d$, and let $Q_{F}^{+}$ be the super-completion of $F$. Let $Q_{1},\ldots,Q_{n}$ be the quadtree operands of the different atomic expressions of $F$, and $R_{1}(\mathcal{A}_{1}),\ldots,R_{n}(\mathcal{A}_{n})$ be the (not necessarily different) relations represented by these quadtrees, respectively. Let $M$ be the maximum number of nodes in a level of $Q_{F}^{+}$. Then, there is a database with relations $R_{1}^{\prime}(\mathcal{A}_{1}),\ldots,R_{n}^{\prime}(\mathcal{A}_{n})$ of respective sizes $O(2^{d}|Q_{1}|),\ldots,O(2^{d}|Q_{n}|)$, such that the output of $F$ evaluated over it has size $\Omega(M/2^{d})$.

Proof 4.3**.**

Let $m_{l}$ be the number of nodes in level $l$ of $Q_{F}^{+}$ and $j$ be a level where $M=m_{j}$ is maximum. We assume that $j>1$, otherwise $M=O(1)$ and the result is trivial. We first bound the number of nodes with value ½ at the $(j-1)$-th level. By hypothesis, $m_{j}\geq m_{j-1}$, and since a node in $Q_{F}^{+}$ is present at level $j$ only if its parent at level $j-1$ has value ½, in the $(j-1)$-th level there are at least $m_{j}/2^{d}$ nodes with value ½.

Now, let $A_{1},\ldots,A_{n}$ be the atomic expressions of $F$, and let $Q_{1}^{\prime},\ldots,Q_{n}^{\prime}$ be the quadtrees that result from trimming the levels at depths higher than $j-1$ from $Q_{1},\ldots,Q_{n}$, respectively. Consider the completion $A_{i}^{*}$ of $A_{i}$ evaluated over $Q_{i}$, and the completion ${A_{i}^{*}}^{\prime}$ of $A_{i}$ evaluated over (the possibly non-pruned) $Q_{i}^{\prime}$, for all $1\leq i\leq n$. If it is always the case that the first $j-1$ levels of $A_{i}^{*}$ are respectively equal to the $j-1$ levels of ${A_{i}^{*}}^{\prime}$ then we are done. To see why, let ${Q_{F}^{+}}^{\prime}$ be the super-completion of $F$ when evaluated over $Q_{1}^{\prime},\ldots,Q_{n}^{\prime}$. The first $j-2$ levels of $Q_{F}^{+}$ will be the same as those of ${Q_{F}^{+}}^{\prime}$ because the same results of the operations are propagated up from the leaves of the syntax tree of $F$ before and after the trimming. Moreover, in the $(j-1)$-th level ${Q_{F}^{+}}^{\prime}$ (its last level) the nodes with value $1$ are precisely the nodes with value $1$ or ½ in ${Q_{F}^{+}}$, where we note that: (i) there are at least $m_{j}/2^{d}$ of them; and (2) they belong to the output of $F$ over the relations $R_{1}^{\prime},\ldots,R_{n}^{\prime}$ represented by $Q_{1}^{\prime},\ldots,Q_{n}^{\prime}$.

We know that $|R_{1}^{\prime}|\leq|R_{1}|,\ldots,|R_{n}^{\prime}|\leq|R_{n}|$. However, the values of $R_{1}^{\prime},\ldots,R_{n}^{\prime}$ correspond to a smaller universe. This can be remedied by simply appending $(\log\ell-j)$ 0’s at the beginning of the binary representation of these values. This would yield the desired result: we have $n$ relations over the same set of attributes as the original ones, with same respective cardinality, and such that when $F$ is evaluated over them the output size is $\Omega(m_{j}/2^{d})$.

However, for atomic expressions of the type $A_{i}=(\texttt{{NOT}},Q_{i})$ it is not the case that the first $j-1$ levels of ${A_{i}^{*}}$ coincide with the $j-1$ levels of ${A_{i}^{*}}^{\prime}$. Anyway, we can deal with this case: their first $j-2$ levels will coincide, and in the last level, the value of a node present in ${A_{i}^{*}}$ is the negation of the value of the homologous node in ${A_{i}^{*}}^{\prime}$. Thus, instead of choosing the quadtree $Q_{i}^{\prime}$ that results from trimming $Q_{i}$, we choose the quadtree $Q_{i}^{\prime\prime}$ in which the first $j-2$ levels are the same as $Q_{i}^{\prime}$, and the $(j-1)$-th level results from negating the value of every node in $Q_{i}^{\prime}$. Note that if we let now ${A_{i}^{*}}^{\prime}$ be the completion of $A_{i}$ evaluated over $Q_{i}^{\prime\prime}$, then the first $j-1$ levels of ${A_{i}^{*}}$ will be exactly same as the $j-1$ levels of ${A_{i}^{*}}^{\prime}$. Finally, note that the size of the relation represented by $Q_{i}^{\prime\prime}$ cannot be larger than $2^{d}|Q_{i}|$. The result of the lemma follows.

Using the same reasoning as before we can bound the time needed to compute the super-completion $Q_{F}^{+}$ of an lqdag $L_{F}$ in dimension $d$ involving quadtrees representing $R_{1},\ldots,R_{n}$. Since $M$ is the maximum number of nodes in a level of $Q_{F}^{+}$, the number of nodes in $Q_{F}^{+}$ is at most $M\log\ell$. Now, each node in $Q_{F}^{+}$ results from the application of $|F|$ operations on each of the $2^{d}$ children being generated, all of which take constant time. Thus the super-completion can be computed in time $O(M\cdot 2^{d}|F|\log\ell)$. If we use $F(D)^{*}$ to denote the size of the maximum output of the query $F$ over instances with relations $R_{1},\ldots,R_{n}$ of respective sizes $O(2^{d}|Q_{1}|),\ldots,O(2^{d}|Q_{n}|)$, then by Lemma 4.2 the query $F$ can be computed in time $O(F(D)^{*}\cdot 2^{2d}|F|\log\ell)$. This means that the algorithm is indeed worst-case optimal.

The requirement of different atomic expressions is because we need to consider $R$ and $\text{{\sc Not}}\ R$ as different relations. To see this, consider again our example formula $F^{\prime}=(\texttt{{AND}},(\texttt{{QTREE}},Q_{R}),(\texttt{{NOT}},Q_{R}))$. We clearly have that the answer of this query is always empty, and therefore $|Q_{F^{\prime}}|=0$. However, here $|Q^{+}_{F^{\prime}}|=\Theta(|R|)$ for every $R$, and thus our algorithm is worst-case optimal only if we consider the possible output size of a more general formula, $F^{\prime\prime}=(\texttt{{AND}},(\texttt{{QTREE}},Q_{R}),(\texttt{{NOT}},Q_{R}^{\prime}))$. This impacts in other operations of the relational algebra. We can write all of them as lqdags, but for some of them we will not ensure their optimal evaluation. For instance, the expression $Q_{R}\ \text{\sc And}\ (\text{{\sc Not}}\ (Q_{R}\ \text{\sc And}\ Q_{S}))$, which expresses the antijoin between $R$ and $S$, is not optimal because both $Q_{R}$ and $\text{{\sc Not}}\ Q_{R}$ appear in the formula. A way to ensure that our result applies is to require that the atomic expressions (once the NOT operations are pushed down) refer all to different relations.

Theorem 4.4**.**

Let $F$ be a relational formula represented by an lqdag $L_{F}$. If the number of different relations involved in $F$ equals the number of different atomic expressions, then Algorithm 15 evaluates $F$ in worst-case optimal time in data complexity.

Note that this result generalizes Theorem 3.8. Moreover, it does not matter how we write our formula $F$ to achieve worst-case optimal evaluation. For example, our algorithms behave identically on $((R\bowtie S)\bowtie T)$ and on $(R\bowtie(S\bowtie T))$.

5 Final Remarks

The evaluation of join queries using qdags provides a competitive alternative to current worst-case optimal algorithms [9, 11, 15, 17, 21]. When compared to them, we find the following time-space tradeoffs.

Regarding space, qdags require only just a few extra words per tuple, which is generally much less than what standard database indexes require, and definitely less than those required by current worst-case optimal algorithms (e.g. [9, 17, 21]). Moreover, in both NPRR [17] and leapfrog [21], the required index structure only works for a specific ordering of the attributes. Thus, in order to efficiently evaluate any possible query using these two algorithms, a separate index is required for every possible attribute order (i.e., $d!$ indexes). In contrast, all we need to store is one quadtree per relation, and that works for any query. Even if we resort to the (simpler) $k^{d}$-tree representation by Brisaboa et al. [4], the extra space increases by a factor of $2^{d}$ bits, which is still considerably less than the alternative of $d!$ standard indexes for any order of variables (e.g., for $d=10$, $2^{d}=1024$, while $d!=3628800$, i.e., $\approx 3500$ times bigger).

Regarding time, the first comparison that stands aside is the $\log(N)$ factor, present in our solution but not in others like NPRR [17] and leapfrog [21]. Note, however, that NPRR assumes to be able to compute a join of two relations $R$ and $S$ in time $O(|R|+|S|+|R\bowtie S|)$, which is only possible when using a hash table and when time is computed in an amortized way or in expectation [17, footnote 3]. This was also noted for leapfrog [21, Section 5], where they state that their own $\log(N)$ factor can be avoided by using hashes instead of tries, but they leave open whether this is actually better in practice. More involved algorithms such as PANDA [11] build upon algorithms to compute joins of two relations, and therefore the same $\log(N)$ factor appears if one avoids hashes or amortized running time bounds. Our algorithm incurs in an additional $2^{d}$ factor in time when compared to NPRR or leapfrog, similarly to other worst-case optimal solutions based on geometric data structures [9, 15]. This is, as far as we are aware, unavoidable: it is the price to pay for using so little space. Note, however, that this factor does not depend on the data, and that it can be compensated by the fact that our native indexes are compressed, and thus might fit entirely in faster memory.

One important benefit of our framework is that answers to queries can be delivered in their compact representation. As such, we can iterate over them, or store them, or use them as materialized views, either built eagerly, as quadtrees, or in lazy form, as lqdags. One could even cache the top half of the (uncompacted) tree containing the answer, and leave the bottom half in the form of lqdags. The upper half, which is used the most, is cached, and the bottom half is computed on demand. Our framework also permits sharing lqdags as common subexpressions that are computed only once. Additionally, we envision two main uses for the techniques presented in this paper. On one hand, one could take advantage of the low storage cost of these indexes, and add them as a companion to a more traditional database setting. Smaller joins and selections could be handled by the database, while multijoins could be processed faster because they would be computed over the quadtrees. On the other hand, we could use lqdags instead, so as to evaluate more expressive queries over quadtrees. Even if some operations are not optimal, what is lost in optimality may be gained again because these data structures allow operating in faster memory levels.

There are several directions for future work. For instance, we are trying to improve our structures to achieve good bounds for acyclic queries (see Appendix A), and we see an opportunity to apply quadtrees in the setting of parallel computation (see, e.g., Suciu [20]). We also comment in Appendix A on bounds for clustered databases, another topic deserving further study.

Appendix A Appendix

A.1 Additional comments on Projections

Including projection in our framework is not difficult: in a quadtree $Q$ storing a relation $R$ with attributes $\mathcal{A}$, one can compute the projection $\pi_{\mathcal{A}^{\prime}}(R)$, for $\mathcal{A}^{\prime}\subseteq\mathcal{A}$ as follows. Assume that $|\mathcal{A}|=d$ and $|\mathcal{A}^{\prime}|=d^{\prime}$. Then the projection is the quadtree defined inductively as follows. If $\text{\sc Value}(Q)$ is [math] or $1$ then the projection is a leaf with the same value. Otherwise $Q$ has $2^{d}$ children. The quadtree for $\pi_{\mathcal{A}^{\prime}}(R)$ has instead $2^{d^{\prime}}$ children, where the $i$-th child is defined as the Or of all children $j$ of $Q$ such that the projection of the $d$-bit representation of $j$ to the positions in which attributes in $\mathcal{A}^{\prime}$ appear in $\mathcal{A}$ is precisely the $d^{\prime}$-bit representation of $i$. For example, computing $\pi_{A_{1},A_{2}}R(A_{1},A_{2},A_{3})$ means creating a tree with four children, resulting of the Or of children [math] and $1$, $2$ and $3$, $4$ and $5$ and $6$ and $7$, respectively.

Having defined the projection, a natural question is whether one can use it to obtain finer bounds for acyclic queries or for queries with bounded treewidth. For example, even though the AGM bound for $R(A,B)\bowtie S(B,C)$ is quadratic, one can use Yannakakis’ algorithm [23] to compute it in time $O(|R|+|S|+|R\bowtie S|)$. This is commonly achieved by first computing $\pi_{B}(R)$ and $\pi_{B}(S)$, joining them, and then using this join to filter out $R$ and $S$. Unfortunately, adopting this strategy in our lqdag framework would still give us a quadratic algorithm, even for queries with small output, because after the projection we would need to extend the result again. The same holds for the general Yannakakis’ algorithm when computing the final join after performing all necessary semijoins.

More generally, this also rules out the possibility to achieve optimal bounds for queries with bounded treewidth or similar measures. Of course, this is not much of a limitation because one can always compute the most complex queries with our compact representation and then carry out Yannakakis’ algorithms on top of these results with standard database techniques, but it would be better to resolve all within our framework. We are currently looking at improving our data structures in this regard.

A.2 Geometric representation and finer analysis

As quadtrees have a direct geometric interpretation, it is natural to compare them to the algorithm based on gap boxes proposed by Khamis et al. [9]. In a nutshell, this algorithm uses a data structure that stores relations as a set of multidimensional cubes that contain no data points, which the authors call gap boxes. Under this framework, a data point is in the answer of the join query $R_{1}\bowtie\cdots\bowtie R_{n}$ if the point is not part of a gap box in any of the relations $R_{i}$. The authors then compute the answers of these queries using an algorithm that finds and merges appropriate gap boxes covering all cells not in the answer of the query, until no more gap boxes can be found and we are left with a covering that misses exactly those points in the answer of the query. Perhaps more interestingly, the algorithm is subject of a finer analysis: the runtime of queries can be shown to be bounded by a function of the size of a certificate of the instance (and not its size). The certificate in their case is simply the minimum amount of gap boxes from the input relations that is needed to cover all the gaps in the answer of the query. Finding such a minimal cover is NP-hard, but a slightly restricted notion of gap boxes maintains the bounds within an $O(\log^{d}\ell)$ approximation factor.

While any index structure can be thought of as providing a set of gap boxes [9], quadtrees provide a particularly natural and compact representation. Each node valued [math] in a quadtree signals that there are no points in its subgrid, and can therefore be understood as a $d$-dimensional gap box. We can understand qdags as a set of gap boxes as well: precisely those in its completion. Now let $J=R_{1}\bowtie\cdots\bowtie R_{n}$ be a join query over $d$ attributes, and let $R_{1}^{*},\dots,R_{n}^{*}$ denote the extension of each $R_{i}$ with the attributes of $J$ that are not in $R_{i}$. As in Khamis et al. [9], a quadtree certificate for $J$ is a set of gap boxes (i.e., empty $d$-dimensional grids obtained from any of the $R_{i}^{*}$s) such that every coordinate not in the answer of $J$ is covered by at least one of these boxes. Let $C_{J,D}$ denote a certificate for $J$ of minimum size.

Proposition A.1**.**

Given query $J$ and database $D$, Algorithm 4 runs in time $O((|C_{J,D}|+|J(D)|)\cdot 2^{d}n\log\ell)$, where $J(D)$ is the output of the query $J$ over $D$.

Now, one can easily construct instances and queries such that the minimal certificate $C_{J,D}$ is comparable to $2^{\rho^{*}(J,D)}$. So this will not give us optimality results, as discovered [9, 15] for acyclic queries or queries with bounded treewidth. This is a consequence of increasing the dimensionality of the relations. Nevertheless, the bound does yield a good running time when we know that $C_{J,D}$ is small. It is also worth mentioning that our algorithms directly computes the only possible representation of the output as gap boxes (because its boxes come directly from the representation of the relations). This means that there is a direct connection between instances that give small certificates and instances for which the representation of the output is small.

A.3 Better runtime on clustered databases

Quadtrees have been shown to work well in applications such as RDF stores or web graphs, where data points are distributed in clusters [4, 1]. It turns out that combining the analysis described in Section 2 for clustered grids with the technique we used to show that joins are worst-case optimal, results in a better bound for the running time of our algorithms, and a small refinement of the AGM bound itself.

Consider again the triangle query $R(A,B)\bowtie S(B,C)\bowtie T(A,C)$, and assume the points in each relation are distributed in $c$ clusters, each of them of size at most $s\times s$, and with $p$ points in total. Then, at depth $\log(\ell/s)$, the quadtrees of $T$, $R$, and $S$ have at most $2^{d}$ internal nodes per cluster (where we are in dimension $d=3$): at this level one can think of the trimmed quadtree as representing a coarser grid of cells of size $s^{d}$, and therefore each cluster can intersect at most two of these coarser cells per dimension. Thus, letting $Q_{R}^{\prime}$, $Q_{S}^{\prime}$, and $Q_{T}^{\prime}$ be the quadtrees for $R$, $S$ and $T$ trimmed up to level $\log(\ell/s)$ (and where internal nodes take value $1$), then the proof of Theorem 3.8 yields a bound for the number of internal nodes at level $\log(\ell/s)$ of the quadtree $Q^{+}$ of the output before the compaction step (or, equivalently, of the super-completion of the lqdag of the triangle query): this number must be bounded by the AGM bound of the instances given by $Q_{R}^{\prime}$, $Q_{S}^{\prime}$ and $Q_{T}^{\prime}$, which is at most $(c\cdot 2^{d})^{3/2}$. Going back to the data for the quadtree $Q^{+}$, the bound on the number of internal nodes means that the points of the output are distributed in at most $(c\cdot 2^{d})^{3/2}$ clusters of size at most $s^{d}$. In turn, the maximal number of $1$s in the answer is bounded by the AGM bound itself, which here is $p^{3/2}$. This means that the size of $Q^{+}$ is bounded by $O((c\cdot 2^{d})^{3/2}\log\ell+p^{3/2}\log s)$, and therefore the running time of the algorithm is O\big{(}((c\cdot 2^{d})^{3/2}\log\ell+p^{3/2}\log s)\cdot 2^{d}\big{)}. This is an important reduction in running time if the number $c$ of clusters and their width $s$ are small, as we now multiply the number of answers by $\log s$ instead of $\log\ell$.

To generalize, let us use $D^{c,d}$ as the database “trimmed” to $c\cdot 2^{d}$ points. The discussion above can be extended to prove the following.

Proposition A.2**.**

Let $J=R_{1}\bowtie\cdots\bowtie R_{n}$ be a full join query, and $D$ a database over schema $\{R_{1},\ldots,R_{n}\}$, with $d$ attributes in total, where the domains of the relations are in $[0,\ell-1]$, and where the points in each relation are distributed in $c$ clusters of width $s$. Then Algorithm 4 works in time O\big{(}(2^{\rho^{*}(J,D^{c,d})}\log\ell+2^{\rho^{*}(J,D)}\log s)\cdot 2^{d}n).