Depth Distribution in High Dimensions

TL;DR

This paper introduces an algorithm to compute the Depth Distribution of axis-aligned boxes in high dimensions, generalizing existing measures like Klee's Measure and Maximum Depth, with improved complexity bounds.

Contribution

It presents a new algorithm for depth distribution computation in high-dimensional spaces, extending previous methods for Klee's Measure and Maximum Depth.

Findings

Algorithm complexity within O(n^{(d+1)/2} log n)

Space complexity within O(n log n)

Refined results based on input difficulty measures

Abstract

Motivated by the analysis of range queries in databases, we introduce the computation of the Depth Distribution of a set $\mathcal{B}$ of axis aligned boxes, whose computation generalizes that of the Klee's Measure and of the Maximum Depth. In the worst case over instances of fixed input size $n$, we describe an algorithm of complexity within $O({n^\frac{d+1}{2}\log n})$, using space within $O({n\log n})$, mixing two techniques previously used to compute the Klee's Measure. We refine this result and previous results on the Klee's Measure and the Maximum Depth for various measures of difficulty of the input, such as the profile of the input and the degeneracy of the intersection graph formed by the boxes.

Equations

$$V_k=\sum _{\begin{array}{c}0\le j_1,\dots ,j_d\le n\\ j_1+\dots +j_d=k\end{array}}\left(\prod _{i=1}^d{V}_{j_i}^i\right),$$

$$T_I(n)$$

$$\subseteq \mathcal{O}(n\sum _{i=1}^h{2}^{i(d-1)/d})$$

$$\subseteq \mathcal{O}(n{2}^{\frac{d-1}{d}h})$$

$$\subseteq \mathcal{O}(n{2}^{\frac{d-1}{d}\frac{d}{2}\log n})$$

$$=\mathcal{O}(n^{\frac{d+1}{2}})$$

Full text

1 Introduction

Problems studied in Computational Geometry have found important applications in the processing and querying of massive databases [1], such as the computation of the Maxima of a set of points [2, 4], or compressed data structures for Point Location and Rectangle Stabbing [3]. In particular, we consider cases where the input or queries are composed of axis-aligned boxes in $d$ dimensions: in the context of databases it corresponds for instance to a search for cars within the intersection of ranges in price, availability and security ratings range.

Consider a set $\mathcal{B}$ of $n$ axis-parallel boxes in $\mathbb{R}^{d}$, for fixed $d$. {LONG} We focus on two measures on such set of boxes: the Klee’s measure and the Maximum Depth.

The Klee’s Measure of $\mathcal{B}$ is {LONG} the size of the “shadow” projected by $\mathcal{B}$, more formally

the volume of the union of the boxes in $\mathcal{B}$. Originally suggested on the line by Klee [18], its computation is well studied in higher dimensions [7, 8, 9, 10, 22, 26, 25, 24], and can be done in time within $\mathcal{O}(n^{d/2})$, using an algorithm introduced by Chan [10] based on a new paradigm called “Simplify, Divide and Conquer”. The Maximum Depth of $\mathcal{B}$ is the maximum number of boxes covering a same point, and its computational complexity is similar to that of Klee’s Measure’s, converging to the same complexity within $\mathcal{O}(n^{d/2})$ [10].

Hypothesis. The known algorithms to compute these two measures are all strikingly similar{LONG}, to the point that Chan [10] states that all known techniques used for computing the Klee’s Measure can be applied to the computation of the Maximum Depth. That would suggest a reduction from one to the other, but those two measures are completely distinct: the Klee’s measure is a volume whose value can be a real number, while the Maximum Depth is a cardinality whose value is an integer in the range $[1..n]$. Is there any way to formalize the close relationship between the computation of these two measures?

Our Results. We describe a first step towards such a formalization, in the form of a new problem, which we show to be intermediary in terms of the techniques being used, between the Klee’s Measure and the Maximum Depth, slightly more costly in time and space, and with interesting applications and results of its own. We introduce the notion of Depth Distribution of a set $\mathcal{B}$ of $n$ axis-parallel boxes in $\mathbb{R}^{d}$, formed by the vector of $n$ values $(V_{1},\ldots,V_{n})$, where $V_{i}$ corresponds to the volume covered by exactly $i$ boxes from $\cal B$. The Depth Distribution of a set $\mathcal{B}$ can be interpreted as a probability distribution function (hence the name): if a point $p$ is selected uniformly at random from the region covered by the boxes in $\mathcal{B}$, the probability that $p$ hits exactly $k$ boxes from $\mathcal{B}$ is $\left({V_{k}}\middle/{\sum_{i=1}^{n}{V_{i}}}\right)$, for all $k\in[1..n]$.

The Depth Distribution refines both the Klee’s Measure and the Maximum Depth. It is a measure finer than the Klee’s Measure in the sense that the Klee’s Measure of a set $\cal B$ can be obtained in time linear in the size $n$ of $\cal B$ by summing the components of the Depth Distribution of $\cal B$. Similarly, the Depth Distribution is a measure finer than the Maximum Depth in the sense that the Maximum Depth of a set $\cal B$ can be obtained in linear time by finding the largest $i\in[1..n]$ such that $V_{i}\neq 0$. In the context of a database, when receiving multidimensional range queries (e.g. about cars), the Depth Distribution of the queries yields valuable information to the database owner (e.g. a car dealer) about the repartition of the queries in the space of the data, to allow for informed decisions on it (e.g. to orient the future purchase of cars to resell based on the clients’ desires, as expressed by their queries).

In the classical computational complexity model where one studies the worst case over instances of fixed size $n$, {LONG} the trivial approach of partitioning the space in cells that are completely contained within all the boxes they intersect, results in a solution with prohibitive running time (within $\mathcal{O}(n^{d+1})$). Simple variants of the techniques previously used to compute the Klee’s Measure [10, 22] results in a solution running in time within $\mathcal{O}(n^{d/2+1})$, using linear space, or a solution running in time within $\mathcal{O}(n^{(d/2+1/2}\log n)$, but using space within $\mathcal{O}(n^{d/2}\log n)$. We combine those two into a single technique to compute the Depth Distribution in time within $\mathcal{O}(n^{\frac{d+1}{2}}\log n)$, using space within $\mathcal{O}(n\log n)$ (in Section 3.1)

{SHORT} we combine techniques previously used to compute the Klee’s Measure [10, 22] to compute the Depth Distribution in time within $\mathcal{O}(n^{\frac{d+1}{2}}\log n)$, using space within $\mathcal{O}(n\log n)$ (in Section 3.1).

This solution is slower by a factor within $\mathcal{O}(\sqrt{n}\log n)$ than the best known algorithms for computing the Klee’s Measure and the Maximum Depth: we show in Section 3.2 that such a gap might be ineluctable, via a reduction from the computation of Matrix Multiplication.

In the refined computational complexity model where one studies the worst case complexity taking advantage of additional parameters describing the difficulty of the instance [4, 17, 21], we consider (in Section 4) distinct measures of difficulty for the instances of these problems, such as the profile and the degeneracy of the intersection graph of the boxes, and describe algorithms in these model to compute the Depth Distribution, the Klee’s Measure and the Maximum Depth of a set $\cal B$. {INUTILE} Javiel: can we use those results on polynome multiplication too?

After a short overview of the known results on the computation of the Klee’s Measure and the Maximum Depth (in Section 2), we describe in Section 3 the results in the worst case over instances of fixed size. In Section 4, we describe results on refined partitions of the instance universe, both for the computation of the Depth Distribution, and for the previously known problems of computing the Klee’s Measure and the Maximum Depth. We conclude in Section 5 with a discussion on discrete variants and further refinements of the analysis.

2 Background

The techniques used to compute the Klee’s Measure have evolved over time, and can all be used to compute the Maximum Depth. We retrace some of the main results, which will be useful for the definition of an algorithm computing the Depth Distribution (in Section 3), and for the refinements of the analysis for Depth Distribution, Klee’s Measure and Maximum Depth (in Section 4).

The computation of the Klee’s Measure of a set $\mathcal{B}$ of $n$ axis-aligned $d$-dimensional boxes was first posed by Klee [18] in 1977. After some initial progresses [6, 15, 18], Overmars and Yap [22] described a solution running in time within $\mathcal{O}(n^{d/2}\log n)$. This remained the best solution for more than 20 years until 2013, when Chan [10] presented a simpler and faster algorithm running in time within $\mathcal{O}(n^{d/2})$.

The algorithms described by Overmars and Yap [22] and by Chan [10], respectively, take both advantage of solutions to the special case of the problem where all the boxes are slabs. A box $b$ is said to be a slab within another box $\Gamma$ if $b\cap\Gamma=\{(x_{1},\ldots,x_{d})\in\Gamma\mid\alpha\leq x_{i}\leq\beta\}$, for some integer $i\in[1..d]$ and some real values $\alpha,\beta$ (see Figure 1 for an illustration). Overmars and Yap [22] showed that, if all the boxes in $\mathcal{B}$ are slabs inside the domain box $\Gamma$, then the Klee’s Measure of $\mathcal{B}$ within $\Gamma$ can be computed in linear time (provided that the boxes have been pre-sorted in each dimension).

Overmars and Yap’s algorithm [22] is based on a technique originally described by Bentley [6]: solve the static problem in $d$ dimensions by combining a data structure for the dynamic version of the problem in $d-1$ dimensions with a plane sweep over the $d$-th dimension. The algorithm starts by partitioning the space into $\mathcal{O}(n^{d/2})$ rectangular cells such that the boxes in $\mathcal{B}$ are equivalent to slabs when restricted to each of those cells. Then, the algorithm builds a tree-like data structure whose leaves are the cells of the partition, supporting insertion and deletion of boxes while keeping track of the Klee’s Measure of the boxes.

Chan’s algorithm [10] is a simpler divide-and-conquer algorithm, where the slabs are simplified and removed from the input before the recursive calls (Chan [10] named this technique Simplify, Divide and Conquer, SDC for short). To obtain the recursive subproblems, the algorithm assigns a constant weight of $2^{\frac{i+j}{2}}$ to each ($d$-2)-face intersecting the domain and orthogonal to the $i$-th and $j$-th dimensions, $i,j\in[1..d]$. Then, the domain is partitioned into two sub-domains by the hyperplane $x_{1}=m$, where $m$ is the weighted median of the ($d$-2)-faces orthogonal to the first dimension. This yields a decrease by a factor of $2^{2/d}$ in the total weight of the ($d$-2)-faces intersecting each sub-domain. Chan [10] uses this, and the fact that slabs have no ($d$-2)-face intersecting the domain, to prove that the SDC algorithm runs in time within $\mathcal{O}(n^{d/2})$.

Unfortunately, there are sets of boxes which require partitions of the space into a number of cells within $\Omega(n^{d/2})$ to ensure that, when restricted to each cell, all the boxes are equivalent to slabs. Hence, without a radically new technique, any algorithm based on this approach will require running time within $\Omega(n^{d/2})$. Chan [10] conjectured that any combinatorial algorithm computing the Klee’s Measure requires within $\Omega(n^{d/2})$ operations, via a reduction from the parameterized k-Clique problem, in the worst case over instances of fixed size $n$. As a consequence, recent work have focused on the study of special cases which can be solved faster than $\Omega(n^{d/2})$, like for instance when all the boxes are orthants [10], $\alpha$-fat boxes [7], or cubes [8]. In turn, we show in Section 4 that there are measures which gradually separate easy instances for these problems from the hard ones.

In the next section, we present an algorithm for the computation of the Depth Distribution inspired by a combination of the approaches described above, outperforming naive applications of those techniques.

3 Computing the Depth Distribution

We describe in Section 3.1 an algorithm to compute the Depth Distribution of a set of $n$ boxes. The running time of this algorithm in the worst case over $d$-dimensional instances of fixed size $n$ is within $\mathcal{O}(n^{\frac{d+1}{2}}\log n)$, using space within $\mathcal{O}(n\log n)$. This running time is worse than that of computing only the Klee’s Measure (or the Maximum Depth) by a factor within $\mathcal{O}(\sqrt{n}\log n)$: we argue in Section 3.2 that computing the Depth Distribution is computationally harder than the special cases of computing the Klee’s Measure and the Maximum Depth, unless computing Matrix Multiplication is much easier than usually assumed.

3.1 Upper bound

We introduce an algorithm to compute the Depth Distribution inspired by a combination of the techniques introduced by Chan [10], and by Overmars and Yap [22], for the computation of the Klee’s Measure (described in Section 2). As in those approaches, the algorithm partitions the domain $\Gamma$ into $\mathcal{O}(n^{d/2})$ cells where the boxes of $\mathcal{B}$ are equivalent to slabs, and then combines the solution within each cell to obtain the final answer. Two main issues must be addressed: how to compute the Depth Distribution when the boxes are slabs, and how to partition the domain efficiently.

We address first the special case of slabs. We show in Lemma 1 that computing the Depth Distribution of a set of $n$ $d$-dimensional slabs within a domain $\Gamma$ can be done via a multiplication of $d$ polynomials of degree at most $n$.

Lemma 1

Let $\mathcal{B}$ be a set of $n$ axis-parallel $d$-dimensional axis-aligned boxes whose intersection with a domain box $\Gamma$ are slabs. The computation of the Depth Distribution $\left(V_{1},\ldots,V_{n}\right)$ of $\mathcal{B}$ within $\Gamma$ can be performed via a multiplication of $d$ polynomials of degree at most $n$.

Proof

For all $i\in[1..d]$, let $\mathcal{B}_{i}$ be the subset of slabs that are orthogonal to the $i$-th dimension, and let $\left(V^{i}_{1},\ldots,V^{i}_{n}\right)$ be the Depth Distribution of the intervals that result from projecting $\mathcal{B}_{i}$ to the $i$-th dimension within $\Gamma$. We associate a polynomial $P_{i}(x)$ of degree $n$ with each $\mathcal{B}_{i}$ as follows:

• •

let $\Gamma_{i}$ be the projection of the domain $\Gamma$ into the $i$-th dimension, and

• •

let $V^{i}_{0}$ be the length of the region of $\Gamma_{i}$ not covered by a box in $B_{i}$ (i.e., $V^{i}_{0}=(|\Gamma_{i}|-\sum_{j=1}^{n}{V^{i}_{j}})$); then

• •

$P_{i}(x)=\sum_{j=0}^{n}{V^{i}_{j}}\cdot x^{j}$.

Since any slab entirely covers the domain in all the dimensions but the one to which it is orthogonal, any point $p$ has depth $k$ in $\mathcal{B}$ if and only if it has depth $j_{1}$ in $\mathcal{B}_{1}$, $j_{2}$ in $\mathcal{B}_{2}$, $\ldots,$ and $j_{d}$ in $\mathcal{B}_{d}$, such that $j_{1}+j_{2}+\ldots+j_{d}=k$. Thus, for all $k\in[0..n]$:

[TABLE]

which is precisely the $(k+1)$-th coefficient of $P_{1}(x)\cdot P_{2}(x)\cdot\ldots\cdot P_{d}(x)$. Thus, this product yields the Depth Distribution $\left(V_{1},\ldots,V_{n}\right)$ of $\mathcal{B}$ in $\Gamma$. ∎

{INUTILE}

Lemma 2

Let $\mathcal{B}$ be a set of $n$ axis-parallel $d$-dimensional axis aligned boxes, with $d\geq 2$, that are equivalent to slabs when restricted to a domain box $\Gamma$. Then, the problem of computing the Depth Distribution of $\mathcal{B}$ is equivalent to the problem of multiplying $d$ polynomials of degree at most $n$.

Proof

To prove that the problems are equivalent we show that Depth Distribution is a subproblem of Polynomial Multiplication, and vice versa.

Depth Distribution $\subseteq$ Polynomial Multiplication:

For all $i$ in $[1..n]$ let:

• •

$\mathcal{B}_{i}$ be the slabs that are orthogonal $x_{i}$;

• •

$\left(V^{i}_{1},\ldots,V^{i}_{n}\right)$ be the Depth Distribution of the projection of $\mathcal{B}_{i}$ to $x_{i}$;

• •

$|\Gamma_{i}|$ denote the length of $\Gamma$ when projected to dimension $x_{i}$;

• •

$V^{i}_{0}=(|\Gamma|_{i}-\sum_{j=1}^{n}{V^{i}_{j}})$; and

• •

$P_{i}(x)=\sum_{j=0}^{n}{V^{i}_{j}}\cdot x^{j}$ be a polynomial of degree $n$.

Any slab covers the entire domain in all the dimensions except the one to which it is orthogonal. Therefore, any point $p$ has depth $k$ in $\mathcal{B}$ if it has depth $j_{1},\ldots,j_{d}$ in $\mathcal{B}_{1},\ldots,\mathcal{B}_{d}$, respectively, such that $j_{1}+\ldots+j_{d}=k$. Thus, for all $k=[0..n]$:

[TABLE]

which is precisely the $k$-th coefficient of $P_{1}(x)\times\ldots\times P_{d}(x)$. So computing the product yields the Depth Distribution.

Polynomial Multiplication $\subseteq$ Depth Distribution:

Let $P_{i}(x)=\sum_{j=0}^{n}{a^{i}_{j}}x^{j}$, for $i=[1..n]$ be $d$ polynomials of degree $n$. We show that there is a set of boxes $\mathcal{B}$, such that from the Depth Distribution of $\mathcal{B}$, one obtains with little extra effort the coefficients of $P_{1}(x)\times\ldots\times P_{d}(x)$.

We construct a domain box $\Gamma$, and for the $i$-th polynomial, we create a set of slabs within $\Gamma$ orthogonal to the $i$-th dimension, as follows (see Figure 2 for an illustration):

• •

$\Gamma=\{(x_{1},\ldots,x_{d})\mid 0\leq x_{1}\leq\sum_{j=0}^{n}{a^{1}_{j}}\wedge\ldots\wedge 0\leq x_{d}\leq\sum_{j=0}^{n}{a^{d}_{j}}\}$ (in the $i$-th dimension, $\Gamma$ extends from zero to the sum of the coefficients of $P_{i}$, for $i=[1..d]$);

• •

For all $i=[1..d]$, $\mathcal{B}_{i}=\bigcup_{j=1}^{n}\left(\Gamma\cap\{(x_{1},\ldots,x_{d})\in\Gamma\mid 0\leq x_{i}\leq\sum_{k=0}^{j}{a^{i}_{k}}\}\right)$ ($\mathcal{B}_{i}$ has $n$ boxes such that the length of the region in the $i$-th coordinate covered by exactly $k$ boxes is $a^{i}_{k}$, for $k=[0..n]$); and

• •

$\mathcal{B}=\bigcup_{i}^{d}\mathcal{B}_{i}$ ($\mathcal{B}$ is the set of $d\cdot n$ boxes resulting from the union of $\mathcal{B}_{i}$, for $i=[1..d]$).

When constructed as above, the Depth Distribution of the one dimensional projection of $\mathcal{B}_{i}$ to the $i$-th dimension is precisely $(V^{i}_{1}=a^{i}_{1},\ldots,V^{i}_{n}=a^{i}_{n})$ for all $i=[1..d]$, and the Depth Distribution of $\mathcal{B}$ is given by Section 3.1. Thus, computing the Depth Distribution of $\mathcal{B}$ yield the first $(nd-1)$ coefficients of $P_{1}(x)\times\ldots\times P_{n}(x)$, in descending order of the exponents. Finally, the last coefficient of the product (i.e. the constant coefficient) is given by $|\Gamma|-\sum_{i=1}^{n}{V_{i}}$, where $|\Gamma|$ denotes the volume of $\Gamma$. ∎

Using standard Fast Fourier Transform techniques, two polynomials can be multiplied in time within $O(n\log n)$ [12]. Moreover, the Depth Distribution of a set of intervals (i.e., when $d=1$) can be computed in linear time after sorting, by a simple scan-line algorithm, as for the Klee’s Measure [10]. Thus, as a consequence of Lemma 1, when the boxes in $\mathcal{B}$ are slabs when restricted to a domain box $\Gamma$, the Depth Distribution of $\mathcal{B}$ within $\Gamma$ can be computed in time within $\mathcal{O}(n\log n)$.

Corollary 1

Let $\mathcal{B}$ be a set of $n$ $d$-dimensional axis aligned boxes whose intersections with a $d$-dimensional box $\Gamma$ are slabs. The Depth Distribution of $\mathcal{B}$ inside $\Gamma$ can be computed in time within $\mathcal{O}(n\log n)$.

{INUTILE}

With arguments similar to the ones used in the proof of Lemma 1 one can show that multiplying two polynomials of degree $n$ can be reduced in linear time to an instance of the computation of the Depth Distribution of a set of $\mathcal{O}(n)$ boxes. Hence, the running time of Corollary 1 is tight.

A naive application of previous techniques [10, 22] to the computation of Depth Distribution yields poor results. Combining the result in Corollary 1 with the partition of the space and the data structure described by Overmars and Yap [22] yields an algorithm to compute the Depth Distribution running in time within $\mathcal{O}(n^{\frac{d+1}{2}}\log n)$, and using space within $\mathcal{O}(n^{d/2}\log n)$. Similarly, if the result in Corollary 1 is combined with Chan’s partition of the space [10], one obtains an algorithm using space linear in the number of boxes, but running in time within $\mathcal{O}(n^{\frac{d}{2}+1}\log n)$ (i.e., paying an extra $\mathcal{O}(n^{\frac{1}{2}})$-factor for the reduction in space usage of Overmars and Yap [22]).

We combine these two approaches into an algorithm which achieves the best features of both: it runs in time within $\mathcal{O}(n^{\frac{d+1}{2}}\log n)$, and uses $\mathcal{O}(n\log n)$-space. As in Chan’s approach [10] we use a divide and conquer algorithm, but we show in Theorem 3.1 that the running time is asymptotically the same as if using the partition and data structures described by Overmars and Yap [22] (see Algorithm 1 for a detailed description).

{INUTILE}

We show how to transform any set of $n$ boxes $\mathcal{B}$ into a $\mathcal{B^{\prime}}$ with $\mathcal{O}(n)$ boxes, such that any rectangular region which does not intersect a ($d$-2)-face of a box in $\mathcal{B^{\prime}}$, partially intersects (i.e., intersects but does not contain) at most $\mathcal{O}(\sqrt{n})$ boxes of $\mathcal{B}$.

Lemma 3

Let $\mathcal{B}$ be a set of $n$ axis-parallel boxes in $\mathbb{R}^{d}$, and let $\Gamma$ be a $d$-dimensional domain box. There exists a set $\mathcal{B}^{\prime}$ such that:

1. a)

the Depth Distribution of $\mathcal{B}^{\prime}$ and $\mathcal{B}$ within $\Gamma$ are the same; 2. b)

the size of $\mathcal{B}^{\prime}$ is within $\mathcal{O}(n)$ 3. c)

any box $\gamma$ which does not intersect a ($d$-2)-face of some box in $\mathcal{B}^{\prime}$, partially intersects more than $\sqrt{n}$ boxes from $\mathcal{B}^{\prime}$ in at most one dimension.

Proof

Initially, let $\mathcal{B}^{\prime}=\mathcal{B}$. We add to $\mathcal{B^{\prime}}$ empty boxes of the form $\{(x_{1},\ldots,x_{d})\in\Gamma\mid x_{i}=\alpha,x_{j}=\beta\}$ for some $i,j\in[1..d]$, and some endpoints $\alpha,\beta$ of the boxes of $\mathcal{B}$. For this, for every pair of dimensions $i,j$, let $I$ (resp. $J$) be the set of endpoints at positions $\frac{\sqrt{n}}{d},\frac{2\sqrt{n}}{d},\ldots,\frac{d\sqrt{n}\sqrt{n}}{d}$ in the $i$-th dimension (resp. the $j$-dimension) in ascending order. For each pair $(\alpha\in I,\beta\in J)$ we add the box $\{(x_{1},\ldots,x_{d})\in\Gamma\mid x_{i}=\alpha,x_{j}=\beta\}$ to $\mathcal{B^{\prime}}$. As the boxes added to $\mathcal{B}^{\prime}$ are empty (i.e., their volume is zero), the Depth Distribution of $\mathcal{B}^{\prime}$ is the same as that of $\mathcal{B}$. The total number of boxes added for each pair of dimensions is $2d\sqrt{n}\times 2d\sqrt{n}=4d^{2}n$ boxes, and because the number of distinct pairs of dimensions is constant, the size of $\mathcal{B}^{\prime}$ is within $\mathcal{O}(n)$. The third assertion derives directly from the construction. ∎

The result in Lemma 1 allows to use the partition scheme introduced by Chan [10], while obtaining the nice properties yielded by the partition scheme introduced by Overmars and Yap [22] (which is key to the algorithm we describe below for computing the Depth Distribution of a set of boxes).

Theorem 3.1

Let $\mathcal{B}$ be a set of $n$ axis-parallel boxes in $\mathbb{R}^{d}$. The Depth Distribution of $\mathcal{B}$ can be computed in time within $\mathcal{O}(n^{\frac{d+1}{2}}\log n)$, using space within $\mathcal{O}(n\log n)$.

{SHORT}

Due to lack of space we defer the complete proof to the extended version [2017-ARXIV-DepthDistributionInHighDimension-BabrayPerezRojas].

{LONG}

Proof

First, let us show that the running time $T(n)$ of Algorithm 1 is within $\mathcal{O}(n^{\frac{d+1}{2}}\log n)$. We can charge the number of boxes in the set to the number of ($d$-1)-faces intersecting the domain: if a box in $\mathcal{B}$ does not have a ($d$-1)-face intersecting the domain, then it covers the entire domain, and it would have been removed (simplified) from the input in the parent recursive call. Note that the ($d$-1)-faces orthogonal to dimension $x_{1}$ cannot intersect both the sub-domains $\Gamma_{L}$ and $\Gamma_{R}$ of the recursive calls at the same time (because the algorithm uses a hyperplane orthogonal to $x_{1}$ to split the domain into $\Gamma_{L}$ and $\Gamma_{R}$). Hence, although at the $d$-th level of the recursion there are $2^{d}$ recursive calls, any ($d$-1)-face can appear in at most $2^{d-1}$ of those. In general, for any $i$, there are at most $2^{i}$ recursive calls at the $i$-th level of recursion, but any ($d$-1)-face of the original set can intersect at most $2^{\lfloor i/d\rfloor(d-1)}$ of the cells corresponding to the domain of those calls. Hence, the total number of ($d$-1)-faces which survive until the $i$-th level of the recursion tree is within $\mathcal{O}(n2^{\lfloor i/d\rfloor(d-1)})$ (a similar argument was used by Overmars and Yap [22] to bound the running time of the data structure they introduced).

Let $h$ be the height of the recursion tree of Algorithm 1. Chan [10] showed that the partition in steps 10-12 makes $h$ to be within a constant term of $\frac{d}{2}\log n$. We analyze separately the total cost $T_{I}(n)$ of the interior nodes of the recursion tree (i.e. the nodes corresponding to recursive calls which fail the base case condition in step 2), from the total cost $T_{L}(n)$ of the leaves of the recursion tree.

Since, the cost of each interior node is linear in the number of ($d$-1)-faces intersecting the corresponding domain, $T_{I}(n)$ is bounded by:

[TABLE]

To analyze the total cost of the leaves of the recursion tree, first note that the total number $l$ of such recursive calls is within $\mathcal{O}(n^{d/2})$. Let $n_{1},\ldots,n_{l}$ denote the number of ($d$-1)-faces in each of those recursive calls, respectively. Note that $T_{L}(n)$ is within $\mathcal{O}(\sum_{i=1}^{l}n_{i}\log n_{i})$ because the result of Lemma 1 is used in line step 2 of the algorithm. Besides, since the number of ($d$-1)-faces which survive until the $h$-th level of the recursion tree is within $\mathcal{O}(n^{\frac{d-1}{2}})$ , $\sum_{i=1}^{l}{n_{i}}\in\mathcal{O}(n^{\frac{d+1}{2}})$. That bound, and the fact that $\log{n_{i}}\leq\log n$, for all $i\in[1..l]$, yields $T_{L}(n)\in\mathcal{O}(n^{\frac{d+1}{2}}\log n)$. As $T(n)=T_{I}(n)+T_{L}(n)$, the bound for the running time follows.

With respect to the space used by the algorithm, note that only one path in recursion tree is active at any moment, and that at most $\mathcal{O}(n)$ extra space is needed within each recursive call. Since the height of the recursion tree is within $\mathcal{O}(\log n)$, the total space used by the algorithm is clearly within $\mathcal{O}(n\log n)$. ∎

Note that in SDC-DDistribution the Depth Distribution is accumulated into a parameter. This is only to simplify the description and analysis of the algorithm, it does not impact its computational or space complexity. The initialization of the parameters of SDC-DDistribution should be done as below:

The bound for the running time in Theorem 3.1 is worse than that of computing the Klee’s Measure (and Maximum Depth) by a factor within $\mathcal{O}(\sqrt{n}\log n)$, which raises the question of the optimality of the bound: we consider this matter in the next section.

3.2 Conditional Lower Bound

As for many problems handling high dimensional inputs, the best lower bound known for this problem is $\Omega(n\log n)$ [15], which derives from the fact that the Depth Distribution is a generalization of the Klee’s Measure problem. This bound, however, is tight only when the input is a set of intervals (i.e, $d=1$). For higher dimensions, the conjectured lower bound of $\Omega(n^{d/2})$ described by Chan [9] in 2008 for the computational complexity of computing the Klee’s Measure can be extended analogously to the computation of the Depth Distribution.

One intriguing question is whether in dimension $d=2$, as for Klee’s Measure, the Depth Distribution can be computed in time within $\mathcal{O}(n\log n)$. We argue that doing so would imply breakthrough results in a long standing problem, Matrix Multiplication. We show that any instance of Matrix Multiplication can be solved using an algorithm which computes the Depth Distribution of a set of rectangles in the plane. For this, we make use of the following simple observation:

Observation 1.

Let A,B be two $n\times n$ matrices of real numbers, and let $C_{i}$ denote the $n\times n$ matrix that results from multiplying the $n\times 1$ vector corresponding to the $i$-th column of $A$ with the $1\times n$ vector corresponding to the $i$-th row of $B$. Then, $AB=\sum_{i=1}^{n}C_{i}$.

We show in Theorem 3.2 that multiplying two $n\times n$ matrices can be done by transforming the input into a set of $\mathcal{O}(n^{2})$ axis-aligned rectangles, and computing the Depth Distribution of the resulting set. Moreover, this transformation can be done in linear time, thus, the theorem yields a conditional lower bound for the computation of the Depth Distribution.

Theorem 3.2

Let $A,B$ be two $n\times n$ matrices of non-negative real numbers. There is a set $\mathcal{B}$ of rectangles of size within $O(n^{2})$, and a domain rectangle $\Gamma$, such that the Depth Distribution of $\mathcal{B}$ within $\Gamma$ can be projected to obtain the value of the product $AB$.

{SHORT}

Intuitively, we create a gadget to represent each matrix $C_{i}$. Within the $i$-th gadget, there will be a rectangular region for each component of $C_{i}$ with the value of that component as volume. We arrange the boxes so that two distinct regions have the same depth if and only if they represent the same respective coefficients of two distinct matrices $C_{i}$ and $C_{i^{\prime}}$ (formally, they represent coefficients $(C_{i})_{j,k}$ and $(C_{i^{\prime}})_{j^{\prime},k^{\prime}}$, respectively, such that $i\neq i^{\prime},j=j^{\prime}$, and $k=k^{\prime}$).

Due to lack of space, we defer the complete proof to the extended version [2017-ARXIV-DepthDistributionInHighDimension-BabrayPerezRojas].

{LONG}

Proof

We create a gadget to represent each $C_{i}$. Within the $i$-th gadget, there will be a rectangular region for each coefficient of $(C_{i})$ with the value of that coefficient as volume (see Figure 3 for a general outlook of the instance). We arrange the boxes so that two of such rectangular regions have the same depth if and only if they represent the same respective coefficients of two distinct matrices $C_{i}$ and $C_{i^{\prime}}$ (formally, they represent coefficients $(C_{i})_{j,k}$ and $(C_{i^{\prime}})_{j^{\prime},k^{\prime}}$, respectively, such that $i\neq i^{\prime},j=j^{\prime}$, and $k=k^{\prime}$).

We describe a set of $5n^{2}$ boxes $\mathcal{B}$ (one box for each of the $n^{2}$ coefficients of $A$, two boxes for each of the $n^{2}$ coefficients of $B$, and $2n^{2}$ additional boxes) such that, for each $i,j\in[1..n]$, the $(2ni+2j+1)$-th component of the Depth Distribution of $\mathcal{B}$ is equal to the component $AB_{i,j}$ of the product $AB$. Such a set can be constructed as follows (see Figure 4 for a graphical representation of the instance generated):

• •

Let the domain $\Gamma=\{(x,y)\mid 0\leq x\leq\sum_{i}{\sum_{j}{B_{i,j}}},0\leq y\leq\sum_{i}{\max_{j}\{A_{i,j}\}}\}$.

• •

For all $i\in[1..n]$ we create a gadget for $C_{i}$ that covers the entire domain in the $y$-direction, and that spans from $C_{i}^{start}=\sum_{j=1}^{i-1}{\sum_{k=0}^{n}{B_{j,k}}}$ to $C_{i}^{end}=C_{i}^{start}+\sum_{k=0}^{n}{B_{i,k}}$ in the $x$-direction.

• •

Within the gadget for $C_{i}$ we place one box for each $A_{j,i}$ and two boxes for each $B_{i,j}$, for $i,j\in[1..n]$, as follows: the one corresponding to $A_{j,i}$ will span $C_{i}$ entirely in the $x$-direction, and is bounded by $(\sum_{k=1}^{j}{\max_{l=1}^{n}\{A_{k,l}\}})\leq y\leq(A_{j,i}+\sum_{k=1}^{j}{\max_{l=1}^{n}\{A_{k,l}\}})$ in the $y$-direction. For $B_{i,j}$ we place two identical boxes entirely spanning $C_{i}$ in the $y$-direction, and in the $x$-direction bounded by $(C_{i}^{start}+\sum_{k=1}^{j-1}{B_{i,k}})\leq x\leq C_{i}^{end}$.

• •

Finally, we add $2n^{2}$ boxes to ensure that rectangular regions corresponding to two coefficients $C_{i,j}$ and $C_{i,k}$ in distinct rows $j,k$ of a same $C_{i}$ do not share the same depth, for all $i,j,k\in[1..n]$ . For this, for all $j\in[1..n]$ we add $2n$ identical boxes entirely spanning the domain in the $x$-direction, and spanning from $(\sum_{k=1}^{j}{\max_{l=1}^{n}\{A_{k,l}\}})$ to $(\sum_{i}{\max_{j}\{A_{i,j}\}}))$ in the $y$-direction.

Note that in the instance generated, for $i,j\in[1..n]$:

• •

a region has odd depth if and only if its volume is equal to some coefficient of any $C_{i}$;

• •

the regions corresponding to coefficients of the $i$-th rows have depth between $(2in+3)$ and $(4in+1)$;

• •

within the gadget for each $C_{i}$, the rectangular region with volume corresponding to the coefficient $C_{i,j}$ has depth $(2ni+2j+1)$, and no other rectangular region within the gadget has that depth;

• •

two distinct regions have the same depth if and only if they represent the same respective coefficients of two distinct matrices $C_{i}$ and $C_{i^{\prime}}$.

The arguments above and the fact that, by definition of the Depth Distribution the volumes of regions with the same depth are accumulated together, yield the result of the theorem. ∎

The optimal time to compute the product of two $n\times n$ matrices is still open. It can naturally be computed in time within $O(n^{3})$. However, Strassen showed in 1969 that within $O(n^{2.81})$ arithmetic operations are enough [23]. This gave rise to a new area of research, where the central question is to determine the value of the exponent of the computational complexity of square matrix multiplication, denoted $\omega$, and defined as the minimum value such that two $n\times n$ matrices can be multiplied using within $O(n^{\omega+\varepsilon})$ arithmetic operations for any $\varepsilon>0$.

{LONG}

The result of Theorem 3.2 directly yields a conditional lower bound on the complexity of Depth Distribution: in particular, Depth Distribution in dimension as low as two, can be solved in time within $\mathcal{O}(n\log n)$, then Matrix Multiplication can be computed in time within $\mathcal{O}(n^{2})$, i.e. $\omega=2$. However, this would be a great breakthrough in the area, the best known upper bound to date is approximately $\omega\leq 2.37$, when improvements in the last 30 years [11, 16] have been in the range $[2.3728,2.3754]$.

Corollary 2 (Conditional lower bound)

Computing the Depth Distribution of a set $B$ of $n$ $d$-dimensional boxes requires time within $\Omega(n^{1+c})$, for some constant $c>0$, unless two $n\times n$ matrices can be multiplied in time $\mathcal{O}(n^{2+\varepsilon})$, for any constant $\varepsilon>0$.

The running time of the algorithm that we described in Theorem 3.1 can be improved for large classes of instances (i.e. asymptotically infinite) by considering measures of the difficulty of the input other than its size. We describe two of these improved solutions in the next section.

4 Multivariate analysis

Even though the asymptotic complexity of $\mathcal{O}(n^{\frac{d+1}{2}}\log n)$ is the best we know so far for computing the Depth Distribution of a set of $n$ $d$-dimensional boxes, there are many cases which can be solved faster. Some of those “easy” instances can be mere particular cases, but others can be hints of some hidden measures of difficulty of the Depth Distribution problem. We show that, indeed, there are at least two such difficulty measures, gradually separating instances of the same size $n$ into various classes of difficulty. Informally, the first one (the profile of the input set, Section 4.1) measures how separable the boxes are by axis-aligned hyperplanes, whereas the second one (the degeneracy of the intersection graph, Section 4.2) measures how “complex” the interactions of the boxes are in the set between them. Those measures inspire similar results for the computation of the Klee’s Measure and of the Maximum Depth.

4.1 Profile

The $i$-th profile $p_{i}$ of a set of boxes $\mathcal{B}$ is the maximum number of boxes intersected by any hyperplane orthogonal to the $i$-th dimension; and the profile $p$ of $\mathcal{B}$ is the minimum $p=\min_{i\in[1..d]}\{p_{i}\}$ of those over all dimensions. D’Amore [13] showed how to compute it in linear time (after sorting the coordinate of the boxes in each dimension). The following lemma shows that the Depth Distribution can be computed in time sensitive to the profile of the input set.

Lemma 4

Let $\mathcal{B}$ be a set of boxes with profile $p$, and $\Gamma$ be a $d$-dimensional axis-aligned domain box. The Depth Distribution of $\mathcal{B}$ within $\Gamma$ can be computed in time within $\mathcal{O}(n\log n+np^{\frac{d-1}{2}}\log p)\subseteq\mathcal{O}(n^{\frac{d+1}{2}}\log n)$.

{SHORT}

Due to lack of space we defer the complete proof to the extended version [2017-ARXIV-DepthDistributionInHighDimension-BabrayPerezRojas].

{LONG}

Proof

We describe an algorithm which partitions the domain into independent slabs, computes the Depth Distribution within each slab, and combines the results into the final answer. For this, it sweeps a plane by the dimension with smallest profile, and after every $2p$ endpoints, it creates a new slab cutting the space with a hyperplane orthogonal to this dimension. This partitions the space into $\mathcal{O}(n/p)$ slabs, each intersecting at most $\mathcal{O}(p)$ boxes. Finally, the algorithm computes the Depth Distribution of $\mathcal{B}$ within each slab in time within $\mathcal{O}(p^{\frac{{d+1}}{2}}\log p)$, and obtains the Depth Distribution of $\mathcal{B}$ within $\Gamma$ by summing the respective components of the Depth Distribution within each slab. The total running time of this is within $\mathcal{O}(n\log n+np^{\frac{d-1}{2}}\log p)$. ∎

The lemma above automatically yields refined results for the computation of the Klee’s Measure and the Maximum Depth of a set of boxes $\mathcal{B}$. However, applying the technique in an ad-hoc way to these problems yields better bounds:

Corollary 3

Let $\mathcal{B}$ be a set of boxes with profile $p$, and $\Gamma$ be a domain box. The Klee’s Measure and Maximum Depth of $\mathcal{B}$ within $\Gamma$ can be computed in time within $\mathcal{O}(n\log n+np^{\frac{d-2}{2}})\subseteq\mathcal{O}(n^{d/2})$.

The algorithms from Lemma 4 and Corollary 3 asymptotically outperform previous ones in the sense that their running time is never worse than previous algorithms by more than a constant factor, but can perform faster by more than a constant factor on specific families of instances.

An orthogonal approach is to consider how complex the interactions between the boxes are in the input set $\mathcal{B}$, analyzing, for instance, the intersection graph of $\mathcal{B}$. We study such a technique in the next section.

4.2 Intersections Graph Degeneracy

A $k$-degenerate graph is an undirected graph in which every subgraph has a vertex of degree at most $k$ [19]. Every $k$-degenerate graph accepts an ordering of the vertices in which every vertex is connected with at most $k$ of the vertices that precede it (we refer below to such an ordering as a degenerate ordering).

In the following lemma we show that this ordering can be used to compute the Depth Distribution of a set $\mathcal{B}$ of $n$ boxes in running time sensitive to the degeneracy of the intersection graph of $\mathcal{B}$.

Lemma 5

Let $\mathcal{B}$ be a set of boxes and $\Gamma$ be a domain box, and let $k$ be the degeneracy of the intersection graph $G$ of the boxes in $\mathcal{B}$. The Depth Distribution of $\mathcal{B}$ within $\Gamma$ can be computed in time within $\mathcal{O}(n\log^{d}n+e+nk^{\frac{d+1}{2}})$, where $e\in\mathcal{O}(n^{2})$ is the number of edges of $G$.

Proof

We describe an algorithm that runs in time within the bound in the lemma. The algorithm first computes the intersection graph $G$ of $\mathcal{B}$ in time within $\mathcal{O}(n\log^{d}n+e)$ [14], as well as the $k$-degeneracy of this graph and a degenerate ordering $O$ of the vertices in time within $\mathcal{O}(n+e)$ [20]. The algorithm then iterates over $O$ maintaining the invariant that, after the $i$-th step, the Depth Distribution of the boxes corresponding to the vertices $v_{1},v_{2},\ldots,v_{i}$ of the ordering has been correctly computed.

For any subset $V$ of vertices of $G$, let $DD_{\mathcal{B}}^{\Gamma}(V)$ denote the Depth Distribution within $\Gamma$ of the boxes in $\mathcal{B}$ corresponding to the vertices in $V$. Also, for $i\in[1..n]$ let $O[1..i]$ denote the first $i$ vertices of $O$, and $O[i]$ denote the $i$-th vertex of $O$. From $DD_{\mathcal{B}}^{\Gamma}(O[1..i\text{-1}])$ (which the algorithm “knows” after the ($i$-1)-th iteration), $DD_{\mathcal{B}}^{\Gamma}(O[1..i])$ can be obtained as follows: ($i.$) let $P$ be the subset of $O[1..i\text{-1}]$ connected with $O[i]$; ($ii.$) compute $DD_{\mathcal{B}}^{O[i]}(P\cup\{O[i]\})$ in time within $\mathcal{O}(k^{\frac{d+1}{2}}\log k)$ using SDC-DDistribution (note that the domain this time is $O[i]$ itself, instead of $\Gamma$); ($iii.$) add to $(DD_{\mathcal{B}}^{\Gamma}(O[1..i]))_{1}$ the value of $(DD_{\mathcal{B}}^{O[i]}(P\cup O[i]))_{1}$; and ($iv.$) for all $j=[2..k\text{+1}]$, substract from $(DD_{\mathcal{B}}^{\Gamma}(O[1..i]))_{j-1}$ the value of $(DD_{\mathcal{B}}^{O[i]}(P\cup O[i]))_{j}$ and add it to $(DD_{\mathcal{B}}^{\Gamma}(O[1..i\text{-1}]))_{j}$.

Since the updates to the Depth Distribution in each step take time within $\mathcal{O}(k^{\frac{d+1}{2}}\log k)$, and there are $n$ such steps, the result of the lemma follows. ∎

Unlike the algorithm sensitive to the profile, this one can run in time within $\mathcal{O}(n^{1+\frac{d+1}{2}})$ (e.g. when $k=n$), which is only better than the $\mathcal{O}(n^{\frac{d+1}{2}})$ complexity of SDC-DDistribution for values of the degeneracy $k$ within $\mathcal{O}(n^{1-{\frac{2}{d}}})$.

Applying the same technique to the computation of Klee’s Measure and Maximum Depth yields improved solutions as well:

Corollary 4

Let $\mathcal{B}$ be a set of boxes and $\Gamma$ be a domain box, and let $k$ be the degeneracy of the intersection graph $G$ of the boxes in $\mathcal{B}$. The Klee’s Measure and Maximum Depth of $\mathcal{B}$ within $\Gamma$ can be computed in time within $\mathcal{O}(n\log^{d}n+e+nk^{\frac{d}{2}})$, where $e\in\mathcal{O}(n^{2})$ is the number of edges of $G$.

Such refinements of the worst-case complexity analysis are only examples and can be applied to many other problems handling high dimensional data inputs. We discuss a selection in the next section.

{INUTILE}

4.3 Computing only selected elements of the Depth Distribution

An intriguing question is whether the computation of a particular $V_{i}$ can be performed in time depending on $i$. For instance, $V_{n}$ can be computed in linear time since it is the volume of the intersection of all the boxes. Besides, for any constant $c$, $V_{n-c}$ can be computed in time within $O(n^{c})$. However, we show below that in many cases computing $V_{i}$ requires as many time as solving the Coverage problem, which detects whether a set of boxes $\mathcal{B}$ covers a given domain $\Gamma$.

Lemma 6

Let $\mathcal{B}$ be a set of boxes and $\Gamma$ be a domain box. Computing the volume of the region within $\Gamma$ covered by exactly $n/f(n)$ boxes from $\mathcal{B}$, for any function $f$ such that $1<f(n)\leq n$, is as hard as computing whether $\mathcal{B}$ Coverage can be solved in that time.

Proof

Construct $\mathcal{B}^{\prime}$ by adding $\frac{n}{c-1}$ boxes to $\mathcal{B}$ covering the entire domain. The size $n^{\prime}$ of $\mathcal{B}^{\prime}$ is $n^{\prime}=\frac{cn}{c-1}$, and the fraction of boxes added with respect to $n^{\prime}$ is $n^{\prime}/c$. Since the boxes added cover completely the domain, $\mathcal{B}$ covers the domain if and only if the $(n^{\prime}/c)$-th element of the Depth Distribution of $\mathcal{B}^{\prime}$ is zero. ∎

The above lemma allows to argue that potentially improved running times based on being sensitive to the index $i$ of the selected component will still be within $\Omega(n^{d/2})$, for many values of $i$ (except maybe for $i$ within $n-\emph{o}(n)$).

{TODO}

Add running time base on $i$ which improves the running time within the range $[n^{d/2},n^{(d+1)/2}]$

5 Discussion

The Depth Distribution captures many of the features in common between Klee’s measure and Maximum Depth, so that new results on the computation of the Depth Distribution will yield corresponding results for those two measures, and has its own applications of interest. Nevertheless, there is no direct reduction to Klee’s measure or Maximum Depth from Depth Distribution, as the latter seems computationally more costly, and clarifying further the relationship between these problems will require finer models of computation. We discuss below some further issues to ponder about those measures.

Discrete variants.

In practice, multidimensional range queries are applied to a database of multidimensional points. This yields discrete variants of each of the problems previously discussed [1, 25]. In the Discrete Klee’s measure{TODO} Add here alternative names introduced by  [1] and [25] , the input is composed of not only a set $\cal B$ of $n$ boxes, but also of a set $S$ of $m$ points. The problem is now to compute not the volume of the union of the boxes, but the number (and/or the list) of points which are covered by those boxes. {TODO} Describe here in two sentences each the work of [1] and [25]

Similarly, one can define a discrete version of the Maximum Depth (which points are covered by the maximum number of boxes) and of the Depth Distribution (how many and which points are covered by exactly $i$ boxes, for $i\in[1..n]$). Interestingly enough, the computational complexity of these discrete variants is much less than that of their continuous versions when there are reasonably few points [25]: the discrete variant becomes hard only when there are many more points than boxes [1]. Nevertheless, “easy” configurations of the boxes also yield “easy” instances in the discrete case: it will be interesting to analyze the discrete variants of those problems according to the measures of profile and k-degeneracy introduced on the continuous versions. {TODO}

Finer Multivariate Analysis.

Whereas we refined the worst case complexity over instances of fixed size by introducing and taking account various parameters, there is still room for further refinements. Javiel: add here some of the open questions

Tighter Bounds.

Chan [10] conjectured that a complexity of $\Omega(n^{d/2})$ is required to compute the Klee’s Measure, and hence to compute the Depth Distribution. However, the output of Depth Distribution gives much more information than the Klee’s Measure, of which a large part can be ignored during the computation of the Klee’s Measure (while it is required for the computation of the Depth Distribution). It is not clear whether even a lower bound of $\Omega(n^{d/2+\epsilon})$ can be proven on the computational complexity of the Depth Distribution given this fact. {TODO} JAVIEL: Complete here.

Funding: All authors were supported by the Millennium Nucleus “Information and Coordination in Networks” ICM/FIC RC130003. Jérémy Barbay and Pablo Pérez-Lantero were supported by the projects CONICYT Fondecyt/Regular nos 1170366 and 1160543 (Chile) respectively, while Javiel Rojas-Ledesma was supported by CONICYT-PCHA/Doctorado Nacional/2013-63130209 (Chile).