An Efficient Skyline Computation Framework

TL;DR

This paper introduces SDI, a dimension indexing framework for efficient skyline computation that reduces comparisons and accelerates processing, especially in high-dimensional datasets.

Contribution

It proposes SDI, a novel dimension index-based framework, and SDI-RS, an algorithm that significantly improves skyline computation efficiency with guaranteed complexity.

Findings

SDI reduces the number of comparisons needed for skyline determination.

SDI-RS outperforms baseline algorithms in speed, especially on high-dimensional data.

The framework guarantees space and time complexity bounds.

Abstract

Skyline computation aims at looking for the set of tuples that are not worse than any other tuples in all dimensions from a multidimensional database. In this paper, we present SDI (Skyline on Dimension Index), a dimension indexing conducted general framework to skyline computation. We prove that to determine whether a tuple belongs to the skyline, it is enough to compare this tuple with a bounded subset of skyline tuples in an arbitrary dimensional index, but not with all existing skyline tuples. Base on SDI, we also show that any skyline tuple can be used to stop the whole skyline computation process with outputting the complete set of all skyline tuples. We develop an efficient algorithm SDI-RS that significantly reduces the skyline computation time, of which the space and time complexity can be guaranteed. Our experimental evaluation shows that SDI-RS outperforms the baseline algorithms in general and is especially very efficient on high-dimensional data.

Tables

Table 1. Table 1. A sample database with d = 6 𝑑 6 d=6 , n = 10 𝑛 10 n=10 , and m = 6 𝑚 6 m=6 .

ID	$D_1$	$D_2$	$D_3$	$D_4$	$D_5$	$D_6$	Skyline
$t_0$	7.5	1.3	7.5	4.5	5.3	2.1	Yes
$t_1$	4.7	6.7	6.7	9.3	3.8	5.1	Yes
$t_2$	8.4	9.4	5.3	5.8	6.7	7.5	No
$t_3$	5.3	6.6	6.7	6.8	5.8	9.3	Yes
$t_4$	8.4	5.2	5.1	5.5	4.1	7.5	Yes
$t_5$	9.1	7.6	2.6	4.7	7.3	6.2	Yes
$t_6$	5.3	7.5	1.9	5.9	3.4	1.8	Yes
$t_7$	5.3	7.5	6.7	7.2	6.3	8.8	No
$t_8$	6.7	7.3	7.6	9.7	5.3	8.7	No
$t_9$	7.5	9.6	4.8	8.9	9.5	6.5	No

Table 2. Table 2. Dimension indexing of the sample database shown in Table 1 .

${ℐ}_1$	${ℐ}_2$	${ℐ}_3$	${ℐ}_4$	${ℐ}_5$	${ℐ}_6$
4.7:1	1.3:0	1.9:6	4.5:0	3.4:6	1.8:6
5.3:3	5.2:4	2.6:5	4.7:5	3.8:1	2.1:0
5.3:6	6.6:3	4.8:9	5.5:4	4.1:4	5.1:1
5.3:7	6.7:1	5.1:4	5.8:2	5.3:0	6.2:5
6.7:8	7.3:8	5.3:2	5.9:6	5.3:8	6.5:9
7.5:0	7.5:6	6.7:1	6.8:3	5.8:3	7.5:2
7.5:9	7.5:7	6.7:3	7.2:7	6.3:7	7.5:4
8.4:2	7.6:5	6.7:7	8.9:9	6.7:2	8.7:8
8.4:4	9.4:2	7.5:0	9.3:1	7.3:5	8.8:7
9.1:5	9.6:9	7.6:8	9.7:8	9.5:9	9.3:3

Table 3. Table 3. A block view of the dimensional index ℐ i subscript ℐ 𝑖 \mathcal{I}_{i} .

${ℐ}_1$
4.7:1
5.3:3	5.3:6	5.3:7
6.7:8
7.5:0	7.5:9
8.4:2	8.4:4
9.1:5

Table 4. Table 4. The stop line created from tuple t 6 subscript 𝑡 6 t_{6} covers 41 index entries in total.

${ℐ}_1$	${ℐ}_2$	${ℐ}_3$	${ℐ}_4$	${ℐ}_5$	${ℐ}_6$
4.7:1	1.3:0	1.9:6	4.5:0	3.4:6	1.8:6
5.3:3	5.2:4	2.6:5	4.7:5	3.8:1	2.1:0
5.3:6	6.6:3	4.8:9	5.5:4	4.1:4	5.1:1
5.3:7	6.7:1	5.1:4	5.8:2	5.3:0	6.2:5
6.7:8	7.3:8	5.3:2	5.9:6	5.3:8	6.5:9
7.5:0	7.5:6	6.7:1	6.8:3	5.8:3	7.5:2
7.5:9	7.5:7	6.7:3	7.2:7	6.3:7	7.5:4
8.4:2	7.6:5	6.7:7	8.9:9	6.7:2	8.7:8
8.4:4	9.4:2	7.5:0	9.3:1	7.3:5	8.8:7
9.1:5	9.6:9	7.6:8	9.7:8	9.5:9	9.3:3

Table 5. Table 5. The stop line created from tuple t 0 subscript 𝑡 0 t_{0} covers 37 index entries in total.

${ℐ}_1$	${ℐ}_2$	${ℐ}_3$	${ℐ}_4$	${ℐ}_5$	${ℐ}_6$
4.7:1	1.3:0	1.9:6	4.5:0	3.4:6	1.8:6
5.3:3	5.2:4	2.6:5	4.7:5	3.8:1	2.1:0
5.3:6	6.6:3	4.8:9	5.5:4	4.1:4	5.1:1
5.3:7	6.7:1	5.1:4	5.8:2	5.3:0	6.2:5
6.7:8	7.3:8	5.3:2	5.9:6	5.3:8	6.5:9
7.5:0	7.5:6	6.7:1	6.8:3	5.8:3	7.5:2
7.5:9	7.5:7	6.7:3	7.2:7	6.3:7	7.5:4
8.4:2	7.6:5	6.7:7	8.9:9	6.7:2	8.7:8
8.4:4	9.4:2	7.5:0	9.3:1	7.3:5	8.8:7
9.1:5	9.6:9	7.6:8	9.7:8	9.5:9	9.3:3

Table 6. Table 6. Skyline size of synthetic datasets.

Dataset		$d=2$	$d=4$	$d=6$	$d=8$	$d=16$	$d=24$
	100K	12	282	2534	9282	82546	99629
Independent	1M	17	423	6617	30114	629091	981611
	100K	3	9	49	135	3670	13479
Correlated	1M	1	19	36	208	8688	58669
	100K	56	3865	26785	55969	96816	99730
Anti-correlated	1M	64	8044	99725	320138	892035	984314

Table 7. Table 7. Statistics of real datasets.

Dataset	Cardinality ($n$)	Dimensionality ($d$)	Skyline Size ($m$)
NBA	17264	8	1796
HOUSE	127931	6	5774
WEATHER	566268	15	63398

Table 8. Table 8. Skyline searching time (msec) total run-time (msec) on correlated datasets.

		$d=2$		$d=4$		$d=6$		$d=8$
		$t_S$	$t_T$	$t_S$	$t_T$	$t_S$	$t_T$	$t_S$	$t_T$
SDI-RS	100K	0.14	271	0.38	664	423	1657	243	1348
+BFS	1M	0.17	5161	0.64	10293	1.34	23388	9896	40830
SDI-RS	100K	0.17	238	0.25	692	0.98	1263	4.75	1443
+DFS	1M	0.14	5772	0.42	12069	1.32	21845	5.86	33012
BNL	100K	2.51	75	5.36	141	5.33	243	10.29	284
	1M	25.49	744	45.13	1468	62.56	2267	94.04	2901
SaLSa	100K	2686	2784	386	543	26.03	278	51.67	361
max	1M	88.63	1067	451	2117	377	2987	674	3829
SFS	100K	1.49	91	4.53	162	4.91	263	10.99	320
vol	1M	13.33	931	33.78	1605	43.04	2346	88.83	3128

Table 9. Table 9. Performance evaluation on real datasets.

	SDI-RS+BFS	SDI-RS+DFS	BNL	SaLSa	SFS
Dominance	680,388	662,832	8,989,690	6,592,178	8,989,690
Search Time (msec)	54	38	151	108	147
Total Time (msec)	172	158	191	152	189
Stop Line Update	15	32	–	–	–

Equations

$$S_p={\mathrm{argmin}}_p(max\{o_i(p)\},\sum _{i=1}^d{o}_i(p)/d)$$

$$\mathcal{O}(\frac{n(n-1)}{2})$$

$$\mathcal{O}(k^2+\frac{(n-k)(n-k-1)}{2})$$

Full text

1. Introduction

Skyline computation aims at looking for the set of tuples that are not worse than any other tuples in all dimensions with respect to given criteria from a multidimensional database. Indeed, the formal concept of skyline was first proposed in 2001 by extending SQL queries to find interesting tuples with respect to multiple criteria (Borzsony2001Operator, ), with the notion of dominance: we say that a tuple $t$ dominates another tuple $t^{\prime}$ if and only if for each dimension, the value in $t$ is better than the respective value in $t^{\prime}$. The predicate better can be defined by any total order, such as less than or greater than.

For instance, Figure 1 shows the most mentioned example in skyline related literature where we consider the prices of hotels with respect to their distances to the city center (sometimes to the beach, or to the railway station, etc.). If we are interested in hotels which are not only cheap but also close to the city center (the less value is the better), those represented by $a$, $d$, $f$, $h$, and $i$ constitute the skyline. It’s obvious that the hotel $d$ dominates the hotel $b$ since $d$ is better than $b$ in both distance and price; however, $a$ does not dominate $b$ because $a$ is better than $b$ in price but $b$ is however better than $a$ in distance. In real-world database and user centric applications, such Price-Distance liked queries are doubtless interesting and useful, and have been widely recognized.

Since the first proposed BNL algorithm (Borzsony2001Operator, ), the skyline computation problem has been deeply studied for about two decades and many algorithms have been developed to compute the Skyline, such as Bitmap/Index (Tan2001IndexBitmap, ), NN (Kossmann2002NN, ), BBS (Papadias2005BBS, ), SFS (Jan2005SFS, ), LESS (Godfrey2005Less, ), SaLSa (Bartolini2006SaLSa, ), SUBSKY (Tao2007Subsky, ), ZSearch (Lee2007ZSearch, ), and ZINC (Liu2010ZINC, ). However, the efficiencies brought by existing algorithms often depend on either complex data structures or specific application/data settings. For instance, Bitmap is based on a bitmap representation of dimensional values but is also limited by the cardinality; Index is integrated into a B+-tree construction process; NN and BBS rely to a specific data structure as R-tree besides NN handles difficultly high-dimensional data (for instance $d>4$ (Tan2001IndexBitmap, )); SUBSKY specifically requires the B-tree and tuning the number of anchors; ZSearch and ZINC are built on top of the ZB-tree. On the other hand, as well as paralleling Skyline computation (chester2015scalable, ), different variants of standard skylines have been defined and studied, such as top-k Skylines (Tao2007Subsky, ), streaming data Skylines (lin2005stabbing, ), partial ordered Skylines (Liu2010ZINC, ), etc., which are not in our scope.

In this paper, we present the SDI (Skyline on Dimension Index) framework that allows efficient skyline computation by indexing dimensional values. We first introduce the notion of dimensional index, based on which we prove that in order to determine whether a tuple belongs the skyline, it is enough to compare it only with existing skyline tuples present in any one dimensional index instead of comparing it with all existing skyline tuples, which can significantly reduce the total count of dominance comparisons while computing the skyline. Furthermore, within the context of dimension indexing, we show that in most cases, one comparison instead of two is enough to confirm the dominance relation. These properties can significantly reduce the total count of dominance comparisons that is mostly the bottleneck of skyline computation. Different form all existing sorting/indexing based skyline algorithms, the application of dimension indexing allows to extend skyline computation to any total ordered categorical data such as, for instance, user preference on colors and on forms. Based on SDI, we also prove that any skyline tuple can be used to define a stop line crossing dimension indexes to terminate the skyline computation, which is in particular efficient on correlated data. We therefore develop the algorithm SDI-RS (RangeSearch) for efficient skyline computation with dimension indexing. Our experimental evaluation shows that SDI-RS outperforms our baseline algorithms (BNL, SFS, and SaLSa) in general, especially on high-dimensional data.

The remainder of this paper is organized as follows. Section 2 reviews related skyline computation approaches. In Section 3, we present our dimension indexing framework and prove several important properties, based on which we propose the algorithm SDI-RS in Section 4. Section 5 reports our experimental evaluation of the performance of SDI-RS in comparison with several baseline benchmarks. Finally, we conclude in Section 6.

2. Related Work

In this section, we briefly introduce mainstream skyline computation algorithms.

Börzsöny et al.(Borzsony2001Operator, ) first proposed the concept of skyline and several basic computation algorithms, of wihch Nested Loop (NL) is the most straightforward algorithm by comparing each pair of tuples, but always has the the same time complexity $O(n^{2})$ no matter the distribution of the data. Built on top of the naive NL algorithm, Block Nested Loop (BNL) algorithm employees memory window to speed up the efficiency significantly, and of which the best case complexity is reduced to $O(n)$ when there is no temporary file generated during the process of BNL, however the worst case is $O(n^{2})$, such as all tuples in database are incomparable with each other.

Bitmap and Index (Tan2001IndexBitmap, ) are two efficient algorithms for skyline computation. Bitmap based skyline computation is very efficient, however it limits to databases with limited distinct value of each dimension; it also consumes high I/O cost and requires large memory when the database is huge. Index generates the index based on the best value’s dimension of tuples. It is clear that skyline tuples are more likely to be on the top of each index table, so index tables can prune tuple if one tuple’s minimum value in all dimensions is larger than the maximal value of all dimensions of another tuple.

Sorted First Skyline (SFS) (Jan2005SFS, ) and Sort and Limit Skyline algorithm (SaLSa) (Bartolini2006SaLSa, ) are another two pre-sort based algorithms. SFS has a similar process as BNL but presorts tuples based on the skyline criteria before reading them into window. SaLSa shares the same idea as SFS to presort tuples, but the difference between SFS and SaLSa is that they use different approach to optimize the comparison passes: SFS uses entropy function to calculate the probability of one tuple being skyline and SaLSa uses stop point. Indeed, SaLSa is designed on top of such an observation: if a skyline tuple can dominate all unread tuples, then the skyline computation can be terminated. Such a special tuple is called the stop point in SaLSa, which can effectively prune irrelevant tuples that they cannot be in the Skyline. However, the selection of the stop point depends on dominance comparisons that is completely different from our notion of stop line, which is determined by dimensional indexes without dominance comparison.

SUBSKY algorithm (Tao2007Subsky, ) converts the $d$-dimensional tuples into 1D value $f(t)$ so all tuples will be sorted based on $f(t)$ value and that helps to determine whether a tuple is dominated by a skyline tuple. SUBSKY sorts the whole database on full space but calculates skyline on subspace based on user criteria. Nevertheless, the full space index may not be accurate when pruning data as the index maybe calculated on unrelated dimension. SDI also supports to calculate skyline on subspace but without re-sorting tuples. Moreover, dimension index could guarantee the best sorting of subspace and prune more tuples.

Besides sorting based algorithms, there are some algorithms solve the skyline computation problem using R-tree structure, such as NN (Nearest Neighbors) (Kossmann2002NN, ) and BBS (Branch-and-Bound Skyline) (Papadias2005BBS, ). NN discovers the relationships between nearest neighbors and skyline results. It is observed that the skyline tuple must be close to the coordinate origin: the tuple which stays closest to the coordinate origin must be a part of the skyline. Using the first skyline tuple, the database can be further split to several regions, and the first skyline tuple becomes the coordinate origin of these regions. The nearest point of each region are part of skyline tuples as well, so the whole process iterates until there is no more region split. BBS uses the similar idea as NN. The main difference between NN and BBS is that NN process may include redundant searches but BBS only needs one traversal path. NN and BBS are both efficient but nevertheless rely on complex data structure which is not necessary for SDI algorithm.

3. Dimension Indexing for Skyline Computation

We present in this section the SDI (Skyline on Dimension Index) framework, within which we prove several interesting properties that allow to significantly reduce the total count of dominance comparisons during the skyline computation.

Let $\mathcal{D}$ be a $d$-dimensional database that contains $n$ tuples, each tuple $t\in\mathcal{D}$ is a vector of $d$ attributes with $|t|=d$. We denote $t[i]$, for $1\leq i\leq d$, the dimensional value of a tuple $t$ in dimension $i$ (in the rest of this paper, we consider by default that $i$ satisfies $1\leq i\leq d$). Given a total order $\succ_{i}$ on all values in dimension $i$, we say that the value $t[i]$ of the tuple $t$ is better than the respective value $t^{\prime}[i]$ of the tuple $t^{\prime}$ if and only if $t[i]\succ_{i}t^{\prime}[i]$; if $t[i]=t^{\prime}[i]$, we say that $t[i]$ is equal to $t^{\prime}[i]$, and so that $t^{\prime}[i]$ is not worse than $t[i]$ if and only if $t[i]\succ_{i}t^{\prime}[i]\lor t[i]=t^{\prime}[i]$, denoted by $t[i]\succeq_{i}t^{\prime}[i]$. Besides, $t^{\prime}[i]$ is not better than $t[i]$ is denoted by $t[i]\not\succ t^{\prime}[i]$ We have that $t[i]\succ_{i}t^{\prime}[i]\Rightarrow t[i]\succeq_{i}t^{\prime}[i]$. Without lose of the generality, we denote by the total order $\succ$ the ensemble of all total orders $\succ_{i}$ on all dimensions and, without confusion, $\{\succ,\succeq,\not\succ\}$ instead of $\{\succ_{i},\succeq_{i},\not\succ_{i}\}$.

Definition 0 (Dominance).

Given the total order $\succ$ and a database $\mathcal{D}$, a tuple $t\in\mathcal{D}$ dominates a tuple $t^{\prime}\in\mathcal{D}$ if and only if $t[i]\succeq t^{\prime}[i]$ on each dimension $i$, and $t[k]\succ t^{\prime}[k]$ for at least one dimension $k$, denoted by $t\succ t^{\prime}$.

Further, we denote $t\prec\succ t^{\prime}\iff t\not\succ t^{\prime}\land t^{\prime}\not\succ t$ that the tuple $t$ and the tuple $t^{\prime}$ are incomparable. A tuple is a skyline tuple if and only if there is no tuple can dominate it. We therefore formally define skyline as follows.

Definition 0 (Skyline).

Given the total order $\succ$ and a database $\mathcal{D}$, a tuple $t\in\mathcal{D}$ is a skyline tuple if and only if $\not\exists u\in\mathcal{D}$ such that $u\succ t$. The skyline $\mathcal{S}$ on $\succ$ is the complete set of all skyline tuples that $\mathcal{S}=\{t\in\mathcal{D}\mid\not\exists u\in\mathcal{D},u\succ t\}$.

It’s easy to see that the skyline $\mathcal{S}$ of a database $\mathcal{D}$ is the complete set of all incomparable tuples in $D$, that is, $s\prec\succ t$ for any two tuples $s,t\in\mathcal{S}$.

Table 1 shows a sample database of 6 dimensions ($d=6$) that contains 10 tuples ($n=10$), of which 6 are skyline tuples ($|\mathcal{S}|=6$, we also note the size of Skyline as $m$ with reference to most literature) while the order less than is applied to all dimensions.

Example 0.

Among all the 10 tuples $t_{0},t_{1},\ldots,t_{9}$ listed in Table 1, $t_{1}\succ t_{8}$, $t_{4}\succ t_{2}$, $t_{6}\succ t_{7}$, and $t_{6}\succ t_{9}$; $t_{0}$, $t_{3}$, and $t_{5}$ do not dominate any tuples and are not dominated by any tuples. The Skyline is therefore $\mathcal{S}=\{t_{0},t_{1},t_{3},t_{4},t_{5},t_{6}\}$. ∎

The basis of our approach is to build dimensional indexes with respect to the concerned per-dimension total orders that allow to determine the skyline without performing dominance comparisons neither to all tuples in the database nor to all tuples in current skyline. In general, our approach can significantly reduce the total number of dominance comparisons, which plays an essential role that definitively affects the total processing time of Skyline computation. Furthermore, our approach constructs the Skyline progressively so no delete operation is required.

For each dimension $i$ of the database $\mathcal{D}$, the total order $\prec_{i}$ can be considered as a sorting function $f_{i}:\mathcal{D}[i]\rightarrow\mathcal{I}_{i}$, where $\mathcal{I}_{i}$ is an ordered list of all tuple values in the dimension $i$ of database. We call such a list $\mathcal{I}_{i}$ a dimensional index.

Definition 0 (Dimensional Index).

Given a database $\mathcal{D}$, the dimensional index $\mathcal{I}_{i}$ for a dimension $i$ is an ordered list of tuple IDs sorted first by dimensional values with respect to the total order $\succ$, and then, in case of ties, by their lexicographic order.

In order to avoid unnecessary confusions, we represent a dimensional index $\mathcal{I}_{i}$ as a list of entries $\left<t[i]:t.id\right>$ such as which shown in Table 2 (where all skyline tuples are in bold).

Example 0.

Table 2 shows the 6 dimensional indexes $\mathcal{I}_{1},\mathcal{I}_{2},\ldots,\mathcal{I}_{6}$ with respect to all the 6 dimensions $D_{I},D_{2},\ldots,D_{6}$ of the sample database shown in Table 1. We show in detail that in $\mathcal{I}_{1}$, the dimensional value 5.3 appears in 3 tuples so these 3 entries are secondarily sorted by tuple IDs for $3<6<7$. ∎

Now let us consider the dimensional indexes containing distinct dimensional values only, such as $\mathcal{I}_{4}$ shown in Table 2. In such an index $\mathcal{I}_{i}$ without duplicate dimensional values, we see that a tuple $t$ can only be dominated by a tuple $s$ such that $o_{i}(s)<o_{i}(t)$ (implies that $s[i]<t[i]$ since for any tuple $u$ such that $o_{i}(u)>o_{i}(t)$, we have that $t[i]\succ u[i]$ so $u$ cannot dominates $t$.

Lemma 0.

Given a database $\mathcal{D}$, let $\mathcal{S}$ be the skyline of $\mathcal{D}$, $\mathcal{I}_{i}$ be a dimensional index containing only distinct dimensional values, and $t\in\mathcal{D}$ be a tuple. Then, $t\in\mathcal{S}$ if and only if we have $s\not\succ t$ for any skyline tuple $s\in\mathcal{S}$ such that $o_{i}(s)<o_{i}(t)$ on $\mathcal{I}_{i}$. ∎

Proof.

If $o_{i}(t)=0$, then $t$ is a skyline tuple because no tuple is better than $t$ in the dimension $D_{i}$ since all dimensional values are distinct. If $o_{i}(t)>0$, let $s\in\mathcal{S}$ be a skyline tuple such that $o_{i}(s)<o_{i}(t)$, then $s[i]<t[i]$, thus, $s\not\succ t\Rightarrow\exists l\neq i$ such that $t[l]\succ s[l]$, that is, $s\prec\succ t$; now let $s^{\prime}\in\mathcal{S}$ be a skyline tuple such that $o_{i}(t)<o_{i}(s^{\prime})$, then $s^{\prime}\in\mathcal{S}\Rightarrow t\not\succ s^{\prime}$, further, $t[i]\prec_{i}s^{\prime}[i]\Rightarrow s^{\prime}\not\succ s^{\prime}$, so we also have $t\prec\succ s^{\prime}$. Thus, $t$ is incomparable to any skyline tuple so $t$ is a skyline tuple, that is, $t\in\mathcal{S}$. ∎

With Lemma 6, to determine whether a tuple $t$ is a skyline tuple, it is only necessary to compare $t$ with each skyline tuple $s$ in one dimension $i$ such that $o_{i}(s)<o_{i}(t)$, instead of comparing $t$ with all skyline tuples. Furthermore, we recall that BNL-like algorithms dynamically update the early skyline set that require a second dominance comparison between an incoming tuple $t$ and early skyline tuple $s$ to determine whether $t\succ s$. However, with dimensional indexes, Lemma 6 shows that one dominance comparison $s\not\succ t$ is enough to determine $t\in\mathcal{S}$, instead of two comparisons. Lemma 6 also ensures a progressive construction of the skyline.

However, in most cases and particularly in real data, there are often duplicate values in each dimension where Lemma 6 cannot be established. As shown in Table 2, we can find that there are duplicate values in most dimensions, where a typical instance is $\mathcal{I}_{1}$, in which two different cases should be identified:

1. (1)

The dimensional value 5.3 appears in three entries $\left<5.3:3\right>$, $\left<5.3:6\right>$, and $\left<5.3:7\right>$ where $t_{3}$ and $t_{6}$ are skyline tuples and $t_{7}$ is not skyline tuple. 2. (2)

The dimensional value 8.4 appears in both of the two entries $\left<8.4:2\right>$ and $\left<8.4:4\right>$, where $t_{2}$ is not skyline tuple but is indexed before the skyline tuple $t_{4}$.

In the case (1), a simple straightforward scan on these three dimensional index entries can progressively identify that $t_{3}$ ($t_{1}\not\succ t_{3}$) and $t_{6}$ ($t_{1}\not\succ t_{6}$ and $t_{3}\not\succ t_{6}$) are skyline tuples and filter out $t_{7}$ ($t_{6}\succ t_{7}$). However, in the case (2), a straightforward scan cannot progressively identify skyline tuples because: there is no precedent tuple dominating $t_{2}$ so $t_{2}$ will be first identified as a skyline tuple; then, since no tuple can dominate $t_{4}$, $t_{4}$ will identified as a skyline tuple without checking whether $t_{4}\not\succ t_{2}$, hence, finally the output skyline is wrong.

To resolve such misidentifications of skyline tuples, we propose a simple solution that first divides a dimensional index into different logical blocks of entries with respect to each distinct dimensional value, then apply the BNL algorithm to each block containing more than one entry to find block skyline tuples in order to establish Lemma 6.

Definition 0 (Index Block).

Given a database $\mathcal{D}$, let $\mathcal{I}_{i}$ be the dimensional index of a dimension $i$. An index block of $\mathcal{I}_{i}$ is a set of dimensional index entries that share the same dimensional value sorted by the lexicographical order of tuple IDs.

If each block contains one entry, the only tuple will be compared with existing skyline tuples with respect to Lemma 6; otherwise, for any block contains more than one entry, each block skyline tuple must be compared with existing skyline tuples with respect to Lemma 6. We can generalize the notion of tuples in Lemma 6 to block skyline tuples because one block contains one entry, the concerned tuples are block skyline tuples.

Theorem 8.

Given a database $\mathcal{D}$, let $\mathcal{S}$ be the Skyline of $\mathcal{D}$, $\mathcal{I}_{i}$ be a dimensional index, and $t\in\mathcal{D}$ be a block skyline tuple on $\mathcal{I}_{i}$. Then, $t\in\mathcal{S}$ if and only if we have $s\not\succ t$ for any skyline tuple $s\in\mathcal{S}$ such that $o_{i}(s)<o_{i}(t)$ on $\mathcal{I}_{i}$.

Proof.

With the proof of Lemma 6 and the statement of block skyline tuples, the proof of Theorem 8 is immediate. ∎

Example 0.

As shown in Table 3, 6 blocks can be located from $\mathcal{I}_{1}$ with respect to all 6 distinct values: 4.7, 5.3, 6.7, 7.5, 8.4, and 9.1. According to Theorem 8: the block 4.7 contains $t_{1}$, so $t_{1}$ is a block skyline tuple and is the first skyline tuple; the block 5.3 contains $t_{3}$, $t_{6}$, and $t_{7}$ where $t_{3}\prec\succ t_{6}$ and $t_{6}\succ t_{7}$, so $t_{3}$ and $t_{6}$ block skyline tuples such that $t_{1}\prec\succ t_{3}$ and $t_{1}\prec\succ t_{6}$, hence, $t_{3}$ and $t_{6}$ are new skyline tuples; the block 6.7 contains $t_{8}$, so $t_{8}$ is a block skyline tuple that is dominated by $t_{6}$; the block 7.5 is different from the block 5.3, where $t_{0}\prec\succ t_{9}$ so both of them are block skylines, and we have $t_{6}\succ t_{9}$ so $t_{0}$ is a skyline tuple; the block 8.3 is the same case as the block 5.3, where $t_{4}$ is a skyline tuple; finally, no skyline tuple dominates $t_{5}$, so the Skyline is $\{t_{0},t_{1},t_{3},t_{4},t_{5},t_{6}\}$. ∎

It is important to note that Theorem 8 allows dominance comparisons to be performed on arbitrary dimensional indexes and the computation stops while the last entry in any index is reached. Therefore, we see that a dynamic dimension switching strategy can further improve the efficiency of the Skyline computing based on dimension indexing. For instance, if we proceed a breadth-first search strategy among all dimensional indexes shown in Table 2, while we examine the second entry $\left<2.6:5\right>$ in $\mathcal{I}_{3}$, although currently $\mathcal{S}=\{t_{0},t_{1},t_{6},t_{3},t_{4}\}$, we do not have to compare $t_{5}$ with all those skyline tuples but only with $t_{6}$; if we continue to examine the second entry in $\mathcal{I}_{4}$, $t_{5}$ can be ignored since it is already a skyline tuple. We also note that duplicate dimensional values present in tuples severely impact the overall performance of dimensional index based Skyline computation, therefore, reasonable dimension selection/sorting heuristics shall be helpful.

4. A Range Search Approach to Skyline

In this section, we first propose the notion of stop line that allows terminate searching skyline tuples by pruning non relevant tuples, then present the algorithm SDI-RS (RangeSearch) for skyline computation based on the SDI framework. Notice that the name RangeSearch stands for the bounded search range while determining skyline tuples.

4.1. Stop Line

Let us consider again the Skyline and the dimensional indexes shown in Table 2. It is easy to see that all 6 skyline tuples can be found at the first two entries of all dimensional indexes, hence, a realistic question is whether we can stop the Skyline computation before reaching the end of any dimensional index.

Definition 0 (Stop Line).

Given a database $\mathcal{D}$, let $p\in\mathcal{D}$ be a skyline tuple. A stop line established from $p$, denoted by $S_{p}$, is a set of dimensional index entries $\left<p[i]:p\right>$ such that $p$ appears in each dimension. An index entry $e\S_{p}$ is a stop line entry and an index block containing a stop line entry is a stop line block.

Let $t$ be a tuple, we denote $b_{i}(t)$ the offset of the index block on a dimensional index $\mathcal{I}_{i}$ that contains $t$, that is, the position of the index block that contains $t$. Hence, let $p$ be a stop line tuple and $t$ be a tuple, we say that the stop line $S_{p}$ covers the index entry $\left<t[i]:t\right>$ on a dimensional index $\mathcal{I}_{i}$ if $b_{i}(p)<b_{i}(t)$. For instance, Table 4 shows the stop line created from the tuple $t_{6}$, which totally covers 41 index entries without $\left<5.3:7\right>$ on $\mathcal{I}_{1}$ neither $\left<7.5:7\right>$ on $\mathcal{I}_{2}$.

Obviously, let $p$ be a stop line tuple and $t$ be a tuple such that $p\prec t$, then we have that $b_{i}(p)\leq b_{i}(t)$ on each dimensional index $\mathcal{I}_{i}$ and $b_{k}(p)<b_{k}(t)$ on at least one dimensional index $\mathcal{I}_{k}$.

Theorem 2.

Given a database $\mathcal{D}$, let $S_{p}$ be the stop line with respect to a skyline tuple $p$. By following any top-down traversal of all dimensional indexes, if all stop line blocks have been traversed, then the complete set of all skyline tuples has been generated and the skyline computation can stop.

Proof.

Let $p$ be a skyline tuple and $t\in\mathcal{S}\setminus p$ be a skyline tuple, we have (1) $t\prec\succ p$ or (2) $t=p$, if $t$ and $p$ have identical dimensional values. In the first case, $t\prec\succ p\Rightarrow\exists k,p[k]\succ t[k]\Rightarrow b_{k}(p)<b_{k}(t)$, that is, if the index traversal passes the stop line $S_{p}$, the tuple $t$ must have been identified at least in the dimension $D_{k}$. In the second case, we have $b_{i}(p)=b_{i}(t)$ for any dimension $D_{i}$. In both cases, if all stop line blocks have been processed, then all skyline tuples have been found. ∎

In principle, any skyline tuple can be chosen to form a stop line, however, different stop lines behave differently in pruning useless tuples. For instance, as shown in Table 4, the stop line $S_{6}$ created from $t_{6}$ covers totally 41 index entries and two tuples $\{t_{7},t_{9}\}$ can be pruned; however, as shown in Table 5, the the stop line $S_{0}$ created from $t_{0}$ covers only 37 index entries and no tuple can be pruned. Obviously, a good stop line shall cover index entries at much as possible, so we can use an optimal function, $min_{p}$, to minimize the offsets of a skyline tuple in all dimensional indexes for building a stop line $S_{p}$, defined as:

[TABLE]

The function min(p) sorts tuples first by the maximum offset, then by the mean offset in all dimensional indexes, so the minimized skyline tuple is the best stop line tuple. Hence, a dynamically updated stop tuple $p$ can be maintained by keeping $min(p)<min(t)$ for any new skyline tuple $t$.

Nevertheless, the use of stop lines requires that all stop line blocks in all dimensions being examined, so it is difficult to judge whether a scan reaches first at the end of any dimensional index or first finishes to examine all stop line blocks although we can state that the setting of stop lines can effectively help the Skyline computation in correlated data. We also note that the use of stop lines require that all dimensions are indexed, which is an additional constraint while applying Theorem 8 and Theorem 2 together since Theorem 8 does not impose that all dimensions must be constructed. We propose, thus, to consider different application strategies of Theorem 8 and Theorem 2 with respect to particular use cases and data types to accelerate the Skyline computation.

4.2. The RangeSearch Algorithm

Theorem 8 allows to reduce the count of dominance comparisons while computing the skyline. However, as mentioned in Section 3, the duplicate dimensional values severely augment the dominance comparisons count because a BNL based local comparisons must be applied. Notice that it is useless to apply SFS or SaLSa to such local comparisons because their settings of sorting functions disable one of the most important features of our dimension indexing based approach: individual criterion including that for non-numerical values of skyline selection can be independently applied to each dimension.

In order to reduce the impact of duplicate dimensional values, we propose a simple solution based on sorting dimensional indexes by their cardinalities $|\mathcal{I}_{i}|$. The computation starts from the best dimensional index so the calls of BNL can be minimized. For instance, in Table 2, all dimensional indexes can be sorted as $|\mathcal{I}_{4}|>|\mathcal{I}_{2}|=|\mathcal{I}_{5}|=|\mathcal{I}_{6}|>|\mathcal{I}_{3}|>|\mathcal{I}_{1}|$, where the best dimensional index $\mathcal{I}_{4}$ contains no duplicate dimensional values so Lemma 6 can be directly established so dimension switching can be performed earlier.

We present then SDI-RS (RangeSearch), an algorithm with the application of Theorem 8 and Theorem 2 by performing dominance comparisons only with a range of skyline tuples instead of all, as shown in Algorithm 1, to the skyline computation on sorted dimensional indexes.

The algorithm accepts a set of sorted dimensional indexes $\mathcal{I}_{\mathcal{D}}$ of a $d$-dimensional database $\mathcal{D}$ as input and outputs the complete set $\mathcal{S}$ of all skyline tuples. First, we initialize an empty stop line $L$, then we enter a Round Robin loop that find the complete set of all skyline tuples with respect to Theorem 8 and Theorem 2. In each dimensional index $\mathcal{I}_{i}$ based iteration, we first get the next block $B$ of index entries from $\mathcal{I}_{i}$. According to Theorem 8, if $B$ is null, which means that the end of $\mathcal{I}_{i}$ is reached, we exit the algorithm by returning $\mathcal{S}$; otherwise, we treat all index entries block by block to find skyline tuples. If a tuple $t\in B$ is already compared and marked as non skyline tuple, we should ignore it in order to prevent comparing it with other tuples again; however, if $t$ is a skyline tuple, we shell keep it because $t$ may dominate other new tuples in block-based BNL while computing the block Skyline $\mathcal{S}_{B}$. Therefore, for each tuple $t\in\mathcal{S}_{B}$ such that $t\not\in\mathcal{S}$ (again, we do not want to compare a skyline tuple with other skyline tuples), we compare it with all existing skyline tuples $\mathcal{S}_{i}$ present in current dimension $D_{i}$. Here we introduce a shortcut operator $\mathcal{S}_{i}\not\prec t$ at line 12 that means that none of skyline tuples in $\mathcal{S}_{i}$ dominates $t$, and according to Theorem 8, $t$ must be a skyline tuple in this case and must be added to the dimensional Skyline $\mathcal{S}_{i}$ and the global Skyline $\mathcal{S}$. Furthermore, with respect to Theorem 2, we build a new stop line $L_{t}$ from each new skyline tuple $t$ and if it is better than current stop line $L$ (or no stop line is defined), we update $L$ by $L_{t}$. While the above dominance comparisons are finished, we compare current dimensional iteration position on all dimensions with the latest stop line, if in each dimension the stop line entry is reached, RangeSearch stops by returning the complete Skyline $\mathcal{S}$. Otherwise, RangeSearch switch to the next dimension and repeat the above procedure with respect to a particular [dimension-switching] strategy.

In our approach, we consider breadth-first dimension switching (BFS) and depth-first dimension switching (DFS). With BFS, if a block is examined and if SDI-RS shall continue to run, then the next dimension will be token. However, in depth-first switching, if a block is examined and if SDI-RS shall continue to run, SDI-RS continues to go ahead in current dimension if current block contains new skyline tuples, till to meet a block without any new skyline tuple. The difference between breadth-first switching and depth-first switching is clear. DFS tries to accelerate skyline tuple searching in each dimension, this strategy benefits the most from Theorem 8; furthermore, if the best stop line is balanced in each dimension, then DFS reaches well the stop line in each dimension so more tuples can be pruned. However, DFS is not efficient if there are a large number of duplicate values in some dimensions because each block shall be examined before switching to the next dimension. In this case, depth-first switching takes duplicate dimensional values into account: since all dimensional indexes are sorted with respect to their cardinalities, SDI-RS starts always from the best dimensions that contain less duplicate values and finds skyline tuples as much as possible by depth-first switching, hence, while switching to other dimensions, it is possible that some tuples in some blocks have already been compared or are already skyline tuples so no more comparisons will be performed.

In comparison with sorting based algorithms like SFS and SaLSa, SDI-RS allows to sort tuples with respect to each dimension, which is interesting while different criteria are applied to determine the skyline. For instance, we can specify the order less than ($<$) a one dimension and the order greater than ($>$) to another dimension, without of additional calculation to unifying and normalizing dimensional values. With the same reason, SDI-RS allows to directly process categorical data as numerical data: if any total order can be defined to a categorical attribute, for instance, the user preference on colors such that blue $\succ$ green $\succ$ yellow $\succ$ red, then SDI-RS can treat such values as any ordered numerical values without any adaptation.

With dimensional indexes, SDI-RS is efficient in both space and time complexities. The storage requirement for dimensional indexes is guaranteed: for instance, a C/C++ implementation of SDI-RS may consider an index entry as a struct of tuple ID and dimensional value that requires 16 bytes (64bit ID and 64bit value), therefore if each dimensional index corresponds to a std::vector structure, the in-memory storage size of dimensional indexes is the double of the database size: for instance, 16GB heap memory fits the allocation of 1,000,000,000 structures of ID/value, as 12,500,000 8-dimensional tuples. Let $d$ be the dimensionality, $n$ be the cardinality of data, and $m$ be the size of the skyline. The generation of dimensional indexes requires $\mathcal{O}(dn\lg{n})$ with respect to a general-purpose sorting algorithm of $\mathcal{O}(n\lg{n})$ complexity. For the best-case, that is, $m=1$, SDI-RS finishes in $\mathcal{O}(1)$ since the only skyline tuple is the stop line and the computation stops immediately; for the worst-case, all $n$ tuples are skyline tuples, SDI-RS finishes in

[TABLE]

according to Theorem 8 if each block contains only one tuple (that is, the case without duplicate dimensional values). More generally, if the best dimension index contains $k$ duplicate values, then SDI-RS finishes in

[TABLE]

since the worst-case is that all $k$ duplicate values appear in the same block.

5. Experimental Evaluation

In this section, we report our experimental results on performance evaluation of SDI-RS that is conducted with both of BFS and DFS dimension switching, and is compared with three baseline algorithms BNL, SFS, and SaLSa on synthetic and real benchmark datasets. The vol sorting function and the max sorting function are respectively applied to SFS and SaLSa as mentioned in (Bartolini2006SaLSa, ).

We generate independent, correlated, and anti-correlated synthetic datasets using the standard Skyline Benchmark Data Generator111http://pgfoundry.org/projects/randdataset (Borzsony2001Operator, ) with the cardinality $n\in\{100K,1M\}$ and the dimensionality $d$ in the range of 2 to 24. Three real datasets NBA, HOUSE, and WEATHER (chester2015scalable, ) have also been used. Table 6 and Table 7 show statistics of all these datasets.

We implemented SDI-RS in C++ with C++11 standard, where dimensional indexes were implemented by STL std::vector and std::sort(). In order to to evaluate the overall performance of our SDI-RS, three baseline algorithms BNL, SFS, and SaLSa were also implemented in C++ with the same code-base. All algorithms are compiled using LLVM Clang with -O3 optimization flag. All experiments have been performed on a virtual computation node with 16 vCPU and 32GB RAM hosted in a server with 4 Intel Xeon E5-4610 v2 2.30GHz processors and 256GB RAM.

Figure 2 shows the overall run-time, including loading/indexing data, and the total dominance comparison count of SDI-RS and BNL/SFS/SaLSa on 100K and 1M datasets, where the dimensionality is set to 2, 4 6, 8, 16, and 24. We note that in the case of low-dimensional datasets, such as $d\leq 8$, there are no very big differences between all these 4 algorithms; however, SDI-RS extremely outperforms BNL/SFS/SaLSa in high-dimensional datasets, for instance $d\geq 16$. Indeed, the run-time of SDI-RS is almost linear with respect to the increase of dimensionality, which is quite reasonable since the main cost in skyline computation is dominance comparison and SDI-RS allows to significantly reduce the total count of dominance comparisons. On the other hand, it is surprising that SaLSa did not finish computing on all 24-dimensional datasets as Figure 2 (b) and Figure 2 (j), for more than 5 hours. Notice that SaLSa outperforms BNL and SFS on real datasets.

We note that the total run-time of SDI-RS on low-dimensional correlated datasets is much than BNL/SFS/SaLSa as regards independent and anti-correlated datasets, because SDI-RS requires building dimensional indexes. Table 8 details the skyline searching time $t_{S}$ (the time elapsed on dominance comparisons and data access in msec) and total run-time $t_{T}$ (the time elapsed on the whole process, including data loading and sorting/indexing in msec). It is clear that the construction of dimensional indexes in SDI-RS is essential while the total processing time is short.

Table 9 shows the performance of SDI-RS on real datasets. DFS dimension switching outperforms BFS dimension switching on both NBA and HOUSE datasets however BFS outperforms DFS on WEATHER dataset. After having investigated these datasets, we confirm that there are a large number of duplicate values in several dimension of WEATHER dataset so the BFS dimension switching strategy takes its advantage. BNL outperforms all other tested algorithms on HOUSE dataset, which corresponds to the results obtained from synthetic low-dimensional independent datasets. Furthermore, the update numbers of the best stop line in SDI-RS is quite limited with respect to the size of skylines.

We did not directly compare SDI-RS with all existing skyline algorithms, but with reference to most literature comparing proposed algorithms with BNL, SFS, or SaLSa, the comparative results obtained in our experimental evaluation indicate that SDI-RS outperforms the most of existing skyline algorithms.

6. Conclusion

In this paper, we present a novel efficient skyline computation approach. We proved that in multidimensional databases, skyline computation can be conducted on an arbitrary dimensional index which is constructed with respect to a predefined total order that determines the skyline, we therefore proposed a dimension indexing based general skyline computation framework SDI. We further showed that any skyline tuple can be used to stop the computation process by outputting the complete skyline. Based on our analysis, we developed a new progressive skyline algorithm SDI-RS that first builds sorted dimensional indexes then efficiently finds skyline tuples by dimension switching in order to minimize the count of dominance comparisons. Our experimental evaluation shows that SDI-RS outperforms the most of existing skyline algorithms. Our future research direction includes the further development of the SDI framework as well as adapting the SDI framework to the context of Big Data, for instance with the Map-Reduce programming model.