K-Dominant Skyline Join Queries: Extending the Join Paradigm to K-Dominant Skylines

TL;DR

This paper introduces KSJQ, a new approach for k-dominant skyline queries on joined relations, improving efficiency and scalability in multi-criteria, multi-attribute decision-making scenarios involving complex joins.

Contribution

It extends k-dominant skyline queries to the join paradigm, proposing algorithms for efficient query processing and k-value selection, with practical handling of aggregated attributes.

Findings

Pre-processing reduces query response time.

Algorithms effectively select k for desired skyline size.

Experiments show high scalability and efficiency.

Abstract

Skyline queries enable multi-criteria optimization by filtering objects that are worse in all the attributes of interest than another object. To handle the large answer set of skyline queries in high-dimensional datasets, the concept of k-dominance was proposed where an object is said to dominate another object if it is better (or equal) in at least k attributes. This relaxes the full domination criterion of normal skyline queries and, therefore, produces lesser number of skyline objects. This is called the k-dominant skyline set. Many practical applications, however, require that the preferences are applied on a joined relation. Common examples include flights having one or multiple stops, a combination of product price and shipping costs, etc. In this paper, we extend the k-dominant skyline queries to the join paradigm by enabling such queries to be asked on joined relations. We call such queries KSJQ (k-dominant skyline join queries). The number of skyline attributes, k, that an object must dominate is from the combined set of skyline attributes of the joined relation. We show how pre-processing the base relations helps in reducing the time of answering such queries over the naive method of joining the relations first and then running the k-dominant skyline computation. We also extend the query to handle cases where the skyline preference is on aggregated values in the joined relation (such as total cost of the multiple legs of the flight) which are available only after the join is performed. In addition to these problems, we devise efficient algorithms to choose the value of k based on the desired cardinality of the final skyline set. Experiments on both real and synthetic datasets demonstrate the efficiency, scalability and practicality of our algorithms.

Tables

Table 1. Table 1: Flights from city A ( f 1 subscript 𝑓 1 f_{1} ).

fno	destination	cost	dur	rtg	amn	category
11	C	448	3.2	40	40	$S{S}_1$
12	C	468	4.2	50	38	$N{N}_1$
13	D	456	3.8	60	34	$S{N}_1$
14	D	460	4.0	70	32	$N{N}_1$
15	E	450	3.4	30	42	$S{N}_1$
16	F	452	3.6	20	36	$S{S}_1$
17	G	472	4.6	80	46	$S{N}_1$
18	H	451	3.7	20	37	$S{S}_1$
19	E	451	3.7	40	37	$N{N}_1$

Table 2. Table 2: Flights to city B ( f 2 subscript 𝑓 2 f_{2} ).

fno	source	cost	dur	rtg	amn	category
21	D	348	2.2	40	36	$S{S}_2$
22	D	368	3.2	50	34	$N{N}_2$
23	C	356	2.8	60	30	$S{N}_2$
24	C	360	3.0	70	28	$N{N}_2$
25	E	350	2.4	30	38	$S{N}_2$
26	F	352	2.6	20	32	$S{S}_2$
27	G	372	3.6	80	42	$S{N}_2$
28	H	350	2.4	35	37	$S{N}_2$

Table 3. Table 3: Joined relation ( f 1 ⨝ f 2 join subscript 𝑓 1 subscript 𝑓 2 f_{1}\Join f_{2} ).

fno	stop-over	f1.cost	f1.dur	f1.rtg	f1.amn	f2.cost	f2.dur	f2.rtg	f2.amn	categorization	skyline
(11,23)	C	448	3.2	40	40	356	2.8	60	30	$S{S}_1⨝S{N}_2$	yes
(11,24)	C	448	3.2	40	40	360	3.0	70	28	$S{S}_1⨝N{N}_2$	no
(12,23)	C	468	4.2	50	38	356	2.8	60	30	$N{N}_1⨝S{N}_2$	no
(12,24)	C	468	4.2	50	38	360	3.0	70	28	$N{N}_1⨝N{N}_2$	no
(13,21)	D	456	3.8	60	34	348	2.2	40	36	$S{N}_1⨝S{S}_2$	yes
(13,22)	D	456	3.8	60	34	368	3.2	50	34	$S{N}_1⨝N{N}_2$	no
(14,21)	D	460	4.0	70	32	348	2.2	40	36	$N{N}_1⨝S{S}_2$	no
(14,22)	D	460	4.0	70	32	368	3.2	50	34	$N{N}_1⨝N{N}_2$	no
(15,25)	E	450	3.4	30	42	350	2.4	30	38	$S{N}_1⨝S{N}_2$	yes
(16,26)	F	452	3.6	20	36	352	2.6	20	32	$S{S}_1⨝S{S}_2$	yes
(17,27)	G	472	4.6	80	46	372	3.6	80	42	$S{N}_1⨝S{N}_2$	no
(18,28)	H	451	3.7	20	37	350	2.4	35	39	$S{S}_1⨝S{N}_2$	no
(19,25)	E	451	3.7	40	37	350	2.4	30	38	$N{N}_1⨝S{N}_2$	no

Table 4. Table 4: Fate of k 𝑘 k -dominant skylines.

	$S{S}_2$	$S{N}_2$	$N{N}_2$
$S{S}_1$	yes (Th. 1)	likely (Obs. 1)	no (Th. 2)
$S{N}_1$	likely (Obs. 1)	may be (Obs. 2)	no (Th. 2)
$N{N}_1$	no (Th. 2)	no (Th. 2)	no (Th. 2)

Table 5. Table 5: Joins of groups.

	$S{S}_2$	$S{N}_2$	$N{N}_2$
$S{S}_1$	yes (Th. 4)	likely (Obs. 3)	no (Th. 5)
$S{N}_1$	likely (Obs. 3)	may be (Obs. 4)	no (Th. 5)
$N{N}_1$	no (Th. 5)	no (Th. 5)	no (Th. 5)

Table 6. Table 6: Joined relation ( f 1 ⨝ f 2 join subscript 𝑓 1 subscript 𝑓 2 f_{1}\Join f_{2} ): aggregate.

fno	stop-over	cost	f1.dur	f1.rtg	f1.amn	f2.dur	f2.rtg	f2.amn	categorization	skyline
(11,23)	C	804	3.2	40	40	2.8	60	30	$S{N}_1⨝S{N}_2$	yes
(11,24)	C	808	3.2	40	40	3.0	70	28	$S{N}_1⨝N{N}_2$	no
(12,23)	C	824	4.2	50	38	2.8	60	30	$N{N}_1⨝S{N}_2$	no
(12,24)	C	828	4.2	50	38	3.0	70	28	$N{N}_1⨝N{N}_2$	no
(13,21)	D	804	3.8	60	34	2.2	40	36	$S{N}_1⨝S{N}_2$	yes
(13,22)	D	824	3.8	60	34	3.2	50	34	$S{N}_1⨝N{N}_2$	no
(14,21)	D	808	4.0	70	32	2.2	40	36	$N{N}_1⨝S{N}_2$	no
(14,22)	D	828	4.0	70	32	3.2	50	34	$N{N}_1⨝N{N}_2$	no
(15,25)	E	800	3.4	30	42	2.4	30	38	$S{N}_1⨝S{N}_2$	yes
(16,26)	F	804	3.6	20	36	2.6	20	32	$S{S}_1⨝S{S}_2$	yes
(17,27)	G	844	4.6	80	46	3.6	80	42	$S{N}_1⨝S{N}_2$	no
(18,28)	H	801	3.7	20	37	2.4	35	39	$S{S}_1⨝S{N}_2$	no
(19,25)	E	801	3.7	40	37	2.4	30	38	$N{N}_1⨝S{N}_2$	no

Table 7. Table 7: Parameters for experiments.

Symbol	Parameter	Default value
$n$	Dataset size for base relation	$3,300$
$d$	Dimensionality of base relation	$7$
$k$	Number of skyline attributes	$11$
$a$	Number of aggregate attributes	$2$
$g$	Number of join groups	$10$
$T$	Dataset type	Independent
$\delta$	Threshold of skyline size	$10,000$
$N$	Size of joined relation	$1,089,000$

Equations

$$R_1$$

$$R_2$$

$$R$$

$$R_i=S{S}_i\cup S{N}_i\cup N{N}_i$$

$$k_1^{\prime }=k-d_2{k}_2^{\prime }=k-d_1$$

Full text

1 Introduction and Motivation

Skyline queries are widely used to enable multi-criteria decision making in databases [3]. Consider a scenario where a person wants to buy a good house. Her preferences are low cost, proximity to market, quiet neighborhood, etc. In a real scenario, it is almost impossible to find a single house that is best in all her preferences. The skyline query helps her narrow down the choices by filtering out houses that are worse (or equal) than some other house in all the preferences. Assuming rationality, her choices cannot lie outside the skyline set.

In high-dimensional spaces, however, the skyline set becomes less useful due to its impractically large size. The size tends to increase exponentially with dimensionality [28]. This happens as it becomes harder for any object to dominate another object in all the attributes. The problem is especially severe in real datasets where the data is generally anti-correlated in nature. For example, a quiet neighborhood closer to a market is likely to be more costly.

There have been various works on handling the large cardinality of skyline sets in high dimensions. The most prominent is that of $k$-dominant skylines where, instead of being better in all the $d$ dimensions, an object need only be better in some $k<d$ dimensions to dominate another object [4]. As a result, it becomes easier for an object to be dominated which leads to lesser number of skylines.

The $k$-dominant skylines are quite useful in real scenarios. In the example of housing discussed above, if there are many attributes, it may be rare that one house is better than another on all the counts. Instead, if a smaller number of attributes, say $k=2$, is specified, there are more chances of finding a house that has a lower cost and a quieter neighborhood (but may not be closer to a market) than another house. As a result, more houses can be filtered, and the retrieved skyline set becomes more manageable and useful.

To the best of our knowledge, however, the $k$-dominant skyline queries have not been explored for multiple relations.111A preliminary version of this paper will appear as a poster [1]. Suppose there are two relations having $d_{1}$ and $d_{2}$ skyline attributes. After joining, a bigger relation with $d_{1}+d_{2}$ skyline attributes is formed. (We discuss the different variants and restrictions later.) A $k$-dominant skyline, where $k<d_{1}+d_{2}$, is then sought on this joined relation.

A real-life example of this situation happens often in flight bookings. Suppose a person wants to fly from city A to city B. Her preferences are lower cost, lower duration, higher ratings and higher amenities. While the basic skyline works for direct flights, in many cases, a flight route from A to B includes one (or more) stopovers. Thus, a valid flight path contains the join of all flights from city A to other cities and from those cities to city B where the intermediate city is the same. The preferences a user would want now applies to the entire flight path and not a single leg of the journey. The skyline is, therefore, needed on the joined relation [2, 21].

Once more, it is harder for a flight combination to dominate another flight combination in all the skyline attributes over the two relations. Here, the $k$-dominant skyline query is a natural choice, where $k$ is less than the total number of the skyline attributes in the joined relation.

With the increase in the number of attributes, the size of the skyline set increases further and, hence, computing the $k$-dominant skylines becomes even more relevant and useful.

The naïve approach first creates the joined relation and then subsequently computes the $k$-dominance. This strategy, while straightforward, is inefficient and impractical for large datasets.

A further practical consideration in the flight example is that a user is not really bothered about the individual legs, but rather the total cost and total duration of the journey. Thus, the skyline preferences should be applied on the aggregated values of attributes from the base relations. Note that the aggregated values are available only after the join and are, therefore, harder to process efficiently.

In this paper, we explore the question of finding $k$-dominant skylines on joined relations, where the skyline preferences can be on both aggregated and individual values. Apart from posing the problem, our main contribution is to push the skyline operator before the join as much as possible, thereby making the whole algorithm efficient and practical.

In addition to finding $k$-dominant skylines, an important question that often arises in practical applications is how to choose a “good” value of $k$? While there is no universal answer, one of the guiding principles is the number of skyline objects finally returned [4]. The basic idea of skyline queries is to serve as a filter for poor objects and, thus, a user may find it easier to specify a value of $\delta$ objects that she is interested in examining more thoroughly rather than a value of $k$. Since the size of the skyline set increases with $k$, the “optimal” value of $k$ may be then taken as the smallest one that returns at least $\delta$ skyline objects, or the largest one that returns at most $\delta$ skyline objects.

We address the above question in the context of $k$-dominant skylines over joined relations.

In sum, our contributions are:

We propose the problem of finding $k$-dominant skyline queries over joined relations. We term such queries KSJQ. 2. 2.

We design efficient algorithms to solve the KSJQ problem. 3. 3.

We devise ways to arrive at a good value of “$k$” by specifying a threshold size of the $k$-dominant skyline set.

The rest of the paper is organized as follows. Sec. 2 sets the background on joins of $k$-dominant skylines. Using this, different problem statements are defined in Sec. 3. Sec. 4 outlines the related work. Various optimizations that can improve the efficiency of the problem are explained in Sec. 5. Algorithms that use these optimizations are described in Sec. 6. Sec. 7 analyzes the experimental results before Sec. 8 concludes.

2 Background

2.1 Skylines

Consider a dataset $R$ of objects. For each object $u$, a set of $d$ attributes $\{u_{1},\dots,u_{d}\}$ are specified, which form the skyline attributes. For each of the skyline attributes, without loss of generality, the preference is assumed to be less than ($<$), i.e., a lower value is preferred over a higher one. An object $u$ dominates another object $v$, denoted by $u\succ v$, if and only if, for all the $d$ skyline attributes, $u_{i}$ is preferred over or equal to $v_{i}$, and there exists at least one skyline attribute where $u_{j}$ is strictly preferred over $v_{j}$. The skyline set $S\subseteq R$ contains objects that are not dominated by any other object [3]. In other words, every object in the non-skyline set is dominated by at least one object in the dataset.

2.2 K-Dominant Skylines

The $k$-dominant skyline query [4] relaxes the definition of domination between objects. An object $u$ $k$-dominates another object $v$, denoted by $u\succ_{k}v$, if and only if, for at least $k$ of the $d$ skyline attributes, $u_{i}$ is preferred over or equal to $v_{i}$, and there exists at least one skyline attribute where $u_{j}$ is strictly preferred over $v_{j}$. The $k$-dominant skyline set contains objects that are not $k$-dominated by any other object. The $k$ attributes are not fixed and can be any subset of the $d$ skyline attributes.

The $k$-dominant query is particularly problematic when $k\leq\frac{d}{2}$ since then two objects can dominate each other. Even when $k>\frac{d}{2}$, the $k$-dominance relationship is not transitive and may be even cyclic: $u\succ_{k}v\succ_{k}w\succ_{k}u$.

2.3 Multi-Relational Skylines

Consider two datasets $R_{1}$ and $R_{2}$ with $d_{1}$ and $d_{2}$ skyline attributes respectively. The join of the two relations (using appropriate join conditions) forms the dataset $R=R_{1}\Join R_{2}$ with $d=d_{1}+d_{2}$ skyline attributes. The multi-relational skyline is extracted from $R$ [21].

In a significant variant of the problem, a number of skyline attributes in each relation are marked for aggregation with corresponding attributes from the other relation [2]. Thus, $a$ attributes out of $d_{1}$ in $R_{1}$ and $d_{2}$ in $R_{2}$ are aggregated in the joined relation $R$. As a result, $R$ contains $(d_{1}-a)+(d_{2}-a)+a=d_{1}+d_{2}-a$ skyline attributes. The skyline query is then asked over these attributes. The aggregation function is assumed to be monotonic; this ensures that if the base values of two tuples $u\in R_{1}$ and $t\in R_{2}$ are preferred over the base tuples of two other tuples $v\in R_{1}$ and $s\in R_{2}$ respectively, the aggregated value of $u\Join t\in R$ will be necessarily preferred over the aggregated value of $v\Join s\in R$.

The case for more than two base relations can be handled by cascading the joins.

3 Problem Statements

We propose two main variants of what we call the k-dominant skyline join query (KSJQ) problem.

The first variant works on relations for which the skyline attributes are strictly local.

The number of skyline attributes in the first and second relations, $R_{1}$ and $R_{2}$, are $d_{1}$ and $d_{2}$ respectively.

[TABLE]

where $h_{i}$ captures the join of $h_{1_{i}}$ with $h_{2_{i}}$ in the joined relation $R=R_{1}\Join R_{2}$, and $s_{1_{i}},s_{2_{i}}$ are the skyline attributes.

Problem 1** (KSJQ)**

Given two datasets $R_{1}$ and $R_{2}$ having $d_{1}$ and $d_{2}$ skyline attributes respectively, find $k$-dominant skylines from the joined relation $R=R_{1}\Join R_{2}$ having $d=d_{1}+d_{2}$ skyline attributes.

We restrict $k$ to be at least one more than the dimensionality in the base relations, i.e., $\max\{d_{1},d_{2}\}<k<d$. This restriction constrains at least some skyline attributes from each relation to satisfy the preferences. In other words, the $k$ preferred attributes must span both the relations. However, there is no restriction over how many skyline attributes from each relation must be satisfied. Denoting the number of skyline attributes that are chosen from the two relations by $k_{1}$ and $k_{2}$ where $k_{1}+k_{2}=k$, this implies that $1\leq k_{1}\leq d_{1}$ and $1\leq k_{2}\leq d_{2}$.

The second variant works on the aggregate version of the problem. Assuming a total of $a$ aggregate attributes, the number of attributes on which only local skyline preferences are applied are $d_{1}-a=l_{1}$ and $d_{2}-a=l_{2}$ respectively. The joined relation $R$, therefore, contains $l_{1}+l_{2}$ local attributes and $a$ aggregate attributes.

Problem 2** (Aggregate KSJQ)**

Given two datasets $R_{1}$ and $R_{2}$ having $d_{1}$ and $d_{2}$ skyline attributes respectively, of which $a$ attributes are used for aggregation, find $k$-dominant skylines from the joined relation $R=R_{1}\Join R_{2}$ having $d_{1}+d_{2}-a$ skyline attributes.

Once more, we assume that $\max\{d_{1},d_{2}\}<k\leq d$. Expressing in terms of local and aggregate attributes, $\max\{l_{1},l_{2}\}+a<k\leq l_{1}+l_{2}+a$.

The third and fourth problems address the tuning of the value of $k$. The value can be tuned in two ways.

Problem 3** (At least $\delta$)**

Given a KSJQ query framework and a threshold number of skyline objects $\delta$, determine the smallest value of $k$ that returns at least $\delta$ skyline objects.

Problem 4** (At most $\delta$)**

Given a KSJQ query framework and a threshold number of skyline objects $\delta$, determine the largest value of $k$ that returns at most $\delta$ skyline objects.

Prob. 4 is directly linked with Prob. 3. If $k^{*}$ is the answer to Prob. 3, then the answer to Prob. 4 must be $k^{*}-1$. There are two corner cases. First, if $k^{*}=1$, then the answer to Prob. 4 should be trivially $1$ as well. Second, if $k^{*}$-dominant skyline returns exactly $\delta$ skyline objects (or $k^{*}=d$), then the answer to Prob. 4 is $k^{*}$ as well. Thus, henceforth, we focus only on Prob. 3.

4 Related Work

The skyline operator was introduced in databases by [3] by adopting the maximal vector or the Pareto optimal problem [15]. Several indexed [14], [17] and non-indexed algorithms [3, 5, 22] have been since proposed to retrieve the skyline set.

Analyses of the cardinality of the skyline set [9, 28] have shown that the number of skyline objects can grow exponentially with the increase in dimensionality. Consequently, several attempts have been made to restrict the size of the skyline set. These mostly include notions of approximate skylines or representative skylines [6, 8, 10, 12, 13, 16, 23, 25, 26, 27].

A completely different approach—$k$-dominant skylines—was proposed by [4]. It uses subsets of skyline attributes to control the cardinality of the skyline set. Although the parameter $k$ is an input to the problem, the authors also proposed a way to derive the smallest $k$ that will guarantee a $\delta$ number of skylines.

There have been many recent works on $k$-dominant skylines including extensions to high-dimensional spaces [18, 19], using parallel processing [24], cardinality estimation [11] as well as in update-heavy datasets [20] and combined datasets [7].

The skyline join problem where skylines are retrieved from joined relations, both components of which contain skyline attributes, was introduced in [21]. This work, however, simply assumed that the skyline attributes remain unchanged from the base relations to the joined one. The aggregate skyline join queries [2] removed this restriction by enabling skyline preferences to be posed on aggregate values of attributes formed by combining attributes from the base relations. The aggregation function was assumed to be monotonic.

In this paper, we pose the $k$-dominant skyline problem in the join paradigm (including the aggregated version). To the best of our knowledge, this is the first work in this direction.

5 Optimizations

In this section, we describe the various optimization schemes that can be used to speed-up the process of finding $k$-dominant skylines in joined relations. Sec. 6 uses these optimizations to design the algorithms.

5.1 Join Attributes

We assume an equality join condition. Two base tuples $u\in R_{1}$ and $v\in R_{2}$ can be joined to form $t=u\Join v\in R$ if and only if all their join attributes match. Referring to Eq. (3), $\forall_{j=1,\dots,m}\ h_{1_{j}}=h_{2_{j}}$. (We discuss the relaxation of this assumption in Sec. 6.6.)

Assumption 1** (Equality Join)**

The join conditions are all based on equality, i.e., the join is an equality join.

All the tuples in a base relation are, thus, divided into groups according to the value of the join attributes. In every group, the values of the join attributes, $h_{i_{1}},\dots,h_{i_{m}}(i=1,2)$, are the same.

In the flight example, since the joining criterion is destination of first flight to be the source of the second flight, the first base relation is divided on the basis of destinations while the second one is divided on sources.

5.2 Grouping

Based on the $k$-dominance properties within each group, each base relation is partitioned into different sets as follows.

A tuple may be a $k$-dominant tuple for the entire base relation. However, even when it is not, it may not be $k$-dominated by any other tuple in its group. It is then a $k$-dominant tuple when only its group is concerned.

Based on the above notion of $k$-dominance within a group, each base relation $R_{i}$ is divided into $3$ mutually exclusive and exhaustive sets $SS_{i}$, $SN_{i}$, and $NN_{i}$:

[TABLE]

We next define the three sets.

Definition 1** ($SS$)**

A tuple $u$ is in $SS$ if $u$ is a $k$-dominant skyline in the overall relation; consequently, it is a $k$-dominant skyline in its group as well.

Definition 2** ($SN$)**

A tuple $u$ is in $SN$ if $u$ is a $k$-dominant skyline only in its group but not in the overall relation.

Definition 3** ($NN$)**

A tuple $u$ is in $NN$ if $u$ is not a $k$-dominant skyline in its group; consequently, it is not a $k$-dominant skyline in the overall relation as well.

Consider the examples in Table 1 and Table 2. We assume that all the attributes have lower preferences222Although the preferences for ratings and amenities are generally the other way round, we stick to lower preferences for ease of understanding.. We set $k=3$ for both the relations. The categorization of the tuples are shown in the last column.

Table 3 shows the joined relation.

The important outcome of this division into the three sets is that it allows certain tuples to be automatically designated as $k$-dominant skylines (or not) without computing the join, as explained next.

5.3 Deciding about Skylines before Join

We first analyze a simple case where out of the final $k$ attributes in the joined relation, if a joined tuple $t$ $k$-dominates another joined tuple $s$, $t$ must $k_{1}$-dominate $s$ in the first base relation and must $k_{2}$-dominate in the second base relation ($k=k_{1}+k_{2}$).

Note that, in practice this situation will not be specified by any user. We are only going to use it to explain the basic concepts and then build upon it later by removing this assumption. For the theorems and observations in this section, the first base relation is divided into the three sets $SS_{1}$, $SN_{1}$, $NN_{1}$ according to $k_{1}$-domination while the second base relation is divided into $SS_{2}$, $SN_{2}$, $NN_{2}$ using $k_{2}$-domination.

The first theorem shows that a tuple formed by joining the $SS$ counterparts is always a $k$-dominant skyline.

Theorem 1

The tuples in the set $(SS_{1}\Join SS_{2})$ are $k$-dominant skylines.

Proof 5.1**.**

Consider a joined tuple $t^{\prime}=u^{\prime}\Join v^{\prime}$ formed by joining the tuples $u^{\prime}\in SS_{1}$ and $v^{\prime}\in SS_{2}$. Assume that $t=u\Join v$ $k$-dominates $t^{\prime}$, i.e., $t\succ_{k}t^{\prime}$. Since $u^{\prime}\in SS_{1}$, no tuple and, in particular, $u$ can $k_{1}$-dominate $u^{\prime}$. Similarly, $v\not\succ_{k_{2}}v^{\prime}$. Therefore, $\not\exists t,\ t\succ_{k}t^{\prime}$. Hence, $t^{\prime}$ is a $k$-dominant tuple.

The flight combination (16,26) in Table 3 is an example.

The second theorem shows the reverse: composite tuples formed by joining a base tuple in $NN$ can never be $k$-dominant skylines.

Theorem 2**.**

The tuples in the sets $(SS_{1}\Join NN_{2})$, $(SN_{1}\Join NN_{2})$, $(NN_{1}\Join SS_{2})$, $(NN_{1}\Join SN_{2})$, and $(NN_{1}\Join NN_{2})$ are not $k$-dominant skylines.

Proof 5.2**.**

Consider a joined tuple $t^{\prime}=u^{\prime}\Join v^{\prime}$ formed by joining the tuples $u^{\prime}\in NN_{1}$ and $v^{\prime}\in R_{2}$. Therefore, there must exist a tuple $u$ in the same group that $k_{1}$-dominates $u^{\prime}$, i.e., $\exists u,\ u\succ_{k_{1}}u^{\prime}$. Consider the tuple $t$ formed by joining $u$ with $v^{\prime}$. Since $u$ is in the same group as $u^{\prime}$, this tuple necessarily exists. As $t$ dominates $t^{\prime}$ in $k_{1}$ attributes and is equal in $k_{2}$, overall, it dominates in $k_{1}+k_{2}=k$ attributes, i.e., $t\succ_{k}t^{\prime}$. Therefore, $t^{\prime}$ is not a $k$-dominant skyline. This covers the cases $(NN_{1}\Join SS_{2})$, $(NN_{1}\Join SN_{2})$, and $(NN_{1}\Join NN_{2})$. The cases for $(SS_{1}\Join NN_{2})$ and $(SN_{1}\Join NN_{2})$ are symmetrical.

The flight combinations (11,24), (13,22), (14,21), (14,22), (12,23), and (12,24) in Table 3 exemplify the above cases. For example, the tuple (11,24) is dominated by (11,23).

However, nothing can be concluded surely about the rest of the joined tuples.

Observation 1

The tuples in the sets $(SS_{1}\Join SN_{2})$ and $(SN_{1}\Join SS_{2})$ are most likely to be $k$-dominant skylines, although that is not guaranteed.

Proof 5.3**.**

Consider a joined tuple $t^{\prime}=u^{\prime}\Join v^{\prime}$ formed by joining the tuples $u^{\prime}\in SS_{1}$ and $v^{\prime}\in SN_{2}$. Consider the dominator $v\in R_{2}\succ_{k_{2}}v^{\prime}$. Since $v^{\prime}$ is a $k_{2}$-dominate skyline in $R_{2}$, $v$ is in a different group than $v^{\prime}$. Therefore, $v$ cannot join with $u^{\prime}$, i.e., $\not~{}\!\exists t=u^{\prime}\Join v,\ v\succ_{k_{2}}v^{\prime}$. Although no tuple $u$ can $k_{1}$-dominate $u^{\prime}$, there may exist $u$ whose $k_{1}$ attributes have the same value as $u^{\prime}$. If it happens that $u$ is join-compatible with $v$, then and only then, $t=u\Join v$ exists and $t\succ_{k}t^{\prime}$ (since it is equal in $k_{1}$ and better in $k_{2}$ attributes). The above situation is quite unlikely, although not impossible. The case for $(SN_{1}\Join SS_{2})$ is symmetrical.

Thus, while (11,23) and (13,21) are $k$-dominant skylines, (18,28) is not (Table 3). Flight 18 has same $k_{1}$ values as 19 while 28 is dominated by 25. Therefore, (18,28) is $k$-dominated by (19,25).

The observation for $(SN_{1}\Join SN_{2})$ is similar.

Observation 2

A tuple in the set $(SN_{1}\Join SN_{2})$ may or may not be a $k$-dominant skyline.

Proof 5.4**.**

Consider a joined tuple $t^{\prime}=u^{\prime}\Join v^{\prime}\in(SN_{1}\Join SN_{2})$. Consider the tuples $u$ and $v$ such that $u\succ_{k_{1}}u^{\prime}$ and $v\succ_{k_{2}}v^{\prime}$. Since $u^{\prime}\in SN_{1}$, $u$ is in a different group than $u^{\prime}$. Similarly, $v$ is in a different group than $v^{\prime}$. If and only if $u$ and $v$ are join compatible, they will join to form $t=u\Join v$ which then $k$-dominates $t^{\prime}$. Otherwise, no such $t$ exists, and $t^{\prime}$ is a $k$-dominant skyline.

Consider (15,25) in Table 3. Since its dominators, flights 11 and 21 respectively, are not join compatible (11 reaches city C while 21 takes off from city D), the flight combination (11,21) is not valid. Consequently, no joined tuple dominates it, and (15,25) becomes a $k$-dominant skyline. On the other hand, for (17,27), the dominators 16 and 26 do join (the city F is common) to form the tuple (16,26). As a result, (17,27) is not a $k$-dominant skyline.

The overall situation is summed up in Table 4.

5.4 Skyline Attributes in Joined Relation

We next do away with the assumption that $k_{1}$ and $k_{2}$ attributes have to be satisfied separately from the first and second relations. It is simply required that the joined relation returns $k$-dominant skylines with no restriction on how $k$ is broken up between the base relations. However, to ensure that the skyline preferences are respected for at least one attribute in every base relation, we assume that $k>\max\{d_{1},d_{2}\}$.

A brute-force way to find the skylines is to generate all combinations of $k_{1}$ and $k_{2}$ such that $k_{1}+k_{2}=k$ and, then, combine the answer sets using the results obtained in the previous section. However, it can be done more efficiently as explained next.

We consider the following cases:

[TABLE]

For all combinations of $k_{1}$ and $k_{2}$ such that $k_{1}+k_{2}=k$, the inequalities, $1\leq k^{\prime}_{1}\leq k_{1}\leq d_{1}$ and $1\leq k^{\prime}_{2}\leq k_{2}\leq d_{2}$, hold.

Consider the example in Table 3. If $k=7$, then $k^{\prime}_{1}=k^{\prime}_{2}=3$. Thus, categorization and $k$-dominant skyline sets remain the same.

We first establish the following lemma on the monotonicity of the number of skyline attributes.

Lemma 3**.**

If tuple $u$ is a $j$-dominant skyline, it is also a $i$-dominant skyline for any $i\geq j$.

Proof 5.5**.**

Consider $u$ to be a $j$-dominant tuple. Assume that, however, it is not $i$-dominant for some $i>j$. Thus, there exists $v$ that $i$-dominates $u$, i.e., $v$ is better than $u$ in $i$ attributes. Then, $v$ must be better than $u$ in some subset $j$ of these $i$ attributes, which is a contradiction. Thus, $u$ must be $i$-dominant for every $i\geq j$.

The following theorems and observations implicitly use Lemma 3.

Theorem 4**.**

The tuples in the set $(SS_{1}\Join SS_{2})$ are $k$-dominant skylines.

Proof 5.6**.**

Consider $t^{\prime}=u^{\prime}\Join v^{\prime}\in(SS_{1}\Join SS_{2})$. There are two cases to consider: (i) when there does not exist any tuple $u$ that shares $k^{\prime}_{1}$ attributes with $u^{\prime}$, and (ii) when there exists a tuple $u$ that share $k^{\prime}_{1}$ or more attributes with $u^{\prime}$.

In the first case, if $\exists t,\ t\succ_{k}t^{\prime}$, then $t$ must dominate $t^{\prime}$ in at least $k^{\prime}_{1}$ attributes corresponding to $u^{\prime}$ since it can dominate $t$ in at most $d_{2}$ attributes corresponding to $v^{\prime}$. Since this is impossible, $t^{\prime}$ is a $k$-dominant skyline.

In the second case, $u^{\prime}$ can be dominated by $u\in R_{1}$ in at most $d_{1}-1$ attributes (otherwise $u^{\prime}\not\in SS_{1}$). Thus, to dominate $t^{\prime}$, $u$ must combine with $v\in R_{2}$ such that $v$ dominates $v^{\prime}$ in at least $k-(d_{1}-1)=k_{2}+1$ attributes, which is impossible since $v^{\prime}\in SS_{2}$. Hence, $t^{\prime}$ is a $k$-dominant skyline.

Theorem 5**.**

The tuples in the sets $(SS_{1}\Join NN_{2})$, $(SN_{1}\Join NN_{2})$, $(NN_{1}\Join SS_{2})$, $(NN_{1}\Join SN_{2})$, and $(NN_{1}\Join NN_{2})$ are not $k$-dominant skylines.

Proof 5.7**.**

In each of the sets, at least one base tuple is in $NN$. Consider a joined tuple $t^{\prime}=u^{\prime}\Join v^{\prime}$ where $u^{\prime}\in NN_{1}$. Surely, $\exists u,\ u\succ_{k^{\prime}_{1}}u^{\prime}$ exists and $u$ is in the same group as $u^{\prime}$. Therefore, $u$ is join compatible with $v^{\prime}$. Consider $t=u\Join v^{\prime}$. It dominates $t^{\prime}$ in $k^{\prime}_{1}+d_{2}=k$ attributes. Thus, $t^{\prime}$ is not a $k$-dominant skyline.

For example, (16,26) $\in SS_{1}\Join SS_{2}$ and (14,22) $\in NN_{1}\Join NN_{2}$ (in Table 3) is a $k$-dominant skyline and not a $k$-dominant skyline respectively.

Observation 3

A tuple in the set $(SS_{1}\Join SN_{2})$ or $(SS_{2}\Join SN_{1})$ is most likely to be a $k$-dominant skyline, although that is not guaranteed.

Proof 5.8**.**

Consider $t^{\prime}=u^{\prime}\Join v^{\prime}\in(SS_{1}\Join SN_{2})$. There are two cases to consider. In the first case when there does not exist any tuple $u$ that share $k^{\prime}_{1}$ attributes with $u^{\prime}$, a joined tuple $t$ can dominate $t^{\prime}$ in at most $k^{\prime}_{1}-1+d_{2}=k-1$ attributes. Therefore, $t^{\prime}$ is a $k$-dominant skyline.

In the second case, assume that such a $u$ with equal $k^{\prime}_{1}$ attributes exists. Note, however, that $u$ cannot be better in any other attribute as then $u^{\prime}$ would be $k^{\prime}_{1}$-dominated by $u$. Since $v^{\prime}\in SN_{2}$, there exists a $v\in R_{2}$ that dominates $v^{\prime}$ but is in a different group. If and only if $u$ and $v$ are join compatible, the joined tuple $t=u\Join v$ exists. Comparing $t$ against $t^{\prime}$, we see that the $k^{\prime}_{1}$ attributes corresponding to $u^{\prime}$ (or $u$) are same. Therefore, for $t^{\prime}$ to $k$-dominate $t$, $v^{\prime}$ must dominate $v$ in all the $k-k^{\prime}_{1}=d_{2}$ attributes. This is unlikely, although not impossible.

The proof for $(SS_{2}\Join SN_{1})$ is similar.

For example, consider (18,28) $\in SS_{1}\Join SN_{2}$ in Table 3 against (19,25). While $k^{\prime}_{1}=3$ attributes for 18 and 19 are same, 25 dominates 28 in $d_{2}=4$ attributes. Thus, overall, (19,25) dominates (18,28) in $3+4=7$ attributes and, thus, (18,28) is not a $k$-dominant skyline. On the other hand, (11,23) $\in SS_{1}\Join SN_{2}$ is a $k$-dominant skyline since there is no tuple that shares $k^{\prime}_{1}=3$ attributes with 11.

Observation 4

A tuple in the set $(SN_{1}\Join SN_{2})$ may or may not be a $k$-dominant skyline.

Proof 5.9**.**

Consider a joined tuple $t^{\prime}=u^{\prime}\Join v^{\prime}\in(SN_{1}\Join SN_{2})$. Consider the tuples $u$ and $v$ that dominate $u^{\prime}$ and $v^{\prime}$ respectively. Since $u$ and $v$ are in different groups than $u^{\prime}$ and $v^{\prime}$, they may not be join compatible. If they are, the joined tuple $t=u\Join v$ dominates $t^{\prime}$ in at least $k^{\prime}_{1}+k^{\prime}_{2}$ attributes. The tuple $t$ may additionally dominate $t^{\prime}$ in some other attributes such that it overall $k$-dominates $t^{\prime}$. If these two conditions are not met, $t^{\prime}$ becomes a $k$-dominant skyline.

Out of the two tuples in $SN_{1}\Join SN_{2}$ in table 3, while (15,25) is a $k$-dominant skyline because the dominators 11 and 21 are not in the same group and, therefore, cannot join, the flight combination (17,27) is not a $k$-dominant skyline as the dominators 16 and 26 join to form (16,26) which overall $k$-dominates (17,27).

Table 5 summarizes the situation.

It is important to note that for determining the groups in the base relations, it must be the minimum number of attributes considered, i.e., $k^{\prime}_{1}$ and $k^{\prime}_{2}$; otherwise, the correctness of the above theorems may be violated.

5.5 Unique Value Property

For a tuple in $SS_{1}\Join SN_{2}$ (and $SN_{2}\Join SS_{1}$) to be not a $k$-dominant skyline, the number of attributes in which $u\in R_{1}$ is same as $u^{\prime}\in SS_{1}$ must be at least $k^{\prime}_{1}$. If the base relations follow a unique value property where it is guaranteed that for any $k^{\prime}_{i}\ (i=1,2)$ number of attributes, two tuples will be unique, then the processing becomes simpler.

Definition 6** (Unique Value Property).**

A relation $R$ has the unique value property (UVP) with respect to $i$ if for each subset of $i$ skyline attributes of $R$, all the tuples are unique, i.e., no two tuple will have exactly the same values in any $i$-sized subset of attributes.

If $k^{\prime}_{i}=1$, every tuple for any attribute must be unique. While real-valued attributes generally follow that, categorical attributes do not. However, for reasonable values of $k^{\prime}_{1}$ and $k^{\prime}_{2}$, real datasets having a mix of both real-valued and categorical attributes generally follow the UVP.

The UVP is extremely useful since it ensures that the tuples in $SS_{1}\Join SN_{2}$ and $SN_{2}\Join SS_{1}$ become $k$-dominant skylines as shown next.

Theorem 7**.**

If relations $R_{1}$ and $R_{2}$ follow UVP with respect to $k^{\prime}_{1}$ and $k^{\prime}_{2}$ attributes respectively, the tuples in the sets $(SS_{1}\Join SN_{2})$ and $(SN_{1}\Join SS_{2})$ are $k$-dominant skylines.

Proof 5.10**.**

Consider a joined tuple $t$ in either of the two sets. The generic situation, as shown in Obs. 3 has two cases. While the first case makes $t$ a $k$-dominant skyline, the UVP precludes the second case. Thus, $t$ is always a $k$-dominant skyline.

Although we present Th. 7 for the sake of completeness, we do not assume it in our experiments (Sec. 7).

5.6 Aggregate Attributes

We next consider the case of aggregation where the $k$-dominant skyline is sought over attributes that attain values aggregated from attributes in the base relation.

We assume that there are $l_{1}$ local and $a$ aggregate skyline attributes in $R_{1}$ and $l_{2}$ local and $a$ aggregate skyline attributes in $R_{2}$. The total number of skyline attributes in $R$ is, therefore, $l_{1}+l_{2}+a$ attributes. As earlier, the final skyline query is asked over $k<l_{1}+l_{2}+a$ attributes.

The groups in the base relations are partitioned based on both the local and aggregate attributes. Out of the final number of skyline attributes $k$, $a$ of them can be aggregate. Thus, the minimum number of local attributes that must be dominated in each base relation is $k^{\prime\prime}_{1}=k-a-l_{2}$ and $k^{\prime\prime}_{2}=k-a-l_{1}$. The categorization of the base relations into the sets $SS$, $SN$ and $NN$ are done on the basis of $k^{\prime}_{1}=k^{\prime\prime}_{1}+a$ and $k^{\prime}_{2}=k^{\prime\prime}_{2}+a$. Since $d_{1}=a+l_{1}$ and $d_{2}=a+l_{2}$, these definitions are same as earlier (Sec. 5.4).

We use the following assumption about the monotonicity property of the aggregate attributes.

Assumption 2** **(Monotonicity)

If the value of attribute $u_{1}$ dominates that of $u_{2}$ and the value of attribute $v_{1}$ dominates that of $v_{2}$, the aggregated value of $u_{1}\oplus v_{1}$ will dominate the aggregated value of $u_{2}\oplus v_{2}$, where $\oplus$ denotes the aggregation operator.

Since the categorization remains the same, the fate of the joined tuples using the aggregation remains exactly the same as earlier (as summarized in Table 5).

Table 6 shows the joined relation obtained from Table 1 and Table 2 with the cost values aggregated.

In the example, considering $k=6$ with $a=1$, $k^{\prime\prime}_{1}=6-1-3=2$ and $k^{\prime\prime}_{2}=6-1-3=2$ but $k^{\prime}_{1}=k^{\prime\prime}_{1}+1=3$ and $k^{\prime}_{2}=k^{\prime\prime}_{2}+1=3$ remain as earlier.

6 Algorithms

In this section, we describe the various algorithms for answering the $k$-dominant skyline join queries (Sec. 5.4). We consider the general case where the datasets do not follow UVP (Sec. 5.5) and, where in addition to local skyline attributes, there are aggregate ones as well (Sec. 5.6).

6.1 Naïve Algorithm

The naïve algorithm (Algo. 1) simply computes the join of the two relations first (line 1) and then computes the $k$-dominant skylines from the joined relation (line 2) using any of the standard $k$-dominant skyline computation methods [4]. Being the most basic, it suffers from two major disadvantages. First, the join can require a very large time, thereby rendering the entire algorithm extremely time-consuming and impractical. The second shortcoming is the non-progressive result generation. The user has to wait a fairly large time (at least the complete joining time) before even the first skyline result is presented to her. In online scenarios, the progressive result generation is quite an attractive and useful feature.

6.2 Target Set

To alleviate the problems of the naïve algorithms, we next propose two algorithms, grouping and dominator-based, that use the concepts of optimization from Sec. 5.

However, before we describe them, we first explain and define the concept of target sets. Although a tuple in a set marked by “may be” or “likely” is not guaranteed to be a $k$-dominant skyline, it needs to be checked against only a small set of tuples, called its target set.

Formally, a target set for a tuple $u^{\prime}$ in a base relation is the set of tuples $\tau(u^{\prime})$ that can potentially combine with other tuples from the other base relation and $k$-dominate a joined tuple formed with $u^{\prime}$. In other words, for a joined tuple $t^{\prime}=u^{\prime}\Join v^{\prime}$, there may exist $v$ such that $t=u\Join v$ where $u\in\tau(u^{\prime})$ $k$-dominates $t^{\prime}$. No tuple outside the target set $\tau(u)$ of $u$ may combine with any other tuple and dominate $t$.

Definition 8** (Target Set).**

The target set for a tuple $u^{\prime}\in R_{i}$ is the set of tuples $\tau(u^{\prime})\subseteq R_{i}$ such that $\forall u\not\in\tau(u^{\prime}),\ \not\exists v,v^{\prime},\ t=u\Join v\succ t^{\prime}=u^{\prime}\Join v^{\prime}$.

The definition is one-sided: a tuple in the target set may or may not join and dominate $t^{\prime}$, but no tuple outside the target set can join and dominate $t^{\prime}$. The utility of a target set is easy to understand. To check whether $t^{\prime}$ is a $k$-dominant skyline, $u^{\prime}$ needs to be checked only against its target set and nothing outside it.

The join of target sets for the base tuples produces the potential dominating set for the joined tuple.

The target set for a tuple $u^{\prime}\in SS$ constitutes itself and the set of tuples $\{u\}$ that has at least $k^{\prime}_{1}$ attributes same as $u^{\prime}$. The augmentation is required to guarantee the correctness as explained in Obs. 3. The tuple $u^{\prime}$ must be included in the target set of $u^{\prime}$ for the same reason.

For each tuple in $SS$, the number of such tuples sharing at least $k^{\prime}_{1}$ attributes is typically low. Hence, maintenance of target sets is quite feasible and practical.

However, the target set for a tuple in $SN$ can be any tuple outside its group that dominates it. For simplicity, we consider it as the entire dataset $R_{i}$.

Similarly, the target set for $NN$ is $R_{i}$.

6.3 Grouping Algorithm

Our first algorithm, called the grouping algorithm (Algo. 2), first computes the groups $SS$, $SN$ and $NN$ in the base relations. Next, the summarization from Table 5 is used. Tuples from the sets marked by “yes” are immediately output as $k$-dominant skyline tuples. Tuples marked by “no” are pruned and not even joined.

A tuple in a set marked by “may be” or “likely” is checked against the join of the target sets of the base tuples.

For a base tuple in the $SS$ group, the target set is first augmented with tuples that share at least $k^{\prime}_{i}$ attributes (the Augment subroutine in lines 8 and 9). Then, each group of tuples is checked only against its target set. For example, in line 10, the set $SS_{1}\Join SN_{2}$ is checked only against $A_{1}\Join R_{2}$ for a domination in $k$ attributes. Note that the target set for $u\in SS_{1}$ is only $A_{1}$. Similarly, lines 11 and 12 handle the sets $SN_{1}\Join SS_{2}$ and $SN_{2}\Join SN_{1}$ respectively.

The efficiency of the grouping algorithm stems from the fact that only the tuples in $SN_{1}\Join SN_{2}$ need to be compared against the entire $R_{1}\Join R_{2}$. For other tuples, the decision can be taken without even joining (the “yes” and “no” cases), or the comparison set is small (for the tuples in $SS_{1}\Join SN_{2}$ and $SS_{2}\Join SN_{1}$).

6.4 Dominator-Based Algorithm

The grouping algorithm has the problem that for tuples in $SN_{i}$, the target set is the entire relation $R_{i}$. The next algorithm, dominator-based algorithm (Algo. 3) rectifies this by explicitly storing the set of dominators. Thus, for a tuple $\in SS_{i},SN_{i}$, the dominator set of tuples is first obtained (lines 9 and 13). When the tuple is in $SS_{i}$, this set is empty. The dominator sets are augmented by the tuples themselves and those tuples having the same values in the required number of skyline attributes (lines 10 and 14). In general, the size of the dominator sets is quite low as compared to the entire dataset, i.e., generally $|dom(\cdot)|\ll R_{i}$.

The target sets for the joined tuples are composed of the joins of the dominating tuples. Each tuple in the sets marked by “likely” and “may be” is checked against these joins of the corresponding dominator sets and is added to the answer only if no dominator exists (line 18).

The joins of dominating sets are substantially less in size than the target sets for the grouping algorithm (which is the join of the entire target sets). The saving is largest for tuples in $SN_{1}\Join SN_{2}$.

The saving, however, comes at a cost. For each tuple in $SN$, all the dominators need to found out. In addition, the entire dominator set needs to be stored explicitly.

The advantages of the dominator-based algorithm may not be enough to offset this overhead of time and storage, especially when there are many such tuples. Sec. 7 compares the different algorithms empirically.

6.5 Cartesian Product

When the final relation is a Cartesian product of the two base relations, the algorithms become considerably easier. The Cartesian product can be considered as a special case of join with every tuple having the same value of the join attribute. In other words, all the tuples are in the same join group. As a result, there is no $SN$ set. A tuple is either in $SS$ (when it is a skyline in its local relation) or in $NN$ (when it is not a skyline). Consequently, the tables become much simpler, and the fate of all the joined (i.e., final) tuples can be concluded without the need to explicitly compute them. The tuples in $SS_{1}\Join SS_{2}$ are skylines while none of the other tuples are.

6.6 Non-Equality Join Condition

In certain cases, the join condition may not be an equality. For example, in a flight combination, the arrival time of the first leg needs to be earlier than the departure time of the second, i.e., $f_{1}.\text{arrival}<f_{2}.\text{departure}$. In this section, we discuss how to handle such join cases when the condition is one of $<,\leq,>,\geq$.

Since the main purpose of dividing a base relation into the three sets $SS$, $SN$ and $NN$ is to ensure that certain decisions can be taken about the tuples in these sets without joining, all the optimizations discussed in Sec. 5 work with the following modifications.

A tuple in $SS_{1}\Join SS_{2}$ can never be $k$-dominated and, thus, it does not matter how such a tuple is composed from the base relations. In other words, the semantics of the join condition, equality or otherwise, does not matter for $SS_{1}\Join SS_{2}$ tuples.

Next consider a tuple $t^{\prime}=u^{\prime}\Join v^{\prime}\in(SN_{1}\Join SS_{2})$. A tuple in the $SN$ set, $u^{\prime}$, is originally defined as one that is not dominated by any other tuple in the same group. This ensures that if $u^{\prime}$ joins with $v^{\prime}$ from the other relation, then no other tuple $u$ can join with $v^{\prime}$ to dominate $t^{\prime}$. This is the crucial property that needs to be maintained even when the join condition is non-equality.

Thus, if the join condition is $u^{\prime}.arr<v^{\prime}.dep$, then the set $\{u:u.arr<u^{\prime}.arr\}$ is considered to be in the same group of $u^{\prime}$ since it can also join with $v^{\prime}$ (and can potentially dominate $u$). In other words, this ensures that all such $u$ is join compatible with $v^{\prime}$. The set $SN$ is thus expanded to take care of the non-equality condition. Note that there may exist other tuples $\{u^{\prime\prime}:u^{\prime\prime}.arr<v^{\prime}.dep\}$ that may also join with $v^{\prime}$; these, however, cannot be determined locally without the knowledge of $v^{\prime}$ from the other relation and, therefore, cannot be considered. The target set of a tuple in $SN$ is anyway the entire dataset (or its dominators). Thus, the algorithms will work correctly with the above modification.

Similarly, for the converse set, $SS_{1}\Join SN_{2}$, the $SN$ set for $v^{\prime}$ consists of $\{v:v.dep>v^{\prime}.dep\}$.

A joined tuple with $NN$ as a component is rejected as a $k$-dominant skyline since the $NN$ tuple can be always dominated by another tuple in the same group. Hence, similar to $SN$, the group of a tuple needs to be defined by taking into account the semantics of the join condition.

Considering the earlier example of $u^{\prime}.arr<v^{\prime}.dep$, if $u^{\prime}\in NN$, then to say that $u^{\prime}\in NN_{1}$, there must exist $u:u.arr<u^{\prime}.arr$ and $u\succ_{k^{\prime}_{1}}u^{\prime}$. (The definition is suitably modified for $NN_{2}$ in a suitable manner.) This ensures that the joined tuple $t^{\prime}=u^{\prime}\Join v^{\prime}$ will be dominated by $u\Join v^{\prime}$. Once more, tuples of the form $\{u^{\prime\prime}:u^{\prime\prime}.arr<v^{\prime}.dep\}$ are left out due to lack of knowledge about $v^{\prime}$.

It may happen that there does not exist any such $u$ but there exists such an $u^{\prime\prime}$. In that case, $u^{\prime}$ is classified as an $SN$ tuple instead of an $NN$. This, however, only leads to extra processing of tuples joined with $u^{\prime}$. The correctness is not violated as such joined tuples of the form $t^{\prime}=u^{\prime}\Join v^{\prime}$ will be finally caught by $u^{\prime\prime}\Join v^{\prime}$ and rejected. Thus, the above modification only affects the efficiency of the algorithms, not the final result.

6.7 Algorithms for Aggregation

The algorithms that consider aggregation are essentially the same as the plain KSJQ. The only difference is that when the join is performed, the aggregation of the attributes are done as an additional step. Therefore, they are not discussed separately.

We next discuss the algorithms for finding $k$ (Problem 3).

6.8 Finding k: Naïve Algorithm

The naïve algorithm (Algo. 4) to search for the lowest $k$ that produces at least $\delta$ skylines starts from the least possible value of $k$, i.e., $\max\{d_{1},d_{2}\}+1$, and keeps incrementing it till the number of $k$-dominant skylines is at least $\delta$. The largest possible value of $k$, i.e., $d$, is otherwise returned by default, even if it does not satisfy the $\delta$ criterion.

The algorithm is very inefficient as for each case, it computes the actual $k$-dominant skyline set. Further, it traverses the possibilities of $k$ in a linear manner. The correctness is based on the fact that the number of $k$-dominant skylines is a monotonically non-decreasing function in $k$ (Lemma 3).

We next design algorithms that use Table 5.

6.9 Finding k: Range-Based Algorithm

For a particular value of $k$, the actual number of $k$-dominant skylines, $\Delta_{k}$, is at least the size of the “yes” sets, denoted by $\Delta_{k,lb}$, and at most the sum of sizes of the “yes”, “likely” and “may be” sets, denoted by $\Delta_{k,ub}$. These, thus, denote the lower and upper bounds respectively: $\Delta_{k,lb}<\Delta_{k}<\Delta_{k,ub}$.

The range-based algorithm (Algo. 5) uses these bounds to speed-up the process. Starting from the minimum possible $k=\max\{d_{1},d_{2}\}+1$, the algorithm finds $\Delta_{k,lb}$ and $\Delta_{k,ub}$ (lines 5 and 6 respectively). If $\Delta_{k,lb}\geq\delta$, then the current $k$ is the answer (line 7). If $\Delta_{k,ub}<\delta$, then the current $k$ cannot be the answer and $k$ is incremented (line 9). Otherwise, i.e., if $\Delta_{k,lb}<\delta\leq\Delta_{k,ub}$, then the current $k$ may be an answer. In this case, the actual $k$-dominant skyline set is computed. If its size is $\delta$ or greater, it is returned as the answer (line 11). Else, $k$ is incremented (line 13), and the steps are repeated.

While this algorithm is definitely more efficient than the naïve one, it still suffers from the shortcoming that it examines a large number of $k$’s by incrementing it one by one. If the required $k$ lies towards the higher end of the range, it unnecessarily examines too many lower values of $k$. The next algorithm does a binary search to reduce this overhead.

6.10 Finding k: Binary Search Algorithm

Algo. 6 shows how the binary search proceeds. It starts from the middle of the possible values of $k$ (line 7). The values of $\Delta_{k,lb}$ and $\Delta_{k,ub}$ are computed in the same manner as earlier (lines 8 and 9 respectively).

If $\Delta_{k,lb}\geq\delta$ (line 10), then $k$ is a potential answer. No value in the higher range can be the answer as the current $k$ already satisfies the condition. However, there may be a lower $k$ that satisfies the $\delta$ condition. Hence, the search is continued in the lower range to try and find a better (i.e., lesser) $k$.

If, on the other hand, $\Delta_{k,ub}<\delta$ (line 13), then the current estimate of $k$ is too low. The search is, therefore, continued in the higher range.

If none of these bounds help, the actual number of $k$-dominant skylines, $\Delta_{k}$, is found. If $\Delta_{k}\geq\delta$ (line 15), then the current $k$ is a potential answer. However, a lesser $k$ may be found and, so, the search proceeds to the lower range.

Otherwise, i.e., when $\Delta_{k}<\delta$ (line 18), the required $k$ is searched in the higher range.

The algorithm stops when the lower range of the search becomes larger than or equal to the current answer (line 21), which then is the lowest $k$ that satisfies the $\delta$ condition. It may also stop when the range is exhausted, in which case, the current value of $k$ is returned (line 25).

The binary search based algorithm, thus, speeds up the searching through the possible range of values of $k$. Sec. 7 compares the three algorithms empirically.

7 Experimental Results

We experimented with data synthetically generated using http://randdataset.projects.pgfoundry.org/ on an Intel i7-4770 @3.40 GHz Octacore machine with 16 GB RAM using code written in Java.

We also experimented with a real dataset of two-legged flights from New Delhi to Mumbai. The details are in Sec. 7.4.

We measured the effects of various parameters on the different algorithms proposed in Sec. 6. The parameters and their default values are listed in Table 7. Note that the size of the joined relation is a derived parameter. It is equal to $n^{2}/g$ for two base relations with $n$ tuples and $g$ groups. When the effect of a particular set of parameters are measured, the rest are held to their default values, unless explicitly stated otherwise.

In the figures, the three main algorithms for KSJQ are denoted as: $G$ for grouping, $D$ for dominator-based, and $N$ for naïve. The overall running time for each algorithm is divided into various components: (i) time taken for computing the groups in the base relations, i.e., $SS$, $SN$, and $NN$, (ii) time taken for actually joining the tuples from the two base relations that cannot be pruned, (iii) time taken for finding the dominators of the tuples, and (iv) the rest of the processing. These are marked separately in the figures.

Not all the components are applicable to every algorithm, e.g., the naïve algorithm does not find groups. The components that are not applicable to an algorithm are shown to have zero costs.

The three algorithms for determining the value of $k$ are depicted as: $B$ for binary search, $R$ for range-based, and $N$ for naïve.

7.1 Aggregate

We first show the results where aggregate values have been used. The aggregation function used is sum.

7.1.1 Effect of Dimensionality

The first experiment measures the effect of varying $k$. Fig. 1 shows that the running time increases sharply with $k$. The two different settings of dimensionality, $d$, and number of aggregate attributes, $a$, (Fig. 1a and Fig. 1b) show the robustness of this behavior. As $k$ increases, it becomes increasingly hard to dominate a tuple in $k$ dimensions. As a result, the number of $k$-dominant skyline increases heavily, thereby resulting in increased running times.

Overall, the grouping algorithm is the fastest. The dominator-based algorithm spends a substantial amount of time in finding the dominators for each tuple. This overhead is not compensated enough in the final checking stage. This is due to the fact that the average dominator set sizes are quite large and, hence, joining large dominator sets requires a substantial amount of time. With increasing dimensionality, the time required to find the dominator sets increases as well.

As expected, the naïve algorithm performs the worst as it does not attempt any optimization at all. It is slower than the grouping algorithm by about 1.5-2 times.

The next set of experiments hold $d$ and $k$ constant but increases the number of aggregate attributes, $a$. Fig. 2a depict the results. Note that $a=0$ signifies that no aggregate attributes are used. The trend of the results remain the same with the running time increasing with $a$. Once more, grouping is the best algorithm followed by dominator-based and naïve.

Fig. 2b shows a medley of results across different $d$, $k$ and $a$. When $a$ or $k$ increase, the running time increases as well.

However, it seems that the reverse happens with $d$. Comparing the case of $d=5,k=7,a=1$ with $d=6,k=7,a=1$, we note that in the first case, $k^{\prime}_{1}=k^{\prime}_{2}=3$ while in the second case, $k^{\prime}_{1}=k^{\prime}_{2}=2$. Thus, in the second case, it is easier to find the groups and perform the joins. Consequently, it runs faster. The same reasoning holds true for $d=5,k=7,a=2$ against $d=6,k=7,a=2$.

We next compare $d=5,k=7,a=2$ against $d=6,k=8,a=2$. The values of $k^{\prime}_{1}=k^{\prime}_{2}=4$ are same in both the cases. However, the size of dominator sets is larger in the first case. The time required to divide the base relations into the three sets is also higher. This leads to an overall higher running time.

7.1.2 Effect of Number of Join Groups

Fig. 3a shows the effect of number of join groups. Note that when $g=1$, the join reduces to a Cartesian product. This case can be handled specially as explained in Sec. 6.5. When $g$ is low, the number of $k$-dominant skylines is low since there are more chances of a tuple being $k$-dominated by another one in the same group. Thus, there are less number of $SN$ tuples (none at $g=1$) and their dominators. On the other hand, when $g$ is high, the size of the joined relation (which is $n^{2}/g$) decreases. Therefore, there are two opposing effects on the running time. Empirically, the running times are the highest at medium values.

7.1.3 Effect of Dataset Size

The next experiment varies the size of the base relations, $n$. Note that with increase in $n$, the size of the joined relation increases quadratically ($O(n^{2})$). Consequently, as visible in Fig. 3b, the running time increases drastically. The scalability of the grouping algorithm, in particular, as well as the dominator-based algorithm, is sub-linear in the size of the joined relation, though.

7.1.4 Effect of Type of Data Distribution

The final set of experiments on the aggregated attributes measures the effect of type of data distribution. Fig. 4 shows that correlated datasets are the easiest to process due to higher chances of domination of a tuple by another tuple, thereby resulting in lesser number of skylines. The anti-correlated datasets are the most time consuming due to the opposing effect. The independent datasets are mid-way.

7.2 No Aggregation

The next set of experiments target the scenarios where no aggregation over the skyline attributes is done.

7.2.1 Effect of Dimensionality

Fig. 5a shows the effect of $k$ when $d=5$. Note that since $a=0$, the possible values of $k$ range from $d+1=6$ to $2d-1=9$.

Similar to the case with aggregate attributes, the running time increases sharply with $k$. The grouping algorithm performs the best while the naïve is the worst. The dominator-based algorithm suffers since the time spent in finding the dominators is too large and is not sufficiently compensated later. Interestingly, since the join time is constant for the naïve algorithm irrespective of the value of $k$, the proportion of time spent in joining is much higher for lower $k$.

Fig. 5b holds $k$ constant and varies $d$ over two settings. When $k$ is fixed and $d$ increases, the values of $k^{\prime}_{1}$ and $k^{\prime}_{2}$ decrease. This results in faster grouping time. The dominator sets are also computed faster. Thus, the the overall time decreases.

7.2.2 Effect of Number of Join Groups

As explained earlier in Sec. 7.1.2, the number of join groups has two opposing effects on the running time. Therefore, as shown in Fig. 6a, the results are similar. Note that the values of $d$ and $k$ here are $d=4$ and $k=7$.

7.2.3 Effect of Dataset Size

Fig. 6b shows that the running time increases drastically with $n$, although the scalability is sub-linear in the size of the joined relation, i.e., $n^{2}$. The largest dataset size we tested was for $n=33,000$, leading to a massive size of over $10^{8}$ for the joined relation. The fact that the grouping algorithm produces the result in less than 20 s for this case establishes the practicality of the algorithms.

7.2.4 Effect of Type of Data Distribution

The effect of type of data distribution (Fig. 7) is similar to that in Sec. 7.1.4 with the anti-correlated requiring the largest amount of time and correlated the least.

7.3 Finding $k$

The third and final set of experiments deals with finding the value of $k$ given a threshold $\delta$ of requisite number of $k$-dominant skylines. Aggregate attributes are not used.

7.3.1 Effect of Threshold $\delta$

Fig. 8a shows the effect of varying $\delta$ values. The dimensionality is held fixed at $d=5$ with $a=0$. The possible values of $k$, therefore, range from $d+1=6$ to $2d=10$. The size of each base relation is $n=3,300$ with $g=10$, thereby resulting in more than $10^{6}$ joined tuples (as listed in Table 7).

With increasing $\delta$, the naïve algorithm requires larger running times since it keeps iterating over $k$. The range-based search also iterates over the values of $k$, although it avoids the costly full $k$-dominant skyline computation in the intermediate steps, if possible. When $\delta$ is very large, it falls outside the upper bounds for most values of $k$ and, hence, the algorithm runs very fast. In this experiment, when $\delta\geq 10,000$, the largest possible $k=10$ is returned as the answer.

For low values of $\delta$, the iterations of both naïve and range-based algorithms stop early. The answer for $\delta=10$ is $k=8$ while that for $\delta=100$ and $\delta=1,000$ are both $k=9$.

The binary search method is faster even for medium values of $\delta$ since it avoids the iterative procedure and quickly finds the desired value of $k$. Overall, it is always the fastest algorithm.

7.3.2 Effect of Dimensionality $d$

Fig. 8b shows the complementary effect, that of varying dimensionality $d$ while keeping $\delta$ constant at $10,000$. When $d$ is low, the chosen $\delta$ value is too large and the algorithms terminate fast. When $d$ is increased, the algorithms need to search through a larger range and, therefore, takes a much longer time. The binary search method is consistently the fastest algorithm. It outperforms the range-based algorithm by about 1.2-1.5 times while the naïve algorithm is slower by a factor of 2-2.5.

7.3.3 Effect of Number of Join Groups

There is no appreciable effect of the number of join groups on the algorithms for finding $k$ (Fig. 9a).

7.3.4 Effect of Dataset Size

Fig. 9b shows the effect of increasing dataset size, $n$. The dimensionality and threshold values are kept fixed at $d=5$ and $\delta=1,000$ with $g=10$.

For very low values of $n$ (up to $1,000$), the threshold is too high, and the maximum possible $k=10$ is required to satisfy the threshold $\delta$.

With increasing $n$, the running time increases due to increasing $k$-dominant skyline computation times. Even for larger $n$, the values of $k$ are towards the higher end. Therefore, the binary search algorithm is the most suitable one for finding $k$.

7.3.5 Effect of Data Type

The effect of type of data distribution is as expected with correlated being the fastest and anti-correlated the slowest (Fig. 10a).

7.4 Real Dataset

To test our algorithms on real data, we collected information about various attributes on domestic flights in India from www.makemytrip.com. The first base table contained information about $192$ flights from New Delhi to $13$ important cities of India while the second base table contained information about $155$ flights from those $13$ cities to Mumbai. For each base table, $5$ attributes were considered: cost, flying time, date change fee, popularity, and amenities. The first $2$ attributes were aggregated while the other $3$ were used as local attributes. The join was based on the equality of the intermediate city. Thus, each tuple in the joined relation contained $3+3+2=8$ attributes. The size of the joined relation was $2649$.

We ran experiments on $k=6,7,8$. The results are summarized in Fig. 11. The grouping-based algorithm performed the best followed by the dominator-based method and the naïve algorithm. All the results were produced in milliseconds highlighting the practicality of the methods.

7.5 Summary of Experiments

The experiments show that the grouping algorithm consistently outperforms the other methods in solving the KSJQ queries. For finding the right value of $k$, the binary search algorithm turns out to be the best method always.

8 Conclusions

In this paper, we proposed a novel query, $k$-dominant skyline join query (KSJQ), that incorporates finding $k$-dominant skylines over joined relations where the attributes may be aggregated as well. We analyzed certain optimizations for the query and used them to design efficient algorithms. In addition, given the number of final skylines sought, we also proposed efficient algorithms to find the right value of $k$.

In future, we would like to extend the algorithms to work in parallel, distributed and probabilistic settings.