Preference-driven Similarity Join

TL;DR

This paper introduces a preference-driven approach to similarity join that eliminates the need for threshold tuning by allowing users to select from multiple result-set preferences, optimizing results based on user-defined objectives.

Contribution

It proposes a novel framework that replaces threshold-based methods with preference-based optimization, demonstrated through two preferences and effective algorithms.

Findings

Achieves high-quality results without threshold tuning.

Effectively handles diverse real-world datasets.

Outperforms traditional threshold-driven methods.

Abstract

Similarity join, which can find similar objects (e.g., products, names, addresses) across different sources, is powerful in dealing with variety in big data, especially web data. Threshold-driven similarity join, which has been extensively studied in the past, assumes that a user is able to specify a similarity threshold, and then focuses on how to efficiently return the object pairs whose similarities pass the threshold. We argue that the assumption about a well set similarity threshold may not be valid for two reasons. The optimal thresholds for different similarity join tasks may vary a lot. Moreover, the end-to-end time spent on similarity join is likely to be dominated by a back-and-forth threshold-tuning process. In response, we propose preference-driven similarity join. The key idea is to provide several result-set preferences, rather than a range of thresholds, for a user to choose from. Intuitively, a result-set preference can be considered as an objective function to capture a user's preference on a similarity join result. Once a preference is chosen, we automatically compute the similarity join result optimizing the preference objective. As the proof of concept, we devise two useful preferences and propose a novel preference-driven similarity join framework coupled with effective optimization techniques. Our approaches are evaluated on four real-world web datasets from a diverse range of application scenarios. The experiments show that preference-driven similarity join can achieve high-quality results without a tedious threshold-tuning process.

Tables

Table 1. Table 1. Example of optimal thresholds w.r.t. various tasks.

Dataset	Task	Optimal Threshold	Similarity
Wiki Editors	Spell checking	$0.625$	Jaccard
Restaurants	Record linkage	$0.6$	Jaccard
Scholar-DBLP	Record linkage	$0.34$	Jaccard
Wiki Links	Entity matching	$0.9574$	Tversky

Table 2. Table 2. Summary of set-based similarity functions. Here b o u n d i , j m ​ i ​ n = | r [ : i ] ∩ s [ : j ] | ≤ | r ∩ s | bound^{min}_{i,j}=\left\lvert r[:i]\cap s[:j]\right\rvert\leq\left\lvert r\cap s\right\rvert and b o u n d i , j m ​ a ​ x = | r [ : i ] ∩ s [ : j ] | + min { | r | − i , | s | − j } ≥ | r ∩ s | bound_{i,j}^{max}=\left\lvert r[:i]\cap s[:j]\right\rvert+\min\left\{\left\lvert r\right\rvert-i,\left\lvert s\right\rvert-j\right\}\geq\left\lvert r\cap s\right\rvert .

Similarity	$sim$	$si{m}_{i,j}^{min}$	$si{m}_{i,j}^{max}$	$t_{\theta }$
Jaccard	$\frac{\left|r\cap s\right|}{\left|r\right|+\left|s\right|-\left|r\cap s\right|}$	$\frac{boun{d}_{i,j}^{min}}{\left|r\right|+\left|s\right|-boun{d}_{i,j}^{min}}$	$\frac{boun{d}_{i,j}^{max}}{\left|r\right|+\left|s\right|-boun{d}_{i,j}^{max}}$	$\lceil \theta \cdot \left|r\right|\rceil$
Overlap	$\frac{\left|r\cap s\right|}{\max \{\left|r\right|,\left|s\right|\}}$	$\frac{boun{d}_{i,j}^{min}}{\max \{\left|r\right|,\left|s\right|\}}$	$\frac{boun{d}_{i,j}^{max}}{\max \{\left|r\right|,\left|s\right|\}}$	$\lceil \theta \cdot \left|r\right|\rceil$
Dice	$\frac{2\cdot \left|r\cap s\right|}{\left|r\right|+\left|s\right|}$	$\frac{2\cdot boun{d}_{i,j}^{min}}{\left|r\right|+\left|s\right|}$	$\frac{2\cdot boun{d}_{i,j}^{max}}{\left|r\right|+\left|s\right|}$	$\lceil \frac{\theta }{2}\cdot \left|r\right|\rceil$
Cosine	$\frac{\left|r\cap s\right|}{\sqrt{\left|r\right|\cdot \left|s\right|}}$	$\frac{boun{d}_{i,j}^{min}}{\sqrt{\left|r\right|\cdot \left|s\right|}}$	$\frac{boun{d}_{i,j}^{max}}{\sqrt{\left|r\right|\cdot \left|s\right|}}$	$\lceil {\theta }^2\cdot \left|r\right|\rceil$
Tversky	$\frac{\left|r\cap s\right|}{\alpha \cdot \left|r\right|+\left(1-\alpha \right)\cdot \left|s\right|}$	$\frac{boun{d}_{i,j}^{min}}{\alpha \cdot \left|r\right|+\left(1-\alpha \right)\cdot \left|s\right|}$	$\frac{boun{d}_{i,j}^{max}}{\alpha \cdot \left|r\right|+\left(1-\alpha \right)\cdot \left|s\right|}$	$\lceil \theta \cdot \alpha \cdot \left|r\right|\rceil$

Table 3. Table 3. Dataset characteristics.

Dataset	$R$			$S$			$\left|{ℂ}^{+}\right|$
	$\left|R\right|$	Max. Len.	Avg. Len.	$\left|S\right|$	Max. Len.	Avg. Len.
Wiki Editors	$2,239$	$20$	$9.65$	$1,922$	$16$	$8.66$	$2,455$
Restaurants	$533$	$96$	$48.38$	$331$	$91$	$43.5$	$112$
Scholar-DBLP	$64,259$	$259$	$115.9$	$2,562$	$326$	$106.61$	$5,347$
Wiki Links	$187,122$	$1,393$	$17.08$	$168,652$	$209$	$16.35$	$202,272$

Table 4. Table 4. Sensitivity of thresholds on F 1 subscript 𝐹 1 F_{1} -score. Three thresholds are used: the optimal one θ # superscript 𝜃 # \theta^{\#} , the higher one θ + = min ⁡ { 1 , θ # + 0.1 } superscript 𝜃 1 superscript 𝜃 # 0.1 \theta^{+}=\min\left\{1,\theta^{\#}+0.1\right\} , and the lower one θ − = max ⁡ { 0 , θ # − 0.1 } superscript 𝜃 0 superscript 𝜃 # 0.1 \theta^{-}=\max\left\{0,\theta^{\#}-0.1\right\} . Numbers in red are when ours achieves optimal F 1 subscript 𝐹 1 F_{1} -scores.

Dataset	using ${\theta }^{+}$	using ${\theta }^{\#}$	using ${\theta }^{-}$	Preference-driven
Wiki Editors	$0.599$	$0.764$	$0.705$	$0.764$	using $h_c$ and $h_o$
Restaurants	$0.725$	$0.816$	$0.597$	$0.809$	using $h_o$
Scholar-DBLP	$0.697$	$0.759$	$0.554$	$0.754$	using $h_c$
Wiki Links	$0.774$	$0.780$	$0.624$	$0.780$	using $h_c$

Table 5. Table 5. Accuracy for varying datasets and preferences.

Dataset	$h_c$ (MaxGroups)				$h_o$ (MinOutJoin)
Wiki Editors	$0.625$	$0.837$	$0.704$	$0.764$	$0.625$	$0.837$	$0.704$	$0.764$
Restaurants	$0.429$	$0.291$	$0.938$	$0.444$	$0.556$	$0.805$	$0.813$	$0.809$
Scholar-DBLP	$0.361$	$0.841$	$0.683$	$0.754$	$0.419$	$0.903$	$0.595$	$0.717$
Wiki Links	$0.957$	$0.959$	$0.657$	$0.780$	$0.972$	$0.968$	$0.648$	$0.776$
	${\theta }^{\ast }$	Precision	Recall	$F_1$	${\theta }^{\ast }$	Precision	Recall	$F_1$

Equations

$$join(R,S,sim,\theta )=\left\{(r,s)\in R\times S|sim(r,s)\ge \theta \right\}to0.0pt\text{qed}\text{hss}$$

$$h_c(\theta )=\left|J(G_{\theta })\right|$$

$$outjoin(\theta )=join(\theta )$$

$$\cup \left\{(\text{pagecolor}LightGrayNULL,s)|s\in S\setminus {cover}^S(\theta )\right\}$$

$$h_o(\theta )=\left|R\right|+\left|S\right|-\left|outjoin(\theta )\right|=\left|{cover}^R(\theta )\right|+\left|{cover}^S(\theta )\right|-\left|join(\theta )\right|$$

$$\left|\Theta \right|\le$$

$$\displaystyle\left\lvert\left\{sim(r,s)\,\middle|\,r\in R,s\in S\text{ where }r\in{top}^{R}(s)\right\}\right\rvert\Big{\}}\leq\min\left\{\left\lvert R\right\rvert,\left\lvert S\right\rvert\right\}$$

$$\left|join(\theta )\right|-\left|join({\theta }^{\prime })\right|=\left|joi{n}^{=}(\theta )\right|$$

$$\ge$$

$$+$$

$$=$$

$$\widehat{h_c}({\theta }_i)=h_c({\theta }_i)+\min \left\{\left|R\setminus {cover}^R({\theta }_i)\right|,\left|S\setminus {cover}^S({\theta }_i)\right|\right\}$$

$$\widehat{h_o}({\theta }_i)=h_o({\theta }_i)+\min \left\{\left|R\setminus {cover}^R({\theta }_i)\right|,\left|S\setminus {cover}^S({\theta }_i)\right|\right\}$$

Full text

In response, we propose preference-driven similarity join. The key idea is to provide several result set preferences, rather than a range of thresholds, for a user to choose from. Intuitively, a result set preference can be considered as an objective function to capture a user’s preference on a similarity join result. Once a preference is chosen, we automatically compute the similarity join result optimizing the preference objective. As the proof of concept, we devise two useful preferences and propose a novel preference-driven similarity join framework coupled with effective optimization techniques. Our approaches are evaluated on four real-world web datasets from a diverse range of application scenarios. The experiments show that preference-driven similarity join can achieve high-quality results without a tedious threshold-tuning process.

This work is supported in part by the NSERC Discovery Grant program, the Canada Research Chair program, the NSERC Strategic Grant program, and the SFU President’s Research Start-up Grant. All opinions, findings, conclusions and recommendations in this paper are those of the authors and do not necessarily reflect the views of the funding agencies.

††journalyear: 2017††copyright: acmlicensed††conference: WI ’17; August 23-26, 2017; Leipzig, Germany††price: 15.00††doi: 10.1145/3106426.3106484††isbn: 978-1-4503-4951-2/17/08

1. Introduction

A key characteristic of big data is variety. Data (especially web data) often comes from different sources and the value of data can only be extracted by integrating various sources together. Similarity join, which finds similar objects (e.g., products, people, locations) across different sources, is a powerful tool for tackling the challenge.

For example, suppose a data scientist collects a set of restaurants from Groupon.com and would like to know which restaurants are highly rated on Yelp.com. Since a restaurant may have different representations in the two data sources (e.g., “10 East Main Street” vs. “10 E Main St., #2”), she can use similarity join to find these similar restaurant pairs and integrate the two data sources together.

Threshold-driven similarity join has been extensively studied in the past (DBLP:conf/vldb/ArasuGK06, ; DBLP:conf/icde/ChaudhuriGK06, ; DBLP:conf/www/BayardoMS07, ; DBLP:conf/www/XiaoWLY08, ; DBLP:journals/pvldb/XiaoWL08, ; DBLP:conf/icde/XiaoWLS09, ; DBLP:journals/pvldb/WangLF10, ; DBLP:journals/pvldb/LiDWF11, ; DBLP:conf/sigmod/WangLF12, ; DBLP:journals/pvldb/JiangLFL14, ; DBLP:conf/sigmod/VernicaCL10, ; DBLP:journals/pvldb/MetwallyF12, ; DBLP:conf/icde/DengLHWF14, ; DBLP:conf/sigmod/LuLWLW13, ; DBLP:journals/pvldb/BourosGM12, ). To use it, one has to go through three steps: (a) selecting a similarity function (e.g., Jaccard), (b) selecting a threshold (e.g., $0.8$), and (c) running a similarity join algorithm to find all object pairs whose similarities are at least $0.8$. The existing studies are mainly focused on Step (c). However, both Steps (a) and (b) deeply implicate humans in the loop, which can be orders of magnitude slower than conducting the actual similarity join.

One may argue that, in reality, humans are able to quickly select an appropriate similarity function and a corresponding threshold for a given similarity join task. For choosing similarity function, this may be true because humans can understand the semantics of each similarity function and choose the one that meets their needs.

However, selecting an appropriate threshold may be far from easy. It is extremely difficult for humans to figure out the effect of different thresholds on result quality. Choosing a good threshold depends on not only the specified similarity function but also the underlying data. We conduct an empirical analysis on the optimal thresholds for a diverse range of similarity join tasks, where the optimal thresholds maximize $F_{1}$-scores (DBLP:journals/pvldb/WangLYF11, ). Table 1 shows the results (details of the experiment are in Section 5). We find that the optimal thresholds for the tasks are quite different. Even for the same similarity function, the optimal thresholds may still vary a lot. For example, the optimal threshold of a record-linkage task on the Restaurants dataset is $0.6$, which differs a lot from the optimal threshold of $0.34$ on the Scholar-DBLP dataset using the same similarity function.

To solve this problem, one idea may be to label some data and then use the labeled data as the ground truth of matching pairs to tune the threshold. However, human labeling is error-prone and time-consuming, which significantly increases the (end-to-end) time of data integration or cleaning using similarity join.

In this paper, we tackle this problem from a different angle – can we achieve high-quality similarity join results without requiring humans to label any data or specifying a similarity threshold?

Our key insight is inspired by the concept of preference in areas like economics, which is an ordering of different alternatives (results) (Scholar:preference, ). Taking Yelp.com as an example, the restaurants can be presented in different ways such as by distance, price, or rating. The different ordering may meet different search intents. A user needs to evaluate her query intent and choose the most suitable ranking accordingly.

Similarly, when performing a similarity join, we seek to provide a number of result set preferences for a user to select from. Intuitively, a result set preference can be thought of as an objective function to capture how much a user likes a similarity join result. Once a particular preference is chosen, we automatically tune the threshold such that the preference objective function is maximized, and then we return the corresponding similarity join result. We call this new similarity join model preference-driven similarity join. Compared to the traditional threshold-driven similarity join, this new model does not need any labeled data.

As a proof of concept, our paper proposes two preferences from different yet complementary perspectives. The first preference MaxGroups groups the joined pairs where each group is considered as an entity across two data sources, and returns the join result having the largest number of groups. The second preference MinOutJoin balances between matching highly similar pairs and joining many possibly matching pairs, and favors the join result minimizing the outer-join size.

According to our experiments on various datasets with ground-truth, the preference-driven approach can achieve optimal or nearly optimal $F_{1}$-scores on different tasks without knowing anything about the optimal thresholds.

Given a result set preference, a challenge is how to develop an efficient algorithm for preference-driven similarity join. This problem is more challenging than the traditional threshold-driven similarity join because it involves one additional step: finding the best threshold such that a preference is maximized. The brute-force method needs to compute the similarity values for all the pairs. It is highly inefficient even for datasets of moderate size. We solve this problem by developing a novel similarity join framework along with effective optimization techniques. The experimental results show that the proposed framework achieves several orders of magnitude speedup over the brute-force method.

The rest of the paper is organized as follows. In Section 2, we formally define the problem of preference-driven similarity join. In Section 3, we design two result set preferences from different perspectives. In Section 4, we propose a preference-driven similarity join framework, and develop efficient algorithm for set-based similarity functions. In Section 5, we evaluate our approach on four real-world web datasets from a diverse range of applications. The results suggest that preference-driven similarity join is a promising idea to tackle the threshold selection problem for similarity join, and verify that our method is effective and efficient. We review related work in Section 6 and conclude the paper in Section 7.

2. Problem Definition

Let $R$ and $S$ be two sets of objects, $\mathbb{C^{+}}$ denote the ground-truth that is the set of pairs in $R\times S$ that should be joined/matched, and $\mathbb{C^{-}}$ denote the remaining pairs, i.e., $\mathbb{C^{-}}=(R\times S)\setminus\mathbb{C^{+}}$. We call the pairs in $\mathbb{C^{+}}$ and $\mathbb{C^{-}}$ matching pairs and non-matching pairs, respectively. In general, $\mathbb{C^{+}}$ and $\mathbb{C^{-}}$ are assumed unknown.

Figure 1 shows a toy running example of $R$ and $S$, as well as the ground-truth $\mathbb{C^{+}}$.

Let $sim:(r,s)\in R\times S\rightarrow[0,1]$ denote a similarity function.

Definition 2.1 (Threshold-driven similarity join).

Given a similarity function $sim$ and a threshold $\theta$, return all the object pairs $(r,s)$ whose similarity values are at least $\theta$, that is,

[TABLE]

If $R$, $S$, and $sim$ are clear from the context, we write $join(\theta)$ for the sake of brevity.

We can regard $join(\theta)$ as a classifier, where the pairs returned by the function are the ones classified as positive, and the rest pairs $\left(R\times S\right)\setminus\left(join(\theta)\right)$ classified as negative, that is, not-matching. Figure 1 shows an example of $join(0.5)$ using Jaccard similarity. Here, for simplicity we tokenize a string into a set of characters. For example, jaccard(\texttt{{\color[rgb]{0.618,0,0}db}lp{\color[rgb]{0.618,0,0}\_}},\texttt{{\color[rgb]{0.618,0,0}\_db}})=\frac{\left\lvert r\cap s\right\rvert}{\left\lvert r\cup s\right\rvert}=\frac{\left\lvert\{\texttt{{\color[rgb]{0.618,0,0}\_}},\texttt{{\color[rgb]{0.618,0,0}d}},\texttt{{\color[rgb]{0.618,0,0}b}}\}\right\rvert}{\left\lvert\{\texttt{{\color[rgb]{0.618,0,0}\_}},\texttt{{\color[rgb]{0.618,0,0}d}},\texttt{{\color[rgb]{0.618,0,0}b}},\texttt{l},\texttt{p}\}\right\rvert}=\frac{3}{5}=0.6.

Similarity join seeks to find a threshold that leads to the best result quality. Theoretically, there are an infinite number of thresholds to choose from. However, we only need to consider a finite subset of the possible thresholds, which is the set of the similarity values of all object pairs in $R\times S$, i.e., $\left\{sim(r,s)\,\middle|\,r\in R,s\in S\right\}$, because, for any threshold not in the finite set, there is always a smaller threshold in the finite set having the same join result. For example, threshold $0.9$ is not in the finite set in the example in Figure 1 but $join(0.9)=join(0.8)$.

Threshold tuning is time consuming and labor intensive. Therefore, we develop preference-driven similarity join to overcome the limitations. Before a formal definition, we first introduce the concept of result set preference, to capture the user preferences on a similarity join result. Formally, a result set preference is a score function $h$: $(R,S,sim,\theta)\rightarrow\mathbb{R}$, where $\mathbb{R}$ is the set of real numbers. Obviously, a result set is determined by $R$, $S$, $sim$ and $\theta$. The result set preference gives a score on how well the result set meets a user’s preference. The higher the score, the better. If $R$, $S$, and $sim$ are clear from the context, we write $h(\theta)$ for the sake of brevity.

Definition 2.2 (Preference-driven Similarity Join).

Given a similarity function $sim$ and a result set preference $h$, return the most preferred result $join(\theta^{*})$ where $\theta^{*}$ is the largest threshold in the finite set maximizing $h$. ∎

For the ease of presentation, we introduce some notations. First, we denote by ${join}^{=}(\theta)=\left\{(r,s)\in join(\theta)\,\middle|\,sim(r,s)=\theta\right\}$ the subset of joined pairs w.r.t. similarity $\theta$. For example, in Figure 1, {join}^{=}(0.75)=\left\{\left(\texttt{{\color[rgb]{0.618,0,0}dbs}},\texttt{{\color[rgb]{0.618,0,0}db}m{\color[rgb]{0.618,0,0}s}}\right)\right\}. Second, we denote by ${cover}^{R}(\theta)=\left\{r\,\middle|\,\exists s:(r,s)\in join(\theta)\right\}$ the set of objects in $R$ that are joined when the similarity threshold is $\theta$. Similarly, we have ${cover}^{S}(\theta)=\left\{s\,\middle|\,\exists r:(r,s)\in join(\theta)\right\}$. For example, in Figure 1, ${cover}^{R}(0.75)=\left\{\texttt{{db\_ms}},\texttt{{dbs}},\texttt{{vldb}}\right\}$. Last, for $r\in R$, we denote by $top^{S}(r)$ the set of most similar object(s) in $S$ including ties. Similarly, we have $top^{R}(s)$. For example, in Figure 1, $top^{S}(\texttt{{vldb}})=\left\{\texttt{p{vldb}},\texttt{{vl}\_{db}}\right\}$.

3. Two Result Set Preferences

As a proof of concept, we present two result set preferences.

3.1. MaxGroups: Maximum Number of Non-Trivial Connected Components

Our first preference, MaxGroups, partitions objects into different groups without any prior knowledge. For a similarity threshold $\theta$, we construct a bipartite graph $G_{\theta}=\left(U^{R},V^{S},join(\theta)\right)$, where $U^{R}$ and $V^{S}$ are two disjoint sets of nodes representing the objects in $R$ and $S$, respectively, and every pair in $join(\theta)$ defines an edge. As indicated in (DBLP:conf/dmkd/MongeE97, ), each connected component in the bipartite graph is an entity, where the objects in the same connected component are the entity’s different representations.

MaxGroups prefers the similarity join result with more non-trivial connected components (i.e., connected components with at least two nodes, one in $R$ and another in $S$). The intuition is that heuristically we want to match as many entities as possible across $R$ and $S$. Let $J(G_{\theta})$ denote the set of non-trivial connected components in a bipartite $G_{\theta}$, we define result set preference MaxGroups as

[TABLE]

Figure 2 gives an example of $h_{c}$ on our toy dataset in Figure 1.

Given a similarity join result $join(\theta)$, the time complexity of computing $h_{c}(\theta)$ is $O(\left\lvert R\right\rvert+\left\lvert S\right\rvert+\left\lvert join(\theta)\right\rvert)$ by simply computing the connected components of the bipartite graph $G_{\theta}$. Since we need to compute the preference scores for multiple similarity thresholds $\theta$, there are opportunities to reduce the computational cost. We will discuss the details in Section 4.3.

3.2. MinOutJoin: Minimum Outer-Join Size

We make the following observation. On the one hand, if we set a too high similarity threshold and thus be too strict in similarity, many objects may not be matched with their counterparts due to noise. The extreme case is that, if we set the similarity threshold to $1$, only those perfectly matching objects are joined. On the other hand, if we set a too low similarity threshold and thus be too loose in similarity, many not-matching objects may be joined by mistake. The extreme case is that, by setting the similarity threshold to [math], every pair of objects in the two sets are joined.

We need to find a good balance between the two and strive to a good tradeoff. Technically, full outer-join includes both joined entries and those not joined (by matching with NULL). The size of the full outer-join is jointly determined by the number of objects joined and the number of objects not joined. The two numbers trade off each other. Therefore, if we minimize the size of the full outer-join, we reach a tradeoff between the two ends. This is the intuition behind our second preference, MinOutJoin.

The full outer similarity join result w.r.t. a similarity threshold is

[TABLE]

where $\left\{(r,\hbox{\pagecolor{LightGray}{NULL}})\,\middle|\,r\in R\setminus{cover}^{R}(\theta)\right\}$ is the set of objects in $R$ that are not joined, and $\left\{(\hbox{\pagecolor{LightGray}{NULL}},s)\,\middle|\,s\in S\setminus{cover}^{S}(\theta)\right\}$ is the set of objects in $S$ that are not joined. Figure 3 illustrates an example of a full outer-join.

We define our preference MinOutJoin as

[TABLE]

where $\left\lvert R\right\rvert+\left\lvert S\right\rvert$ is a constant given $R$ and $S$. Figure 2 gives an example of $h_{o}$ on our toy dataset.

This preference gives a penalty when multiple objects in a set are joined with multiple objects in the other set. Joining $x$ objects in $R$ and $y$ objects in $S$ results in $x\cdot y$ pairs in the full outer-join. Not joining them results in at most $x+y$ of pairs in the full outer-join. When $x>1$ and $y>1$, we have $x\cdot y\geq x+y$.

Given a threshold $\theta$, it is straightforward to compute $cover^{R}(\theta)$, $cover^{S}(\theta)$ and $join(\theta)$ according to their definitions by scanning the join result $join(\theta)$. The time complexity of computing $h_{o}(\theta)$ is $O(\left\lvert join(\theta)\right\rvert)$ for each $\theta$. Since we need to compute the preference scores for multiple similarity thresholds $\theta$, there are opportunities to reduce the computational cost. We will discuss the details in Section 4.3.

4. Algorithm Framework

In this section, we present an efficient framework for the preference-driven similarity join problem. A brute-force solution is to compute the similarities for all the object pairs, calculate the preference score w.r.t. each possible threshold, and return the similarity join result with the highest preference score. This brute-force method may be inefficient. Computing similarities for all pairs is often prohibitive. The number of possible thresholds can be very large, $\left\lvert R\right\rvert\times\left\lvert S\right\rvert$ in the worst case. It is crucial to reduce the cost involved in this process.

To tackle the challenges, we propose a preference-driven similarity join framework in Algorithm 1. Central to the framework are four key functions. Function PivotalThresholds (Section 4.1) identifies a small set of thresholds $\Theta$, called the pivotal thresholds, that are guaranteed to cover the best preference score obtained from all the possible thresholds. Function IncrementalSimJoin (Section 4.2) checks the pivotal thresholds in value descending order and incrementally computes the similarity join result for each threshold. We propose a new optimization technique, called lazy evaluation, to further improve the efficiency. Function IncrementalScore (Section 4.3) computes the preference score for each threshold. It is possible to reduce the cost by computing the scores incrementally when checking the pivotal thresholds in value descending order. Function EarlyTermination (Section 4.4) determines whether we can stop checking the remaining pivotal thresholds by comparing the upper bound $\widehat{h}(\theta_{i})$ with currently best score $h(\theta^{*})$ once a similarity join result ${join}(\theta_{i})$ is computed.

4.1. Pivotal Thresholds

Not every threshold has a chance to lead to the maximum preference score. In this section, we study how to identify a small set of thresholds $\Theta$ such that the maximum preference score can be obtained by only evaluating $\Theta$, i.e., $\max_{\theta\in[0,1]}h(\theta)=\max_{\theta\in\Theta}h(\theta)$.

Consider $\Theta=\left\{sim(r,s)\,\middle|\,r\in R,s\in S,r\in{top}^{R}(s)~{}\land~{}s\in{top}^{S}(r)\right\}$. Clearly, $\Theta$ is often dramatically smaller than $\left\lvert R\right\rvert\times\left\lvert S\right\rvert$, as

[TABLE]

For example, in Figure 1, $\Theta=\left\{1,0.8,0.667\right\}$.

We can show that both MaxGroups and MinOutJoin have the same set of pivotal thresholds. The basic idea is that, for any $\theta\notin\Theta$, there exists $\theta^{\prime}>\theta$ such that $h(\theta^{\prime})\geq h(\theta)$. Remind that, through the paper, we only need to discuss the thresholds within the finite set $\left\{sim(r,s)\,\middle|\,r\in R,s\in S\right\}$ as discussed in Section 2.

Lemma 4.1.

Given a threshold $\theta$, if $r\not\in{top}^{R}(s)$ or $s\not\in{top}^{S}(r)$ for any $(r,s)\in{join}^{=}(\theta)$, then $\exists\theta^{\prime}>\theta:h_{c}(\theta^{\prime})\geq h_{c}(\theta)$.

Proof.

Let $\theta^{\prime}$ be a threshold such that $join(\theta)\setminus join(\theta^{\prime})=join^{=}(\theta)$. Bipartite $G_{\theta}$ can be derived by adding those new edges in $join^{=}(\theta)$ to bipartite $G_{\theta^{\prime}}$. For any $(r,s)\in{join}^{=}(\theta)$, if $r\not\in{top}^{R}(s)$, then $r$ must be already in a non-trivial connected component of $G_{\theta^{\prime}}$, and $s$ can only be added to the non-trivial connected component where $r$ belongs to. Similar situation happens if $s\not\in{top}^{S}(r)$. Since there is not any new non-trivial connected component in $G_{\theta}$ comparing to $G_{\theta^{\prime}}$, $h_{c}(\theta^{\prime})\geq h_{c}(\theta)$. ∎

Lemma 4.2.

Given a threshold $\theta$, if $r\not\in{top}^{R}(s)$ and $s\not\in{top}^{S}(r)$ for any $(r,s)\in{join}^{=}(\theta)$, then $\exists\theta^{\prime}>\theta:h_{o}(\theta^{\prime})\geq h_{o}(\theta)$.

Proof.

Let $\theta^{\prime}$ be a threshold such that $join(\theta)\setminus join(\theta^{\prime})=join^{=}(\theta)$. When $r\not\in{top}^{R}(s)~{}\lor~{}s\not\in{top}^{S}(r)$ for any $(r,s)\in{join}^{=}(\theta)$,

[TABLE]

Thus, $h_{o}(\theta^{\prime})-h_{o}(\theta)\geq 0$. ∎

Thanks to the existing fast top-$k$ similarity search algorithms (DBLP:conf/icde/XiaoWLS09, ; DBLP:journals/kais/XuGPWA16, ), obtaining the pivotal thresholds can be efficient by computing the most similar objects of each object in $R$ and $S$, respectively. According to the above lemmas, it is guaranteed that the largest threshold having the maximum score in the finite set of thresholds is always in $\Theta$.

4.2. Incremental Similarity Join

In this section, we present an efficient algorithm that incrementally computes the similarity join result $join(\theta_{i})$ w.r.t. threshold $\theta_{i}$. We do not need to conduct a similarity join for each pivotal threshold. Since $join(\theta)=\cup_{\theta^{\prime}\geq\theta}join^{=}(\theta^{\prime})$, for each threshold $\theta$, we only need to compute the respective newly joined pairs $join^{=}(\theta)$. Thus, we can enumerate the thresholds in the value descending order, and compute the respective join results incrementally.

The algorithm consists of two steps. First, the algorithm incrementally computes a set of candidate pairs $cand(\theta_{i})$. Second, the algorithm evaluates the similarity of each candidate pair and returns the pairs whose similarity values are at least the threshold $\theta_{i}$. While this two-step approach has been used by existing similarity join algorithms (DBLP:conf/icde/ChaudhuriGK06, ; DBLP:conf/www/BayardoMS07, ; DBLP:conf/www/XiaoWLY08, ; DBLP:conf/sigmod/WangLF12, ), our contribution is a novel optimization technique, called lazy evaluation, which lazily evaluate the similarity of each candidate pair and reduces the cost.

We focus on set-based similarity functions in this paper. Similar strategies can be applied to other kinds of similarity functions, like string-based or vector-based. Note that multi-set (bag) can also be used here instead of set. Table 2 shows the definitions of the similarity functions. Jaccard, overlap, dice, and cosine similarity are widely adopted in existing similarity join literature (DBLP:conf/icde/ChaudhuriGK06, ; DBLP:conf/www/BayardoMS07, ; DBLP:conf/www/XiaoWLY08, ; DBLP:conf/sigmod/WangLF12, ). In addition, we include Tversky similarity (Scholar:tversky1977similarity, ), which is a special asymmetric set-based similarity with different weights $\alpha$ and $1-\alpha$ on $r$ and $s$, respectively. This similarity function is very useful in certain scenarios, like matching a text with an entity contained by the text.

Given an object $r$ as a set, we use $r[:i]$ to denote the first $i$ elements of $r$ assuming a global ordering of elements in the set.

4.2.1. Candidate Pair Generation

Established by prefix filtering (DBLP:conf/icde/ChaudhuriGK06, ), if $sim(r,s)\geq\theta$, the number of elements in the overlap of the sets $\left\lvert r\cap s\right\rvert$ is no fewer than an overlap threshold $t_{\theta}$ w.r.t. $\left\lvert r\right\rvert$, where the overlap thresholds for set-based similarities are shown in Table 2. Thus, the candidate pair generation problem is converted to how to filter out the pairs with fewer than $t_{\theta}$ common elements.

To filter out the pairs with less than $t_{\theta}$ common elements, we fix a global ordering on the elements of all the objects, and sort the elements in each object based on the ordering. Like (DBLP:conf/sigmod/WangLF12, ), we use the inverse document frequency as the global ordering. Prefix filtering (DBLP:conf/icde/ChaudhuriGK06, ) establishes that if $\left\lvert r\cap s\right\rvert\geq t_{\theta}$, then $r[:\#prefix_{\theta}(r)]\cap s[:\#prefix_{\theta}(r)]\neq\emptyset$, where $\#prefix_{\theta}(r)=\left\lvert r\right\rvert-t_{\theta}+1$.

Using an inverted index, we do not need to enumerate each pair $(r,s)$ to verify whether $r[:\#prefix_{\theta}(r)]\cap s[:\#prefix_{\theta}(r)]\neq\emptyset$. An inverted index maps an element to a list of objects containing the element. After building the inverted index for $S$, for each $r\in R$, we only need to merge the inverted lists of the elements in $r[:\#prefix_{\theta}(r)]$ to retrieve each $s\in S$ such that $r[:\#prefix_{\theta}(r)]\cap s[:\#prefix_{\theta}(r)]\neq\emptyset$.

Our goal is to generate the candidate pairs for $[\theta_{i},\theta_{i-1})$. We use an incremental prefix filtering approach (DBLP:conf/icde/XiaoWLS09, ) that memorizes previous results to avoid regenerating the candidate pairs for $[\theta_{i-1},1]$.

4.2.2. Lazy Evaluation

Suppose we want to check whether the similarity of a candidate pair $(r,s)\in cand(\theta_{i})$ is no smaller than a threshold $\theta_{i}$ or not. The idea of lazy evaluation is to iteratively compute both a maximum and a minimum possible value of $sim(r,s)$, denoted by $sim^{max}(r,s)$ and $sim^{min}(r,s)$, respectively. Interestingly, both $sim^{max}(r,s)$ and $sim^{min}(r,s)$ get tighter through the process, and finally converge at $sim(r,s)$. During this process, we use the values for lazy evaluation in two ways.

• •

If $sim^{max}(r,s)<\theta_{i}$, then $sim(r,s)<\theta_{i}$. Thus, it is only necessary to resume evaluating $(r,s)$ for a future smaller threshold $\theta_{j}$ where $\theta_{j-1}>sim^{max}(r,s)\geq\theta_{j}$.

• •

If $sim^{min}(r,s)\geq\theta_{i}$, then $\theta_{i-1}>sim(r,s)\geq\theta_{i}$. Thus, $(r,s)$ does not need to be fully evaluated at all.

We scan $r$ and $s$ iteratively together from left to right, according to the global ordering. Assuming $r[:i]$ and $s[:j]$ have been scanned, Table 2 shows the maximum/minimum possible values of $sim(r,s)$. Through the scanning, we iteratively update $sim^{max}(r,s)$ and $sim^{min}(r,s)$ accordingly.

The pseudo-code of the lazy evaluation-powered algorithm is shown in Algorithm 2. The algorithm first computes $cand(\theta_{i})$, by generating a set of candidate pairs for $[\theta_{i},\theta_{i-1})$ together with the previously postponed candidate pairs from larger thresholds. Then, the algorithm examines each candidate pair in $cand(\theta_{i})$ and decides whether it should be added into ${join}^{=}(\theta_{i})$ or postponed and added into $cand(\theta_{j})$ for a smaller threshold $\theta_{j}$ (found by binary search over the rest of the thresholds). Finally, ${join}^{=}(\theta_{i})$ is returned.

4.3. Incremental Score Computation

For both our preferences, computing the preference score for each similarity threshold is straightforward. However, as we need to compute the preference scores for multiple thresholds, it is necessary to explore how to further reduce the cost.

For preference MaxGroups, when computing the join result incrementally by decreasing $\theta$, if two objects of each newly joined pair in $join^{=}(\theta)$ are in different connected components, the connected components are merged together to form a larger connected component. We use a disjoint-set data structure to dynamically track newly joined pairs, and update the connected components accordingly. It only takes almost $O(1)$ amortized time (Scholar:dynamicgraph, ) for each newly joined pair in $join^{=}(\theta)$.

For preference MinOutJoin, when processing incrementally by the value decreasing order of $\theta$, we only need to scan each $join^{=}(\theta)$ to update $join(\theta)$, $cover^{R}(\theta)$, $cover^{S}(\theta)$, and further the preference score. It only takes $O(1)$ time for each newly joined pair in $join^{=}(\theta)$.

4.4. Early Termination

The goal of early termination is to determine if we can return the current most preferred result without evaluating the remaining thresholds. In our algorithm, the thresholds are evaluated in the descending order. Suppose threshold $\theta_{i}$ has just been evaluated. At this point, we have known the preference score for each threshold that is at least $\theta_{i}$. Let $h(\theta^{*})$ denote the current best preference score, i.e., $h(\theta^{*})=\max_{\theta\geq\theta_{i}}h(\theta)$. If we can derive an upper-bound $\widehat{h}(\theta_{i})$ of the preference scores for the remaining thresholds and show that the upper-bound is no larger than $h(\theta^{*})$, then it is safe to stop at $\theta_{i}$ and $h(\theta^{*})$ is the best result overall.

For MaxGroups, as the threshold decreases, a previously unseen non-trivial connected component can only be created by merging two trivial connected components. Since a new non-trivial connected component contains at least one object from $R\setminus{cover}^{R}(\theta_{i})$ and one object from $S\setminus{cover}^{S}(\theta_{i})$, the number of non-trivial connected components in any $G_{\theta^{\prime}}$ such that $\theta^{\prime}<\theta_{i}$ is at most $\min\left\{\left\lvert R\setminus{cover}^{R}(\theta_{i})\right\rvert,\left\lvert S\setminus{cover}^{S}(\theta_{i})\right\rvert\right\}$. Thus, an upper-bound is

[TABLE]

For MinOutJoin, as the threshold decreases, the join result ${join}(\theta_{i})$ includes more pairs. Whenever a new pair $(r,s)$ is joined, $\left\lvert{join}(\theta_{i})\right\rvert$ increases by one. The only way to get the preference score increased by $1$ is that $\left\lvert{cover}^{R}(\theta_{i})\right\rvert$ and $\left\lvert{cover}^{S}(\theta_{i})\right\rvert$ both increase by $1$. In this case, $r$ has to be come from $R\setminus{cover}^{R}(\theta_{i})$ and $s$ has to be come from $S\setminus{cover}^{S}(\theta_{i})$. Therefore, the preference score can at most increase by $\min\left\{\left\lvert R\setminus{cover}^{R}(\theta_{i})\right\rvert,\left\lvert S\setminus{cover}^{S}(\theta_{i})\right\rvert\right\}$. Accordingly, we set an upper-bound to

[TABLE]

5. Experimental Results

We present a series of experimental results in this section. The programs were implemented in Python running with PyPy111PyPy (http://pypy.org/) is an advanced just-in-time compiler, providing $10$ times faster performance for our algorithm than the standard Python interpreter.. The experiments were conducted using a Mac Pro Late $2013$ Server with Intel Xeon $3.70$GHz CPU, $64$GB memory, and $256$GB SSD.

5.1. Datasets

We adopt four real-world web datasets with ground-truth for evaluation. Table 3 shows the characteristics of each dataset.

Wiki Editors (URL:roger, ) is about misspellings of English words, made by Wikipedia page editors where the errors are mostly typos. Each misspelling has at least one correct word. $R$ contains the misspellings, while $S$ contains the correct words. The ground-truth $\mathbb{C^{+}}$ are pairs of each misspelling and the corresponding correct word.

Restaurants (URL:riddle, ) links the restaurant profiles between two websites. Each profile contains the name and address of a restaurant. We remove the phone number and cuisine type, which are available in the original data, to make it more challenging. $R$ and $S$ are profiles of restaurants, and the ground-truth $\mathbb{C^{+}}$ identifies the pairs of profiles linking the same restaurants. Every restaurant has at most one match.

Scholar-DBLP (URL:matching, ) finds the same publications in Google Scholar and DBLP, where each record in DBLP has at least one matching record in Google Scholar. Each record on both websites contains the title, author names, venue, and year. $R$ and $S$ are publications identified by Google Scholar and DBLP, respectively, and the ground-truth $\mathbb{C^{+}}$ are pairs of records linking the same publications.

Wiki Links (URL:wiki_links, ) is a large dataset containing short anchor text on web pages and the Wikipedia link that each anchor text contains. $R$ contains the anchor text, while $S$ contains the Wikipedia entities extracted from Wikipedia links. For example, link https://en.wikipedia.org/wiki/Harry_Potter_(character) is converted to entity “Harry Potter”. The ground-truth $\mathbb{C^{+}}$ are pairs linking each anchor text and its entity. Each anchor text may also contain the text of the entity that the Wikipedia link refers to, such as “…Harry Potter is a title character…”. This dataset has multiple files, where each contains roughly $200,000$ mappings (about 200 MB). We used the first file in most of our evaluation except for the scalability evaluation where the first four files were used.

To adopt set-based similarities, each string needs to be converted to a set. Empirically, people often convert a long string into a bag of words and convert a short string into a set of grams. For the first dataset, since a string represents a word, we convert each string into a bag of $2$-grams. For the rest three datasets, since the strings are much longer, we convert each string into a bag of words. We use Jaccard similarity for the first three datasets. For the Wiki Links dataset, since anchor text often contains both the entity text and some other irrelevant texts, we choose Tversky similarity ($\alpha=0.1$).

5.2. Threshold-driven vs. Preference-driven

In this section, we empirically investigate the pros and cons of both our preference-driven approach and other possible solutions for tuning a similarity join threshold.

We demonstrate the sensitivity of thresholds w.r.t. $F_{1}$-score in Table 4. A small deviation from the optimal threshold affects the result quality dramatically. This clearly shows that threshold tuning is crucial for threshold-driven similarity join. On two datasets, our preference-driven approach can achieve the optimal $F_{1}$-score, and on the other two datasets, they are very close to the optimal.

To compare with human labeling approaches, we adopt two supervised approaches to tune a similarity threshold. The first one is random sampling, where pairs are randomly sampled and labeled. We use the same sampling method as (DBLP:conf/sigmod/GokhaleDDNRSZ14, ), where the larger set $R$ is sampled with a specified sampling rate, and then joined with $S$ to derive all the pairs to be labeled. Apparently, at least $\left\lvert S\right\rvert$ pairs need to be labeled. The second approach (DBLP:conf/cikm/BuchA15, ) uses active learning to tune a threshold by incrementally labeling the most uncertain pair to the classifier. When there is a tie, the one with the highest similarity is selected. The threshold that achieves the best on the labeled data is selected. Figure 4 shows the results. For random sampling, only using a very high sampling rate like $10\%$ can almost catch up with our method. For active learning, the number of necessary labels is significantly reduced, however, still hundreds to thousands of pairs need to be labeled on most of the datasets.

The total running time of the two supervised approaches contains two parts: the labeling part which tunes the threshold, and the joining part using the tuned threshold. In comparison, our approach returns both a threshold and its join results in a unified framework. We calculate the end-to-end time that the threshold-driven approach needs in order to achieve the same quality as our preference-driven approach. For simplicity, we assume that each pair takes $1s$ to be labeled correctly by a human, which is a very conservative estimation.

Figure 5 shows the results. The labeling step is much more costly than the joining step. The end-to-end time of the preference-driven approach is orders of magnitude faster. The preference-driven approach uses one to two times more memory than a threshold-driven approach, due to the need of caching the candidate pairs.

5.3. Accuracy of Preference-driven Approach

In this section, we evaluate the accuracy of our preference-driven approach. Table 5 compares the results between $h_{c}$ and $h_{o}$. The precision, recall, and $F_{1}$-score are presented, together with the preferred threshold $\theta^{*}$. For Wiki Editors, due to its nature of high similarity between misspellings and correct words, both preferences return the same optimal result. For the record-linkage task on Restaurants, $h_{o}$ gives the significantly better result as it favors one-to-one matching. For Scholar-DBLP, $h_{c}$ gives the better result, because maximizing the number of non-trivial connected components actually satisfies the nature of many-to-many matching. For the Wiki Links, $h_{c}$ gives the optimal result, and $h_{o}$ is quite close.

5.4. Efficiency of Preference-driven Approach

There is no existing work solving the same problem. We choose the brute-force method in Section 4 as a baseline to evaluate the efficiency. This method takes almost the same time and memory regardless of the preference due to the same amortized time for processing each newly joined pair incrementally.

Figure 7(a) shows the number of thresholds evaluated by the baseline and our algorithm. On most of the datasets, our algorithm evaluates $10$ to $100$ times less thresholds than the baseline. Our method achieves a significant speedup due to the combination of other optimization techniques (i.e., incremental similarity join and early termination). As Figure 7(b) shows, our algorithm is $10$ to $100$ times faster than the baseline on all the large datasets. Figure 7(c) shows that our method consumes significantly less memory.

For incremental similarity join, we use lazy evaluation for speedup. Instead, a simple approach computes the exact similarities for all the new candidate pairs produced by each threshold, and puts them into a max-heap. Those pairs in the heap whose similarities are no less than the current threshold are popped out for evaluation. On smaller datasets Wiki Editors and Restaurants, the simple approach works slightly better, as there are not many pairs that need lazy evaluation. However, on larger datasets Scholar-DBLP and Wiki Links, our lazy evaluation approach is $2$ to $5$ times faster.

For scalability, we evaluate our algorithm using $h_{c}$ on a larger version of Wiki Links that contains 4 files. There are overlapping objects between different parts. Figure 7 shows the scalability results on number of evaluated thresholds, running time, and peak memory usage. Our method is scalable.

6. Related Work

Due to the crucial role of similarity join in data integration and data cleaning, a large number of similarity join algorithms have been proposed (DBLP:conf/vldb/ArasuGK06, ; DBLP:conf/icde/ChaudhuriGK06, ; DBLP:conf/www/BayardoMS07, ; DBLP:conf/www/XiaoWLY08, ; DBLP:journals/pvldb/XiaoWL08, ; DBLP:conf/icde/XiaoWLS09, ; DBLP:journals/pvldb/WangLF10, ; DBLP:journals/pvldb/LiDWF11, ; DBLP:conf/sigmod/WangLF12, ; DBLP:journals/pvldb/JiangLFL14, ). There are also scalable implementations of the algorithms using the MapReduce framework (DBLP:conf/sigmod/VernicaCL10, ; DBLP:journals/pvldb/MetwallyF12, ; DBLP:conf/icde/DengLHWF14, ). Top-k similarity join is also explored (DBLP:conf/icde/XiaoWLS09, ; DBLP:journals/kais/XuGPWA16, ; DBLP:conf/icde/KimS12, ).

While the majority of the existing work on similarity join needs to specify a similarity threshold or a limit of the number of results returned, there do exist some studies that seek to find a suitable threshold for similarity join in a supervised way (DBLP:journals/pvldb/WangLYF11, ; DBLP:conf/vldb/ChaudhuriCGK07, ; DBLP:conf/sigmod/ArasuGK10, ; DBLP:conf/cikm/BuchA15, ). Both (DBLP:conf/sigmod/ArasuGK10, ) and (DBLP:conf/cikm/BuchA15, ) adopted active learning to tweak the threshold. Chaudhuri et al. (DBLP:conf/vldb/ChaudhuriCGK07, ) learned an operator tree, where each node contains a similarity threshold and a similarity function on each of the splitting attributes. Wang et al. (DBLP:journals/pvldb/WangLYF11, ) modeled this problem as an optimization problem and applied hill climbing to optimize the threshold-selection process. Nevertheless, all these methods need humans to label a number of pairs, which are selected based on either random sampling or active learning. In comparison, our method does not need any labeled data.

There are studies about preferences in the database research (See (DBLP:journals/tods/StefanidisKP11, ) for a thorough survey.). However, they mainly focus on how the joined tuples are ranked and selected, instead of how the tables (and in our case sets of objects) are joined to generate the joined tuples. Alexe et al. (DBLP:journals/pvldb/AlexeRT14, ) discussed user-defined preference rules for integrating temporal data, which is orthogonal to our work. To the best of our knowledge, (DBLP:conf/iri/GaoPWC17, ) is the only work discussing result set preference for joining relational tables.

7. Conclusions

Threshold selection, usually neglected in the similarity join literature, can actually be a bottleneck in an end-to-end similarity join process. To mitigate the challenge, we propose preference-driven similarity join in this paper. We formalize preference-driven similarity join, and propose the notion of result set preference. We present two specific preferences, MaxGroups and MinOutJoin as proofs of concept. We develop a general framework for preference-driven similarity join and the efficient computation methods. We evaluate our approach on four real-world datasets from a diverse range of application scenarios. The results demonstrate that preference-driven similarity join can achieves high-quality results without any labeled data, and our proposed framework is efficient and scalable. Future directions include developing more preferences for various application scenarios and supporting various similarity functions.