Improving Distributed Similarity Join in Metric Space with Error-bounded Sampling

TL;DR

This paper introduces SP-Join, a distributed similarity join framework in metric space that uses error-bounded stratified sampling and a cost model to improve partition quality and scalability in Big Data environments.

Contribution

The paper presents SP-Join, a novel distributed similarity join framework that guarantees pivot quality and optimized data partitioning using estimation-based sampling and cost modeling.

Findings

SP-Join outperforms existing methods in efficiency and accuracy.

The framework achieves balanced data partitions with quality guarantees.

Experimental results demonstrate significant performance improvements.

Abstract

Given two sets of objects, metric similarity join finds all similar pairs of objects according to a particular distance function in metric space. There is an increasing demand to provide a scalable similarity join framework which can support efficient query and analytical services in the era of Big Data. The existing distributed metric similarity join algorithms adopt random sampling techniques to produce pivots and utilize holistic partitioning methods based on the generated pivots to partition data, which results in data skew problem since both the generated pivots and the partition strategies have no quality guarantees. To address the limitation, we propose SP-Join, an end-to-end framework to support distributed similarity join in metric space based on the MapReduce paradigm, which (i) employs an estimation-based stratified sampling method to produce pivots with quality guarantees for any sample size, and (ii) devises an effective cost model as the guideline to split the whole datasets into partition in map and reduce phases according to the sampled pivots. Although obtaining an optimal set of partitions is proven to be NP-Hard, SP-Join adopts efficient partitioning strategies based on such a cost model to achieve an even partition with explainable quality. We implement our framework upon Apache Spark platform and conduct extensive experiments on four real-world datasets. The results show that our method significantly outperforms state-of-the-art methods.