MinJoin: Efficient Edit Similarity Joins via Local Hash Minima

TL;DR

MinJoin introduces a novel string partition method for efficient and accurate edit similarity joins, outperforming existing algorithms especially on long strings and large thresholds, with linear runtime and perfect accuracy.

Contribution

The paper presents a new string partition based algorithm for edit similarity joins that achieves perfect accuracy and scalable linear runtime, improving over prior methods.

Findings

Outperforms state-of-the-art algorithms in experiments

Achieves perfect accuracy on tested datasets

Runs in linear time plus a verification step

Abstract

We study the problem of computing similarity joins under edit distance on a set of strings. Edit similarity joins is a fundamental problem in databases, data mining and bioinformatics. It finds important applications in data cleaning and integration, collaborative filtering, genome sequence assembly, etc. This problem has attracted significant attention in the past two decades. However, all previous algorithms either cannot scale well to long strings and large similarity thresholds, or suffer from imperfect accuracy. In this paper we propose a new algorithm for edit similarity joins using a novel string partition based approach. We show mathematically that with high probability our algorithm achieves a perfect accuracy, and runs in linear time plus a data-dependent verification step. Experiments on real world datasets show that our algorithm significantly outperforms the state-of-the-art algorithms for edit similarity joins, and achieves perfect accuracy on all the datasets that we have tested.