ApproxJoin: Approximate Matching for Efficient Verification in Fuzzy Set Similarity Join

TL;DR

ApproxJoin introduces approximate maximum weight matching algorithms to significantly improve the efficiency of fuzzy set similarity join verification while maintaining high accuracy, outperforming state-of-the-art exact methods.

Contribution

It is the first to apply approximate matching algorithms for verification in fuzzy set similarity joins, achieving substantial performance gains.

Findings

Performance improvements of 2-19x over state-of-the-art methods.

High accuracy with 99% recall maintained.

Evaluation of three approximate matching algorithms.

Abstract

The set similarity join problem is a fundamental problem in data processing and discovery, relying on exact similarity measures between sets. In the presence of alterations, such as misspellings on string data, the fuzzy set similarity join problem instead approximately matches pairs of elements based on the maximum weighted matching of the bipartite graph representation of sets. State-of-the-art methods within this domain improve performance through efficient filtering methods within the filter-verify framework, primarily to offset high verification costs induced by the usage of the Hungarian algorithm - an optimal matching method. Instead, we directly target the verification process to assess the efficacy of more efficient matching methods within candidate pair pruning. We present ApproxJoin, the first work of its kind in applying approximate maximum weight matching algorithms for computationally expensive fuzzy set similarity join verification. We comprehensively test the performance of three approximate matching methods: the Greedy, Locally Dominant and Paz Schwartzman methods, and compare with the state-of-the-art approach using exact matching. Our experimental results show that ApproxJoin yields performance improvements of 2-19x the state-of-the-art with high accuracy (99% recall).