Similarity Join and Similarity Self-Join Size Estimation in a Streaming Environment

TL;DR

This paper introduces a streaming algorithm for estimating the size of similarity self-joins in high-dimensional data, achieving high accuracy with sublinear space and one-pass processing, useful for data cleaning and query optimization.

Contribution

It extends similarity join size estimation to d-ary data in streaming environments, providing an efficient one-pass algorithm with proven error bounds and superior accuracy over competitors.

Findings

The algorithm achieves an order of magnitude less error than competitors.

It scales well with input size and maintains accuracy across various similarity thresholds.

Experimental results validate the theoretical error bounds and efficiency.

Abstract

We study the problem of similarity self-join and similarity join size estimation in a streaming setting where the goal is to estimate, in one scan of the input and with sublinear space in the input size, the number of record pairs that have a similarity within a given threshold. The problem has many applications in data cleaning and query plan generation, where the cost of a similarity join may be estimated before actually doing the join. On unary input where two records either match or don't match, the problem becomes join and self-join size estimation for which one-pass algorithms are readily available. Our work addresses the problem for d-ary input, for d >= 1, where the degree of similarity can vary from 1 to d. We show that our proposed algorithm gives an accurate estimate and scales well with the input size. We provide error bounds and time and space costs, and conduct an extensive experimental evaluation of our algorithm, comparing its estimation accuracy to a few competitors, including some multi-pass algorithms. Our results show that given the same space, the proposed algorithm has an order of magnitude less error for a large range of similarity thresholds.