SimiSketch: Efficiently Estimating Similarity of streaming Multisets

TL;DR

SimiSketch is a novel streaming algorithm for estimating similarity between multisets, significantly outperforming existing methods in accuracy and throughput, and is applicable to real-world network and text data.

Contribution

This paper introduces SimiSketch, a new sketching algorithm specifically designed for streaming multisets, improving accuracy and efficiency over prior set-based similarity estimation methods.

Findings

SimiSketch improves accuracy by up to 42 times compared to state-of-the-art.

SimiSketch increases throughput by up to 360 times.

Validated on synthetic, network, and text datasets.

Abstract

The challenge of estimating similarity between sets has been a significant concern in data science, finding diverse applications across various domains. However, previous approaches, such as MinHash, have predominantly centered around hashing techniques, which are well-suited for sets but less naturally adaptable to multisets, a common occurrence in scenarios like network streams and text data. Moreover, with the increasing prevalence of data arriving in streaming patterns, many existing methods struggle to handle cases where set items are presented in a continuous stream. Consequently, our focus in this paper is on the challenging scenario of multisets with item streams. To address this, we propose SimiSketch, a sketching algorithm designed to tackle this specific problem. The paper begins by presenting two simpler versions that employ intuitive sketches for similarity estimation. Subsequently, we formally introduce SimiSketch and leverage SALSA to enhance accuracy. To validate our algorithms, we conduct extensive testing on synthetic datasets, real-world network traffic, and text articles. Our experiment shows that compared with the state-of-the-art, SimiSketch can improve the accuracy by up to 42 times, and increase the throughput by up to 360 times. The complete source code is open-sourced and available on GitHub for reference.