Maintaining $k$-MinHash Signatures over Fully-Dynamic Data Streams with Recovery

TL;DR

This paper introduces a buffered, dynamic version of the $k$-MinHash sketch that efficiently manages updates in fully-dynamic data streams, enabling fast similarity queries with reduced rebuilds and maintaining high accuracy.

Contribution

It presents a novel buffered $k$-MinHash data structure that improves update efficiency and reduces costly rebuilds in dynamic data streams.

Findings

Uses $O(k \, \log |U|)$ memory per subset.

Achieves $O(k \, \log |U|)$ amortized update time.

Returns exact $k$-MinHash signatures in $O(k)$ time.

Abstract

We consider the task of performing Jaccard similarity queries over a large collection of items that are dynamically updated according to a streaming input model. An item here is a subset of a large universe $U$ of elements. A well-studied approach to address this important problem in data mining is to design fast-similarity data sketches. In this paper, we focus on global solutions for this problem, i.e., a single data structure which is able to answer both Similarity Estimation and All-Candidate Pairs queries, while also dynamically managing an arbitrary, online sequence of element insertions and deletions received in input. We introduce and provide an in-depth analysis of a dynamic, buffered version of the well-known $k$-MinHash sketch. This buffered version better manages critical update operations thus significantly reducing the number of times the sketch needs to be rebuilt from scratch using expensive recovery queries. We prove that the buffered $k$-MinHash uses $O(k \log |U|)$ memory words per subset and that its amortized update time per insertion/deletion is $O(k \log |U|)$ with high probability. Moreover, our data structure can return the $k$-MinHash signature of any subset in $O(k)$ time, and this signature is exactly the same signature that would be computed from scratch (and thus the quality of the signature is the same as the one guaranteed by the static $k$-MinHash). Analytical and experimental comparisons with the other, state-of-the-art global solutions for this problem given in [Bury et al.,WSDM'18] show that the buffered $k$-MinHash turns out to be competitive in a wide and relevant range of the online input parameters.