Distinct Sampling on Streaming Data with Near-Duplicates

TL;DR

This paper introduces algorithms for distinct sampling in streaming data with near-duplicates, ensuring uniformity and efficiency, including extensions to sliding windows, supported by theoretical guarantees and experimental validation.

Contribution

It proposes novel algorithms for near-duplicate-aware distinct sampling in streaming and sliding window models with proven theoretical guarantees.

Findings

Algorithms achieve uniform sampling of distinct elements.

The methods are effective on Euclidean space datasets.

Experimental results validate theoretical claims.

Abstract

In this paper we study how to perform distinct sampling in the streaming model where data contain near-duplicates. The goal of distinct sampling is to return a distinct element uniformly at random from the universe of elements, given that all the near-duplicates are treated as the same element. We also extend the result to the sliding window cases in which we are only interested in the most recent items. We present algorithms with provable theoretical guarantees for datasets in the Euclidean space, and also verify their effectiveness via an extensive set of experiments.