Advanced Bloom Filter Based Algorithms for Efficient Approximate Data De-Duplication in Streams

TL;DR

This paper introduces novel Bloom Filter algorithms, including RSBF and BSBF variants, for efficient approximate duplicate detection in streaming data, with theoretical bounds and empirical validation on large datasets.

Contribution

It presents new Bloom Filter-based algorithms with theoretical analysis and demonstrates their superior performance over existing methods in large-scale data streams.

Findings

Proposed algorithms outperform existing duplicate detection methods.

Theoretical bounds on false positive, false negative, and convergence rates are established.

Empirical results on real and synthetic datasets validate the effectiveness of the models.

Abstract

Applications involving telecommunication call data records, web pages, online transactions, medical records, stock markets, climate warning systems, etc., necessitate efficient management and processing of such massively exponential amount of data from diverse sources. De-duplication or Intelligent Compression in streaming scenarios for approximate identification and elimination of duplicates from such unbounded data stream is a greater challenge given the real-time nature of data arrival. Stable Bloom Filters (SBF) addresses this problem to a certain extent. . In this work, we present several novel algorithms for the problem of approximate detection of duplicates in data streams. We propose the Reservoir Sampling based Bloom Filter (RSBF) combining the working principle of reservoir sampling and Bloom Filters. We also present variants of the novel Biased Sampling based Bloom Filter (BSBF) based on biased sampling concepts. We also propose a randomized load balanced variant of the sampling Bloom Filter approach to efficiently tackle the duplicate detection. In this work, we thus provide a generic framework for de-duplication using Bloom Filters. Using detailed theoretical analysis we prove analytical bounds on the false positive rate, false negative rate and convergence rate of the proposed structures. We exhibit that our models clearly outperform the existing methods. We also demonstrate empirical analysis of the structures using real-world datasets (3 million records) and also with synthetic datasets (1 billion records) capturing various input distributions.