A Shifting Bloom Filter Framework for Set Queries

TL;DR

This paper introduces the Shifting Bloom Filter framework, a memory-efficient probabilistic data structure that improves set query processing for membership, association, and multiplicity queries, demonstrated through real-world network data.

Contribution

The paper presents the novel Shifting Bloom Filter framework that encodes auxiliary information via location offsets, reducing memory usage compared to prior Bloom Filter-based structures.

Findings

ShBF significantly outperforms previous methods on set query tasks.

ShBF uses less memory while maintaining high accuracy.

Experimental results validate the effectiveness of ShBF on real-world data.

Abstract

Set queries are fundamental operations in computer systems and applications.This paper addresses the fundamental problem of designing a probabilistic data structure that can quickly process set queries using a small amount of memory. We propose a Shifting Bloom Filter (ShBF) framework for representing and querying sets. We demonstrate the effectiveness of ShBF using three types of popular set queries: membership, association, and multiplicity queries. The key novelty of ShBF is on encoding the auxiliary information of a set element in a location offset. In contrast, prior BF based set data structures allocate additional memory to store auxiliary information. To evaluate ShBF in comparison with prior art, we conducted experiments using real-world network traces. Results show that ShBF significantly advances the state-of-the-art on all three types of set queries.