Multiple Set Matching and Pre-Filtering with Bloom Multifilters

TL;DR

This paper introduces two novel Bloom Multifilters, Bloom Matrix and Bloom Vector, for efficient multiple set matching, optimizing space and query performance based on data distribution, with a new testing method for selecting the best structure.

Contribution

Proposes Bloom Matrix and Bloom Vector, new probabilistic data structures for multiple set matching, with adaptive space optimization and a Bloom Test for structure selection.

Findings

Bloom Vector exploits Zipf distribution for space savings.

Bloom Matrix offers faster ADD and LOOKUP operations.

Bloom Matrix's false positive rate is less than 10^{-2} only under uniform distribution.

Abstract

Bloom filter is a space-efficient probabilistic data structure for checking elements' membership in a set. Given multiple sets, however, a standard Bloom filter is not sufficient when looking for the items to which an element or a set of input elements belong to. In this article, we solve multiple set matching problem by proposing two efficient Bloom Multifilters called Bloom Matrix and Bloom Vector. Both of them are space efficient and answer queries with a set of identifiers for multiple set matching problems. We show that the space efficiency can be optimized further according to the distribution of labels among multiple sets: Uniform and Zipf. While both of them are space efficient, Bloom Vector can efficiently exploit Zipf distribution of data for further space reduction. Our results also highlight that basic ADD and LOOKUP operations on Bloom Matrix are faster than on Bloom Vector. However, Bloom Matrix does not meet the theoretical false positive rate of less than $10^{-2}$ for LOOKUP operations if the represented data or the labels are not uniformly distributed among the multiple sets. Consequently, we introduce \textit{Bloom Test} which uses Bloom Matrix as the pre-filter structure to determine which structure is suitable for improved performance with an arbitrary input dataset.