ChainedFilter: Combining Membership Filters by Chain Rule
Haoyu Li, Liuhui Wang, Qizhi Chen, Jianan Ji, Yuhan Wu, Yikai Zhao,, Tong Yang, Aditya Akella

TL;DR
This paper introduces a unified theory called chain rule for membership testing, enabling the combination of approximate and exact filters, and presents ChainedFilter, a versatile algorithm that improves space efficiency and performance across various applications.
Contribution
The paper develops a complete theoretical framework for membership problems and proposes ChainedFilter, a novel algorithm that effectively combines elementary filters without information loss.
Findings
Requires only 26% space over the lower bound for static dictionaries
Uses only 0.22 bits per item more than the lower bound for lossless compression
Reduces external memory access by up to 31% compared to Cuckoo Hashing
Abstract
Membership (membership query / membership testing) is a fundamental problem across databases, networks and security. However, previous research has primarily focused on either approximate solutions, such as Bloom Filters, or exact methods, like perfect hashing and dictionaries, without attempting to develop a an integral theory. In this paper, we propose a unified and complete theory, namely chain rule, for general membership problems, which encompasses both approximate and exact membership as extreme cases. Building upon the chain rule, we introduce a straightforward yet versatile algorithm framework, namely ChainedFilter, to combine different elementary filters without losing information. Our evaluation results demonstrate that ChainedFilter performs well in many applications: (1) it requires only 26% additional space over the theoretical lower bound for implicit static dictionary,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsCaching and Content Delivery · Internet Traffic Analysis and Secure E-voting · Covalent Organic Framework Applications
