Fingerprint Filters Are Optimal

TL;DR

This paper proves a tight lower bound on the space complexity of dynamic filters with false-positive rate , showing that fingerprinting-based filters are optimal in space usage regardless of their operation time.

Contribution

It establishes a sharp lower bound matching the space used by fingerprint filters, resolving a decades-old open question about their optimality.

Findings

Proves a tight lower bound of n log ^{-1} + n log e - o(n) bits for dynamic filters.

Shows fingerprint filters are space-optimal among all dynamic filters.

Addresses a central open problem in the design of approximate membership data structures.

Abstract

Dynamic filters are data structures supporting approximate membership queries to a dynamic set $S$ of $n$ keys, allowing a small false-positive error rate $\varepsilon$, under insertions and deletions to the set $S$. Essentially all known constructions for dynamic filters use a technique known as fingerprinting. This technique, which was first introduced by Carter et al. in 1978, inherently requires $$\log \binom{n \varepsilon^{-1}}{n} = n \log \varepsilon^{-1} + n \log e - o(n)$$ bits of space when $\varepsilon = o(1)$. Whether or not this bound is optimal for all dynamic filters (rather than just for fingerprint filters) has remained for decades as one of the central open questions in the area. We resolve this question by proving a sharp lower bound of $n \log \varepsilon^{-1} + n \log e - o(n)$ bits for $\varepsilon = o(1)$, regardless of operation time.