Rateless Bloom Filters: Set Reconciliation for Divergent Replicas with Variable-Sized Elements

TL;DR

This paper introduces Rateless Bloom Filters, a dynamic set reconciliation method that efficiently handles variable-sized elements and large differences without prior parameter knowledge, reducing communication costs.

Contribution

The paper presents the Rateless Bloom Filter, a novel adaptive filter that optimally adjusts to set differences without prior knowledge, improving efficiency in set reconciliation.

Findings

Reduces communication cost by over 20% for Jaccard indices below 85%.

Effectively handles variable-sized elements and large set differences.

Outperforms state-of-the-art protocols in diverse scenarios.

Abstract

Set reconciliation protocols typically make two critical assumptions: they are designed for fixed-sized elements and they are optimized for when the difference cardinality, d, is very small. When adapting to variable-sized elements, the current practice is to synchronize fixed-size element digests. However, when the number of differences is considerable, such as after a network partition, this approach can be inefficient. Our solution is a two-stage hybrid protocol that introduces a preliminary Bloom filter step, specifically designed for this regime. The novelty of this approach, however, is in solving a core technical challenge: determining the optimal Bloom filter size without knowing d. Our solution is the Rateless Bloom Filter (RBF), a dynamic filter that naturally adapts to arbitrary symmetric differences, closely matching the communication complexity of an optimally configured static filter without requiring any prior parametrization. Our evaluation in sets of variable-sized elements shows that for Jaccard indices below 85%, our RBF-IBLT hybrid protocol reduces the total communication cost by up to over 20% compared to the state-of-the-art.