Space- and Computationally-Efficient Set Reconciliation via Parity Bitmap Sketch (PBS)

TL;DR

This paper introduces PBS, a set reconciliation method combining low computational complexity and near-minimal communication overhead, improving efficiency over existing schemes.

Contribution

The paper presents PBS, a novel ECC-based set reconciliation scheme with O(d) complexity and near-optimal communication, along with a rigorous analytical framework for performance analysis.

Findings

PBS achieves O(d) computational complexity.

PBS's communication overhead is roughly twice the theoretical minimum.

The analytical framework enables precise performance metric calculations.

Abstract

Set reconciliation is a fundamental algorithmic problem that arises in many networking, system, and database applications. In this problem, two large sets A and B of objects (bitcoins, files, records, etc.) are stored respectively at two different network-connected hosts, which we name Alice and Bob respectively. Alice and Bob communicate with each other to learn $A\Delta B$, the difference between A and B, and as a result the reconciled set $A\bigcup B$. Current set reconciliation schemes are based on either Invertible Bloom Filters (IBF) or Error-Correction Codes (ECC). The former has a low computational complexity of O(d), where d is the cardinality of $A\Delta B$, but has a high communication overhead that is several times larger than the theoretical minimum. The latter has a low communication overhead close to the theoretical minimum, but has a much higher computational complexity of $O(d^2)$. In this work, we propose Parity Bitmap Sketch (PBS), an ECC- based set reconciliation scheme that gets the better of both worlds: PBS has both a low computational complexity of O(d) just like IBF-based solutions and a low communication overhead of roughly twice the theoretical minimum. A separate contribution of this work is a novel rigorous analytical framework that can be used for the precise calculation of various performance metrics and for the near-optimal parameter tuning of PBS.