Highly Dynamic and Fully Distributed Data Structures

TL;DR

This paper introduces a fully distributed, scalable skip list maintenance algorithm that remains robust under high churn in dynamic P2P networks, with low overhead and no global knowledge required.

Contribution

The authors present the first fully distributed data structure that maintains a skip list efficiently under high churn, using novel merging and deletion algorithms.

Findings

Maintains a skip list for polynomially many rounds despite high churn.

Uses small messages and limited communication per node.

Includes novel algorithms for merging and deleting skip list elements in logarithmic rounds.

Abstract

We study robust and efficient distributed algorithms for building and maintaining distributed data structures in dynamic Peer-to-Peer (P2P) networks. P2P networks are characterized by a high level of dynamicity with abrupt heavy node \emph{churn} (nodes that join and leave the network continuously over time). We present a novel algorithm that builds and maintains with high probability a skip list for $poly(n)$ rounds despite $\mathcal{O}(n/\log n)$ churn \emph{per round} ($n$ is the stable network size). We assume that the churn is controlled by an oblivious adversary (that has complete knowledge and control of what nodes join and leave and at what time and has unlimited computational power, but is oblivious to the random choices made by the algorithm). Moreover, the maintenance overhead is proportional to the churn rate. Furthermore, the algorithm is scalable in that the messages are small (i.e., at most $polylog(n)$ bits) and every node sends and receives at most $polylog(n)$ messages per round. Our algorithm crucially relies on novel distributed and parallel algorithms to merge two $n$-elements skip lists and delete a large subset of items, both in $\mathcal{O}(\log n)$ rounds with high probability. These procedures may be of independent interest due to their elegance and potential applicability in other contexts in distributed data structures. To the best of our knowledge, our work provides the first-known fully-distributed data structure that provably works under highly dynamic settings (i.e., high churn rate). Furthermore, they are localized (i.e., do not require any global topological knowledge). Finally, we believe that our framework can be generalized to other distributed and dynamic data structures including graphs, potentially leading to stable distributed computation despite heavy churn.