A Lightweight Approach for Improving the Lookup Performance in Kademlia-type Systems

TL;DR

This paper introduces a neighbor diversification scheme for Kademlia-type systems that improves lookup performance by reducing hop counts, validated through theoretical modeling, simulations, and real-world deployment.

Contribution

It proposes a backward-compatible neighbor selection scheme that enhances lookup efficiency by increasing identifier diversity within routing tables.

Findings

Reduced hop count in simulations

Performance gains in real KAD system

Theoretical model matches simulation results

Abstract

Discovery of nodes and content in large-scale distributed systems is generally based on Kademlia, today. Understanding Kademlia-type systems to improve their performance is essential for maintaining a high service quality for an increased number of participants, particularly when those systems are adopted by latency-sensitive applications. This paper contributes to the understanding of Kademlia by studying the impact of \emph{diversifying} neighbours' identifiers within each routing table bucket on the lookup performance. We propose a new, yet backward-compatible, neighbour selection scheme that attempts to maximize the aforementioned diversity. The scheme does not cause additional overhead except negligible computations for comparing the diversity of identifiers. We present a theoretical model for the actual impact of the new scheme on the lookup's hop count and validate it against simulations of three exemplary Kademlia-type systems. We also measure the performance gain enabled by a partial deployment for the scheme in the real KAD system. The results confirm the superiority of the systems that incorporate our scheme.