A Distributed Learned Hash Table

TL;DR

LEAD introduces a machine learning-based approach to significantly improve range query efficiency in distributed hash tables, reducing latency and message costs while maintaining scalability and robustness.

Contribution

The paper presents LEAD, a novel system that integrates learned models into DHTs to optimize range query performance and resilience.

Findings

Reduces range query latency and message cost by 80-90%.

Demonstrates high scalability and robustness in large-scale systems.

Outperforms existing range query methods in efficiency.

Abstract

Distributed Hash Tables (DHTs) are pivotal in numerous high-impact key-value applications built on distributed networked systems, offering a decentralized architecture that avoids single points of failure and improves data availability. Despite their widespread utility, DHTs face substantial challenges in handling range queries, which are crucial for applications such as LLM serving, distributed storage, databases, content delivery networks, and blockchains. To address this limitation, we present LEAD, a novel system incorporating learned models within DHT structures to significantly optimize range query performance. LEAD utilizes a recursive machine learning model to map and retrieve data across a distributed system while preserving the inherent order of data. LEAD includes the designs to minimize range query latency and message cost while maintaining high scalability and resilience to network churn. Our comprehensive evaluations, conducted in both testbed implementation and simulations, demonstrate that LEAD achieves tremendous advantages in system efficiency compared to existing range query methods in large-scale distributed systems, reducing query latency and message cost by 80% to 90%+. Furthermore, LEAD exhibits remarkable scalability and robustness against system churn, providing a robust, scalable solution for efficient data retrieval in distributed key-value systems.