Local Rendezvous Hashing: Bounded Loads and Minimal Churn via Cache-Local Candidates

TL;DR

This paper introduces Local Rendezvous Hashing (LRH), a cache-efficient method for consistent hashing that reduces load imbalance and churn while significantly improving lookup speed in distributed systems.

Contribution

LRH preserves a token ring structure but limits candidate selection to a cache-local window, reducing memory traffic and improving load balance and churn management compared to existing methods.

Findings

LRH reduces maximum to average load ratio from 1.2785 to 1.0947.

LRH achieves approximately 6.8 times faster throughput than multi-probe hashing.

LRH maintains load balance while significantly decreasing memory traffic and lookup latency.

Abstract

Consistent hashing is fundamental to distributed systems, but ring-based schemes can exhibit high peak-to-average load ratios unless they use many virtual nodes, while multi-probe methods improve balance at the cost of scattered memory accesses. This paper introduces Local Rendezvous Hashing (LRH), which preserves a token ring but restricts Highest Random Weight (HRW) selection to a cache-local window of C distinct neighboring physical nodes. LRH locates a key by one binary search, enumerates exactly C distinct candidates using precomputed next-distinct offsets, and chooses the HRW winner (optionally weighted). Lookup cost is O(log|R| + C). Under fixed-topology liveness changes, fixed-candidate filtering remaps only keys whose original winner is down, yielding zero excess churn. In a benchmark with N=5000, V=256 (|R|=1.28M), K=50M and C=8, LRH reduces Max/Avg load from 1.2785 to 1.0947 and achieves 60.05 Mkeys/s, about 6.8x faster than multi-probe consistent hashing with 8 probes (8.80 Mkeys/s) while approaching its balance (Max/Avg 1.0697). A microbenchmark indicates multi-probe assignment is dominated by repeated ring searches and memory traffic rather than probe-generation arithmetic.