Size-aware Sharding For Improving Tail Latencies in In-memory Key-value Stores

TL;DR

This paper proposes size-aware sharding in in-memory key-value stores to significantly reduce tail latencies by preventing head-of-line blocking, with implementation in Minos showing substantial performance improvements.

Contribution

It introduces size-aware sharding for in-memory KV stores, optimizing tail latency and throughput by segregating small and large item requests and implementing hardware dispatch.

Findings

Minos achieves up to 100x lower 99th percentile latency.

Minos attains up to 7.4x higher throughput compared to state-of-the-art.

Size-aware sharding effectively reduces tail latencies in in-memory key-value stores.

Abstract

This paper introduces the concept of size-aware sharding to improve tail latencies for in-memory key-value stores, and describes its implementation in the Minos key-value store. Tail latencies are crucial in distributed applications with high fan-out ratios, because overall response time is determined by the slowest response. Size-aware sharding distributes requests for keys to cores according to the size of the item associated with the key. In particular, requests for small and large items are sent to disjoint subsets of cores. Size-aware sharding improves tail latencies by avoiding head-of-line blocking, in which a request for a small item gets queued behind a request for a large item. Alternative size-unaware approaches to sharding, such as keyhash-based sharding, request dispatching and stealing do not avoid head-of-line blocking, and therefore exhibit worse tail latencies. The challenge in implementing size-aware sharding is to maintain high throughput by avoiding the cost of software dispatching and by achieving load balancing between different cores. Minos uses hardware dispatch for all requests for small items, which form the very large majority of all requests. It achieves load balancing by adapting the number of cores handling requests for small and large items to their relative presence in the workload. We compare Minos to three state-of-the-art designs of in-memory KV stores. Compared to its closest competitor, Minos achieves a 99th percentile latency that is up to two orders of magnitude lower. Put differently, for a given value for the 99th percentile latency equal to 10 times the mean service time, Minos achieves a throughput that is up to 7.4 times higher.