Achieving High Throughput and Elasticity in a Larger-than-Memory Store

TL;DR

Shadowfax is a distributed key-value store that achieves high throughput and elastic scaling across cloud resources by efficiently managing data across different storage tiers and enabling rapid reconfiguration without costly data migrations.

Contribution

It introduces a novel distributed reconfiguration approach that avoids server-side key ownership checks, enabling fast load shifting and high throughput in a scalable key-value store.

Findings

Achieves 130 Mops/s per VM on commodity Azure VMs.

Retains 930 Mops/s across 12 machines, the highest reported for large-scale data ingestion.

Reconfigures in 17 seconds with minimal disruption.

Abstract

Millions of sensors, mobile applications and machines now generate billions of events. Specialized many-core key-value stores (KVSs) can ingest and index these events at high rates (over 100 Mops/s on one machine) if events are generated on the same machine; however, to be practical and cost-effective they must ingest events over the network and scale across cloud resources elastically. We present Shadowfax, a new distributed KVS based on FASTER, that transparently spans DRAM, SSDs, and cloud blob storage while serving 130 Mops/s/VM over commodity Azure VMs using conventional Linux TCP. Beyond high single-VM performance, Shadowfax uses a unique approach to distributed reconfiguration that avoids any server-side key ownership checks or cross-core coordination both during normal operation and migration. Hence, Shadowfax can shift load in 17 s to improve system throughput by 10 Mops/s with little disruption. Compared to the state-of-the-art, it has 8x better throughput (than Seastar+memcached) and avoids costly I/O to move cold data during migration. On 12 machines, Shadowfax retains its high throughput to perform 930 Mops/s, which, to the best of our knowledge, is the highest reported throughput for a distributed KVS used for large-scale data ingestion and indexing.