Cyclone: High Availability for Persistent Key Value Stores

TL;DR

Cyclone is a high-performance, replicated log system for key-value stores that leverages non-volatile memory and user-space networking to significantly improve throughput and availability.

Contribution

Cyclone introduces a novel replicated log plugin that reduces bottlenecks by using non-volatile memory and user-space networking, enhancing performance of persistent key-value stores.

Findings

Replicates millions of small updates per second over 10GbE

Improves RocksDB performance and availability by an order of magnitude

Reduces CPU and storage bottlenecks in distributed key-value stores

Abstract

Persistent key value stores are an important component of many distributed data serving solutions with innovations targeted at taking advantage of growing flash speeds. Unfortunately their performance is hampered by the need to maintain and replicate a write ahead log to guarantee availability in the face of machine and storage failures. Cyclone is a replicated log plug-in for key value stores that systematically addresses various sources of this bottleneck. It uses a small amount of non-volatile memory directly addressable by the CPU - such as in the form of NVDIMMs or Intel 3DXPoint - to remove block oriented IO devices such as SSDs from the critical path for appending to the log. This enables it to address network overheads using an implementation of the RAFT consensus protocol that is designed around a userspace network stack to relieve the CPU of the burden of data copies. Finally, it provides a way to efficiently map the commutativity in key-value store APIs to the parallelism available in commodity NICs. Cyclone is able to replicate millions of small updates per second using only commodity 10 gigabit ethernet adapters. As a practical application, we use it to improve the performance (and availability) of RocksDB, a popular persistent key value store by an order of magnitude when compared to its own write ahead log without replication.