DKVF: A Framework for Rapid Prototyping and Evaluating Distributed Key-value Stores

TL;DR

DKVF is a flexible framework that allows rapid prototyping and benchmarking of distributed key-value store protocols, facilitating comparison and evaluation of their trade-offs in consistency, availability, and performance.

Contribution

The paper introduces DKVF, a framework that simplifies and accelerates the development and evaluation of new key-value store protocols using existing benchmarks like YCSB.

Findings

Performance of protocols similar to original implementations

Comparison results align with existing literature

Implementation time reduced to 1-2 days per protocol

Abstract

We present our framework DKVF that enables one to quickly prototype and evaluate new protocols for key-value stores and compare them with existing protocols based on selected benchmarks. Due to limitations of CAP theorem, new protocols must be developed that achieve the desired trade-off between consistency and availability for the given application at hand. Hence, both academic and industrial communities focus on developing new protocols that identify a different (and hopefully better in one or more aspect) point on this trade-off curve. While these protocols are often based on a simple intuition, evaluating them to ensure that they indeed provide increased availability, consistency, or performance is a tedious task. Our framework, DKVF, enables one to quickly prototype a new protocol as well as identify how it performs compared to existing protocols for pre-specified benchmarks. Our framework relies on YCSB (Yahoo! Cloud Servicing Benchmark) for benchmarking. We demonstrate DKVF by implementing four existing protocols --eventual consistency, COPS, GentleRain and CausalSpartan-- with it. We compare the performance of these protocols against different loading conditions. We find that the performance is similar to our implementation of these protocols from scratch. And, the comparison of these protocols is consistent with what has been reported in the literature. Moreover, implementation of these protocols was much more natural as we only needed to translate the pseudocode into Java (and add the necessary error handling). Hence, it was possible to achieve this in just 1-2 days per protocol. Finally, our framework is extensible. It is possible to replace individual components in the framework (e.g., the storage component).