Erasure Coding for Small Objects in In-Memory KV Storage

TL;DR

MemEC is an in-memory key-value store that uses erasure coding to efficiently store small objects, reducing redundancy and enabling high availability with fast recovery, suitable for read- and update-heavy workloads.

Contribution

MemEC introduces a novel erasure-coding approach tailored for small objects in in-memory KV stores, achieving lower redundancy and seamless mode transitions.

Findings

60% less storage redundancy compared to existing methods

High throughput and low latency in normal and degraded modes

Fast transition support between operational modes

Abstract

We present MemEC, an erasure-coding-based in-memory key-value (KV) store that achieves high availability and fast recovery while keeping low data redundancy across storage servers. MemEC is specifically designed for workloads dominated by small objects. By encoding objects in entirety, MemEC is shown to incur 60% less storage redundancy for small objects than existing replication- and erasure-coding-based approaches. It also supports graceful transitions between decentralized requests in normal mode (i.e., no failures) and coordinated requests in degraded mode (i.e., with failures). We evaluate our MemEC prototype via testbed experiments under read-heavy and update-heavy YCSB workloads. We show that MemEC achieves high throughput and low latency in both normal and degraded modes, and supports fast transitions between the two modes.