CompassDB: Pioneering High-Performance Key-Value Store with Perfect Hash

TL;DR

CompassDB introduces a high-performance key-value store using a perfect hash-based index to significantly reduce read/write amplification, improving throughput and latency over existing LSM-tree based engines especially on SSDs.

Contribution

It presents a novel Two-tier Perfect Hash Table design that drastically decreases index size and lookup time, outperforming traditional LSM-tree based storage engines.

Findings

Increases throughput by 2.5x to 4x over RocksDB.

Reduces latency by 50% to 85% compared to RocksDB.

Achieves 5x to 17x throughput improvement over PebblesDB.

Abstract

Modern mainstream persistent key-value storage engines utilize Log-Structured Merge tree (LSM-tree) based designs, optimizing read/write performance by leveraging sequential disk I/O. However, the advent of SSDs, with their significant improvements in bandwidth and IOPS, shifts the bottleneck from I/O to CPU. The high compaction cost and large read/write amplification associated with LSM trees have become critical bottlenecks. In this paper, we introduce CompassDB, which utilizes a Two-tier Perfect Hash Table (TPH) design to significantly decrease read/write amplification and compaction costs. CompassDB utilizes a perfect hash algorithm for its in-memory index, resulting in an average index cost of about 6 bytes per key-value pair. This compact index reduces the lookup time complexity from $O(log N)$ to $O(1)$ and decreases the overall cost. Consequently, it allows for the storage of more key-value pairs for reads or provides additional memory for the memtable for writes. This results in substantial improvements in both throughput and latency. Our evaluation using the YCSB benchmark tool shows that CompassDB increases throughput by 2.5x to 4x compared to RocksDB, and by 5x to 17x compared to PebblesDB across six typical workloads. Additionally, CompassDB significantly reduces average and 99th percentile read/write latency, achieving a 50% to 85% reduction in comparison to RocksDB.