Tidehunter: Large-Value Storage With Minimal Data Relocation

TL;DR

Tidehunter is a storage engine that eliminates value compaction in LSM-trees, enabling high-throughput, low-latency key-value storage for large values with minimal data movement, suitable for blockchain and content-addressable storage.

Contribution

It introduces a novel approach that treats the WAL as permanent storage, avoiding value overwrites and reducing write amplification, with lock-free writes and epoch-based pruning.

Findings

Achieves 830K writes/sec on 1TB dataset, outperforming RocksDB and BlobDB.

Improves query performance by up to 15.6x.

Successfully integrated into Sui blockchain, maintaining stable throughput and latency.

Abstract

Log-Structured Merge-Trees (LSM-trees) dominate persistent key-value storage but suffer from high write amplification from 10x to 30x under random workloads due to repeated compaction. This overhead becomes prohibitive for large values with uniformly distributed keys, a workload common in content-addressable storage, deduplication systems, and blockchain validators. We present Tidehunter, a storage engine that eliminates value compaction by treating the Write-Ahead Log (WAL) as permanent storage rather than a temporary recovery buffer. Values are never overwritten; and small, lazily-flushed index tables map keys to WAL positions. Tidehunter introduces (a) lock-free writes that saturate NVMe drives through atomic allocation and parallel copying, (b) an optimistic index structure that exploits uniform key distributions for single-roundtrip lookups, and (c) epoch-based pruning that reclaims space without blocking writes. On a 1 TB dataset with 1 KB values, Tidehunter achieves 830K writes per second, that is 8.4x higher than RocksDB and 2.9x higher than BlobDB, while improving point queries by 1.7x and existence checks by 15.6x. We validate real-world impact by integrating Tidehunter into Sui, a high-throughput blockchain, where it maintains stable throughput and latency under loads that cause RocksDB-backed validators to collapse. Tidehunter is production-ready and is being deployed in production within Sui.