Endure: A Robust Tuning Paradigm for LSM Trees Under Workload Uncertainty

TL;DR

Endure introduces a robust tuning paradigm for LSM trees that optimizes worst-case throughput under workload uncertainty, outperforming traditional tuning strategies especially in noisy, unpredictable cloud environments.

Contribution

The paper proposes a novel robust tuning framework for LSM trees that accounts for workload variability, improving performance stability and throughput in uncertain cloud settings.

Findings

Up to 5× throughput improvement under workload uncertainty.

Robust tuning maintains near-optimal performance when workload matches expectations.

Endure outperforms classical tuning strategies across diverse noisy workloads.

Abstract

Log-Structured Merge trees (LSM trees) are increasingly used as the storage engines behind several data systems, frequently deployed in the cloud. Similar to other database architectures, LSM trees take into account information about the expected workload (e.g., reads vs. writes, point vs. range queries) to optimize their performance via tuning. Operating in shared infrastructure like the cloud, however, comes with a degree of workload uncertainty due to multi-tenancy and the fast-evolving nature of modern applications. Systems with static tuning discount the variability of such hybrid workloads and hence provide an inconsistent and overall suboptimal performance. To address this problem, we introduce Endure - a new paradigm for tuning LSM trees in the presence of workload uncertainty. Specifically, we focus on the impact of the choice of compaction policies, size-ratio, and memory allocation on the overall performance. Endure considers a robust formulation of the throughput maximization problem, and recommends a tuning that maximizes the worst-case throughput over a neighborhood of each expected workload. Additionally, an uncertainty tuning parameter controls the size of this neighborhood, thereby allowing the output tunings to be conservative or optimistic. Through both model-based and extensive experimental evaluation of Endure in the state-of-the-art LSM-based storage engine, RocksDB, we show that the robust tuning methodology consistently outperforms classical tun-ing strategies. We benchmark Endure using 15 workload templates that generate more than 10000 unique noisy workloads. The robust tunings output by Endure lead up to a 5$\times$ improvement in through-put in presence of uncertainty. On the flip side, when the observed workload exactly matches the expected one, Endure tunings have negligible performance loss.