Toward a Better Understanding and Evaluation of Tree Structures on Flash SSDs

TL;DR

This paper highlights the complexities and pitfalls in benchmarking tree-based data structures on SSDs, emphasizing the need for careful evaluation to ensure accurate performance assessment and fair comparison.

Contribution

It identifies seven common benchmarking pitfalls in evaluating tree structures on SSDs and provides guidelines to improve measurement accuracy and reproducibility.

Findings

Seven benchmarking pitfalls identified in SSD-based tree evaluations.

Incorrect measurements can mislead deployment decisions.

Guidelines proposed for more reliable and fair performance comparisons.

Abstract

Solid-state drives (SSDs) are extensively used to deploy persistent data stores, as they provide low latency random access, high write throughput, high data density, and low cost. Tree-based data structures are widely used to build persistent data stores, and indeed they lie at the backbone of many of the data management systems used in production and research today. In this paper, we show that benchmarking a persistent tree-based data structure on an SSD is a complex process, which may easily incur subtle pitfalls that can lead to an inaccurate performance assessment. At a high-level, these pitfalls stem from the interaction of complex software running on complex hardware. On one hand, tree structures implement internal operations that have nontrivial effects on performance. On the other hand, SSDs employ firmware logic to deal with the idiosyncrasies of the underlying flash memory, which are well known to lead to complex performance dynamics. We identify seven benchmarking pitfalls using RocksDB and WiredTiger, two widespread implementations of an LSM-Tree and a B+Tree, respectively. We show that such pitfalls can lead to incorrect measurements of key performance indicators, hinder the reproducibility and the representativeness of the results, and lead to suboptimal deployments in production environments. We also provide guidelines on how to avoid these pitfalls to obtain more reliable performance measurements, and to perform more thorough and fair comparison among different design points.