OSM-tree: A Sortedness-Aware Index

TL;DR

The paper introduces the OSM-tree, a new index structure optimized for data ingestion efficiency by leveraging partial sortedness, achieving significant performance improvements over existing indexes while maintaining competitive query performance.

Contribution

The paper proposes the OSM-tree, a novel index that combines multiple techniques to optimize ingestion in near-sorted data, outperforming state-of-the-art indexes in both ingestion and query performance.

Findings

OSM-tree outperforms existing indexes by up to 8.8x in ingestion performance.

OSM-tree maintains competitive query performance, with benefits up to 5x for mixed workloads.

The design effectively leverages partial data sortedness to optimize index construction.

Abstract

Indexes facilitate efficient querying when the selection predicate is on an indexed key. As a result, when loading data, if we anticipate future selective (point or range) queries, we typically maintain an index that is gradually populated as new data is ingested. In that respect, indexing can be perceived as the process of adding structure to an incoming, otherwise unsorted, data collection. The process of adding structure comes at a cost, as instead of simply appending incoming data, every new entry is inserted into the index. If the data ingestion order matches the indexed attribute order, the ingestion cost is entirely redundant and can be avoided (e.g., via bulk loading in a B+-tree). However, state-of-the-art index designs do not benefit when data is ingested in an order that is close to being sorted but not fully sorted. In this paper, we study how indexes can benefit from partial data sortedness or near-sortedness, and we propose an ensemble of techniques that combine bulk loading, index appends, variable node fill/split factor, and buffering, to optimize the ingestion cost of a tree index in presence of partial data sortedness. We further augment the proposed design with necessary metadata structures to ensure competitive read performance. We apply the proposed design paradigm on a state-of-the-art B+-tree, and we propose the Ordered Sort-Merge tree (OSM-tree). OSM-tree outperforms the state of the art by up to 8.8x in ingestion performance in the presence of sortedness, while falling back to a B+-tree's ingestion performance when data is scrambled. OSM-tree offers competitive query performance, leading to performance benefits between 28% and 5x for mixed read/write workloads.