SALI: A Scalable Adaptive Learned Index Framework based on Probability Models

TL;DR

SALI is a scalable adaptive learned index framework that improves concurrency, efficiency, and robustness for multi-core data storage by employing node-evolving and statistical maintenance strategies.

Contribution

The paper introduces SALI, a novel framework that enhances scalability and performance of learned indexes through adaptive strategies and statistical information management.

Findings

SALI increases insertion throughput by 2.04x with 64 threads.

SALI achieves lookup throughput comparable to LIPP+.

The framework effectively adapts to workload skews and improves scalability.

Abstract

The growth in data storage capacity and the increasing demands for high performance have created several challenges for concurrent indexing structures. One promising solution is learned indexes, which use a learning-based approach to fit the distribution of stored data and predictively locate target keys, significantly improving lookup performance. Despite their advantages, prevailing learned indexes exhibit constraints and encounter issues of scalability on multi-core data storage. This paper introduces SALI, the Scalable Adaptive Learned Index framework, which incorporates two strategies aimed at achieving high scalability, improving efficiency, and enhancing the robustness of the learned index. Firstly, a set of node-evolving strategies is defined to enable the learned index to adapt to various workload skews and enhance its concurrency performance in such scenarios. Secondly, a lightweight strategy is proposed to maintain statistical information within the learned index, with the goal of further improving the scalability of the index. Furthermore, to validate their effectiveness, SALI applied the two strategies mentioned above to the learned index structure that utilizes fine-grained write locks, known as LIPP. The experimental results have demonstrated that SALI significantly enhances the insertion throughput with 64 threads by an average of 2.04x compared to the second-best learned index. Furthermore, SALI accomplishes a lookup throughput similar to that of LIPP+.