DobLIX: A Dual-Objective Learned Index for Log-Structured Merge Trees

TL;DR

DobLIX is a dual-objective learned index for LSM-tree systems that optimizes both lookup efficiency and data access costs, significantly enhancing read performance and adaptability in real-world storage engines.

Contribution

It introduces DobLIX, a novel dual-objective learned index with reinforcement learning for dynamic tuning, addressing both lookup and data access optimization in LSM-trees.

Findings

Reduces indexing overhead in RocksDB.

Improves throughput by 1.19 to 2.21 times.

Balances read and write performance effectively.

Abstract

In this paper, we introduce DobLIX, a dual-objective learned index specifically designed for Log-Structured Merge(LSM) tree-based key-value stores. Although traditional learned indexes focus exclusively on optimizing index lookups, they often overlook the impact of data access from storage, resulting in performance bottlenecks. DobLIX addresses this by incorporating a second objective, data access optimization, into the learned index training process. This dual-objective approach ensures that both index lookup efficiency and data access costs are minimized, leading to significant improvements in read performance while maintaining write efficiency in real-world LSM-tree systems. Additionally, DobLIX features a reinforcement learning agent that dynamically tunes the system parameters, allowing it to adapt to varying workloads in real-time. Experimental results using real-world datasets demonstrate that DobLIX reduces indexing overhead and improves throughput by 1.19 to 2.21 times compared to state-of-the-art methods within RocksDB, a widely used LSM-tree-based storage engine.