HIRE: A Hybrid Learned Index for Robust and Efficient Performance under Mixed Workloads

TL;DR

HIRE is a hybrid index structure that combines traditional and learned indexing techniques to improve efficiency, robustness, and stability across diverse workloads, outperforming existing methods significantly.

Contribution

The paper introduces HIRE, a novel hybrid index that integrates model-based predictions with traditional structures for consistent, high-performance data retrieval and updates.

Findings

HIRE achieves up to 41.7× higher throughput under mixed workloads.

HIRE reduces tail latency by up to 98%.

HIRE outperforms state-of-the-art learned and traditional indexes in multiple datasets.

Abstract

Indexes are critical for efficient data retrieval and updates in modern databases. Recent advances in machine learning have led to the development of learned indexes, which model the cumulative distribution function of data to predict search positions and accelerate query processing. While learned indexes substantially outperform traditional structures for point lookups, they often suffer from high tail latency, suboptimal range query performance, and inconsistent effectiveness across diverse workloads. To address these challenges, this paper proposes HIRE, a hybrid in-memory index structure designed to deliver efficient performance consistently. HIRE combines the structural and performance robustness of traditional indexes with the predictive power of model-based prediction to reduce search overhead while maintaining worst-case stability. Specifically, it employs (1) hybrid leaf nodes adaptive to varying data distributions and workloads, (2) model-accelerated internal nodes augmented by log-based updates for efficient updates, (3) a nonblocking, cost-driven recalibration mechanism for dynamic data, and (4) an inter-level optimized bulk-loading algorithm accounting for leaf and internal-node errors. Experimental results on multiple real-world datasets demonstrate that HIRE outperforms both state-of-the-art learned indexes and traditional structures in range-query throughput, tail latency, and overall stability. Compared to state-of-the-art learned indexes and traditional indexes, HIRE achieves up to 41.7$\times$ higher throughput under mixed workloads, reduces tail latency by up to 98% across varying scenarios.