Querying in Constant Expected Time with Learned Indexes

TL;DR

This paper proves that learned indexes can achieve constant expected query time with linear space under weaker assumptions, advancing the theoretical understanding and practical estimation of their performance.

Contribution

It establishes the first constant expected time bound for learned indexes with linear space under weaker probabilistic assumptions, and introduces a new measure of data complexity.

Findings

Achieves $O(1)$ expected query time with linear space.

Introduces a new statistical complexity measure for datasets.

Generalizes previous results with weaker assumptions.

Abstract

Learned indexes leverage machine learning models to accelerate query answering in databases, showing impressive practical performance. However, theoretical understanding of these methods remains incomplete. Existing research suggests that learned indexes have superior asymptotic complexity compared to their non-learned counterparts, but these findings have been established under restrictive probabilistic assumptions. Specifically, for a sorted array with $n$ elements, it has been shown that learned indexes can find a key in $O(\log(\log n))$ expected time using at most linear space, compared with $O(\log n)$ for non-learned methods. In this work, we prove $O(1)$ expected time can be achieved with at most linear space, thereby establishing the tightest upper bound so far for the time complexity of an asymptotically optimal learned index. Notably, we use weaker probabilistic assumptions than prior research, meaning our work generalizes previous results. Furthermore, we introduce a new measure of statistical complexity for data. This metric exhibits an information-theoretical interpretation and can be estimated in practice. This characterization provides further theoretical understanding of learned indexes, by helping to explain why some datasets seem to be particularly challenging for these methods.