Efficient Direct-Access Ranked Retrieval

TL;DR

This paper introduces efficient algorithms for direct-access ranked retrieval in high-dimensional datasets, balancing query speed and space complexity, and proposes a relaxed retrieval variant with practical scalability.

Contribution

It formalizes the DAR problem, proposes geometric and sampling-based algorithms, and introduces CSR and SRR for scalable, approximate ranked retrieval in large, high-dimensional datasets.

Findings

Logarithmic query time with geometric arrangements

Linear space algorithms based on ε-sampling

Scalable performance on datasets with millions of tuples and hundreds of dimensions

Abstract

We study the problem of Direct-Access Ranked Retrieval (DAR) for interactive data tooling, where evolving data exploration practices, combined with large-scale and high-dimensional datasets, create new challenges. DAR concerns the problem of enabling efficient access to arbitrary rank positions according to a ranking function, without enumerating all preceding tuples. To address this need, we formalize the DAR problem and propose a theoretically efficient algorithm based on geometric arrangements, achieving logarithmic query time. However, this method suffers from exponential space complexity in high dimensions. Therefore, we develop a second class of algorithms based on $\varepsilon$-sampling, which consume a linear space. Since exactly locating the tuple at a specific rank is challenging due to its connection to the range counting problem, we introduce a relaxed variant called Conformal Set Ranked Retrieval (CSR), which returns a small subset guaranteed to contain the target tuple. To solve the CSR problem efficiently, we define an intermediate problem, Stripe Range Retrieval (SRR), and design a hierarchical sampling data structure tailored for narrow-range queries. Our method achieves practical scalability in both data size and dimensionality. We prove near-optimal bounds on the efficiency of our algorithms and validate their performance through extensive experiments on real and synthetic datasets, demonstrating scalability to millions of tuples and hundreds of dimensions.