Towards an Optimal Space-and-Query-Time Index for Top-k Document Retrieval

TL;DR

This paper introduces a space-efficient index for top-k document retrieval that balances space and query time, improving on previous methods for relevance-based search in large document collections.

Contribution

It proposes a novel index structure that reduces space usage while maintaining efficient query times for top-k document retrieval based on term-frequency.

Findings

Index size is close to compressed full text index plus additional terms.

Query time is optimized to handle large document sets efficiently.

Space reduction is achieved with minimal impact on retrieval speed.

Abstract

Let $\D = $$ \{d_1,d_2,...d_D\}$ be a given set of $D$ string documents of total length $n$, our task is to index $\D$, such that the $k$ most relevant documents for an online query pattern $P$ of length $p$ can be retrieved efficiently. We propose an index of size $|CSA|+n\log D(2+o(1))$ bits and $O(t_{s}(p)+k\log\log n+poly\log\log n)$ query time for the basic relevance metric \emph{term-frequency}, where $|CSA|$ is the size (in bits) of a compressed full text index of $\D$, with $O(t_s(p))$ time for searching a pattern of length $p$ . We further reduce the space to $|CSA|+n\log D(1+o(1))$ bits, however the query time will be $O(t_s(p)+k(\log \sigma \log\log n)^{1+\epsilon}+poly\log\log n)$, where $\sigma$ is the alphabet size and $\epsilon >0$ is any constant.