Quasi-Succinct Indices

TL;DR

This paper introduces a quasi-succinct index architecture for compressed inverted indices, offering theoretical elegance and practical performance improvements for various query types by representing monotone sequences efficiently.

Contribution

It proposes a novel quasi-succinct index structure that simplifies implementation and enhances query performance over traditional gap compression methods.

Findings

Expected constant-time query operations

Significant practical performance improvements

Theoretically elegant and simple index architecture

Abstract

Compressed inverted indices in use today are based on the idea of gap compression: documents pointers are stored in increasing order, and the gaps between successive document pointers are stored using suitable codes which represent smaller gaps using less bits. Additional data such as counts and positions is stored using similar techniques. A large body of research has been built in the last 30 years around gap compression, including theoretical modeling of the gap distribution, specialized instantaneous codes suitable for gap encoding, and ad hoc document reorderings which increase the efficiency of instantaneous codes. This paper proposes to represent an index using a different architecture based on quasi-succinct representation of monotone sequences. We show that, besides being theoretically elegant and simple, the new index provides expected constant-time operations and, in practice, significant performance improvements on conjunctive, phrasal and proximity queries.