Compressed Text Indexes:From Theory to Practice!

TL;DR

This paper reviews the development of compressed full-text self-indexes, introduces a standardized platform for their implementation and testing, and demonstrates their practical effectiveness through extensive experiments.

Contribution

It provides a comprehensive overview of existing implementations, introduces the Pizza&Chili platform with standardized APIs, and evaluates their performance in practical scenarios.

Findings

Compressed indexes significantly reduce storage compared to traditional methods.

The Pizza&Chili platform enables easier deployment and comparison of compressed indexes.

Experimental results confirm the practicality and efficiency of compressed full-text self-indexes.

Abstract

A compressed full-text self-index represents a text in a compressed form and still answers queries efficiently. This technology represents a breakthrough over the text indexing techniques of the previous decade, whose indexes required several times the size of the text. Although it is relatively new, this technology has matured up to a point where theoretical research is giving way to practical developments. Nonetheless this requires significant programming skills, a deep engineering effort, and a strong algorithmic background to dig into the research results. To date only isolated implementations and focused comparisons of compressed indexes have been reported, and they missed a common API, which prevented their re-use or deployment within other applications. The goal of this paper is to fill this gap. First, we present the existing implementations of compressed indexes from a practitioner's point of view. Second, we introduce the Pizza&Chili site, which offers tuned implementations and a standardized API for the most successful compressed full-text self-indexes, together with effective testbeds and scripts for their automatic validation and test. Third, we show the results of our extensive experiments on these codes with the aim of demonstrating the practical relevance of this novel and exciting technology.