Hybrid Indexes for Repetitive Datasets

TL;DR

This paper presents a technique that leverages LZ77 compression to create smaller, faster indexes for highly repetitive genomic datasets, improving search efficiency.

Contribution

It introduces a novel hybrid indexing method combining LZ77 preprocessing with conventional indexes for repetitive texts.

Findings

Reduces index size significantly on repetitive datasets.

Decreases query times compared to traditional indexes.

Effective for DNA sequence databases.

Abstract

Advances in DNA sequencing mean databases of thousands of human genomes will soon be commonplace. In this paper we introduce a simple technique for reducing the size of conventional indexes on such highly repetitive texts. Given upper bounds on pattern lengths and edit distances, we preprocess the text with LZ77 to obtain a filtered text, for which we store a conventional index. Later, given a query, we find all matches in the filtered text, then use their positions and the structure of the LZ77 parse to find all matches in the original text. Our experiments show this also significantly reduces query times.