E2FM: an encrypted and compressed full-text index for collections of genomic sequences

TL;DR

E2FM is a compact, encrypted full-text index for genomic sequences that significantly reduces storage needs and enables fast pattern searches, addressing data growth and privacy concerns in genomics.

Contribution

The paper introduces E2FM, a novel encrypted and compressed full-text index optimized for nucleotide sequences, achieving high compression and fast search capabilities.

Findings

Indexes occupy up to 1/20 of original storage for similar sequences

Search times range from milliseconds to a few hundred milliseconds

Achieves about 95% storage savings on highly similar sequences

Abstract

Next Generation Sequencing (NGS) platforms and, more generally, high-throughput technologies are giving rise to an exponential growth in the size of nucleotide sequence databases. Moreover, many emerging applications of nucleotide datasets -- as those related to personalized medicine -- require the compliance with regulations about the storage and processing of sensitive data. We have designed and carefully engineered E2FM-index, a new full-text index in minute space which was optimized for compressing and encrypting nucleotide sequence collections in FASTA format and for performing fast pattern-search queries. E2FM-index allows to build self-indexes which occupy till to 1/20 of the storage required by the input FASTA file, thus permitting to save about 95% of storage when indexing collections of highly similar sequences; moreover, it can exactly search the built indexes for patterns in times ranging from few milliseconds to a few hundreds milliseconds, depending on pattern length. Supplementary material and supporting datasets are available through Bioinformatics Online and https://figshare.com/s/6246ee9c1bd730a8bf6e.