COBS: a Compact Bit-Sliced Signature Index

TL;DR

COBS is a compact, scalable index combining inverted index and Bloom filters, optimized for fast approximate pattern matching in DNA and text data, outperforming existing methods in speed and memory efficiency.

Contribution

The paper introduces COBS, a novel compact bit-sliced signature index that scales to large datasets without requiring full RAM, improving construction and query performance over existing indexes.

Findings

COBS outperforms seven other index packages in speed and memory efficiency.

It scales to larger datasets without needing complete index in RAM.

It effectively reduces false positives exponentially with query length.

Abstract

We present COBS, a COmpact Bit-sliced Signature index, which is a cross-over between an inverted index and Bloom filters. Our target application is to index $k$-mers of DNA samples or $q$-grams from text documents and process approximate pattern matching queries on the corpus with a user-chosen coverage threshold. Query results may contain a number of false positives which decreases exponentially with the query length. We compare COBS to seven other index software packages on 100000 microbial DNA samples. COBS' compact but simple data structure outperforms the other indexes in construction time and query performance with Mantis by Pandey et al. in second place. However, unlike Mantis and other previous work, COBS does not need the complete index in RAM and is thus designed to scale to larger document sets.