Compression of next-generation sequencing reads aided by highly efficient de novo assembly

TL;DR

This paper introduces Quip, a lossless compression tool for sequencing data that combines reference-based and novel de novo assembly methods, significantly reducing dataset sizes without information loss.

Contribution

The paper presents the first assembly-based compressor for sequencing data using a new de novo assembly algorithm with a probabilistic data structure.

Findings

Compresses data to less than 15% of original size

Uses a novel assembly algorithm with low memory requirements

Achieves lossless compression of FASTQ and SAM/BAM files

Abstract

We present Quip, a lossless compression algorithm for next-generation sequencing data in the FASTQ and SAM/BAM formats. In addition to implementing reference-based compression, we have developed, to our knowledge, the first assembly-based compressor, using a novel de novo assembly algorithm. A probabilistic data structure is used to dramatically reduce the memory required by traditional de Bruijn graph assemblers, allowing millions of reads to be assembled very efficiently. Read sequences are then stored as positions within the assembled contigs. This is combined with statistical compression of read identifiers, quality scores, alignment information, and sequences, effectively collapsing very large datasets to less than 15% of their original size with no loss of information. Availability: Quip is freely available under the BSD license from http://cs.washington.edu/homes/dcjones/quip.