Prefix-free parsing for merging big BWTs

TL;DR

This paper introduces a method to efficiently merge big BWTs by applying prefix-free parsing to smaller, less similar datasets, significantly reducing memory usage during construction.

Contribution

It presents a novel approach to merge BWTs of large datasets by leveraging prefix-free parsing on smaller datasets, improving memory efficiency.

Findings

Memory usage is drastically reduced when merging BWTs of small datasets.

The method is effective for datasets with low similarity, such as different species genomes.

The approach enables handling larger datasets than previously possible.

Abstract

When building Burrows-Wheeler Transforms (BWTs) of truly huge datasets, prefix-free parsing (PFP) can use an unreasonable amount of memory. In this paper we show how if a dataset can be broken down into small datasets that are not very similar to each other -- such as collections of many copies of genomes of each of several species, or collections of many copies of each of the human chromosomes -- then we can drastically reduce PFP's memory footprint by building the BWTs of the small datasets and then merging them into the BWT of the whole dataset.