A Big Data Approach for Sequences Indexing on the Cloud via Burrows Wheeler Transform

TL;DR

This paper introduces a scalable algorithm for computing the Burrows Wheeler transform using Big Data technologies like Spark and Hadoop, enabling efficient indexing of large sequence datasets in cloud environments for applications like Precision Medicine.

Contribution

It presents the first distributed algorithm for Burrows Wheeler transform computation that leverages cloud resources, improving scalability and efficiency.

Findings

Successfully implemented on Spark and Hadoop

Achieved significant speedup over non-distributed methods

Facilitates large-scale sequence data analysis in cloud environments

Abstract

Indexing sequence data is important in the context of Precision Medicine, where large amounts of ``omics'' data have to be daily collected and analyzed in order to categorize patients and identify the most effective therapies. Here we propose an algorithm for the computation of Burrows Wheeler transform relying on Big Data technologies, i.e., Apache Spark and Hadoop. Our approach is the first that distributes the index computation and not only the input dataset, allowing to fully benefit of the available cloud resources.