Efficient Chromosome Parallelization for Precision Medicine Genomic Workflows

TL;DR

This paper introduces adaptive, RAM-efficient parallelization techniques for large-scale genomic workflows, improving resource utilization and reducing memory errors in chromosome-level bioinformatics processing.

Contribution

It presents novel symbolic regression and scheduling methods for dynamic, memory-aware parallelization of genomic workflows, enhancing efficiency over static approaches.

Findings

Reduced memory overruns in genomic workflows

Faster execution times in real-world pipelines

Effective load balancing across threads

Abstract

Large-scale genomic workflows used in precision medicine can process datasets spanning tens to hundreds of gigabytes per sample, leading to high memory spikes, intensive disk I/O, and task failures due to out-of-memory errors. Simple static resource allocation methods struggle to handle the variability in per-chromosome RAM demands, resulting in poor resource utilization and long runtimes. In this work, we propose multiple mechanisms for adaptive, RAM-efficient parallelization of chromosome-level bioinformatics workflows. First, we develop a symbolic regression model that estimates per-chromosome memory consumption for a given task and introduces an interpolating bias to conservatively minimize over-allocation. Second, we present a dynamic scheduler that adaptively predicts RAM usage with a polynomial regression model, treating task packing as a Knapsack problem to optimally batch jobs based on predicted memory requirements. Additionally, we present a static scheduler that optimizes chromosome processing order to minimize peak memory while preserving throughput. Our proposed methods, evaluated on simulations and real-world genomic pipelines, provide new mechanisms to reduce memory overruns and balance load across threads. We thereby achieve faster end-to-end execution, showcasing the potential to optimize large-scale genomic workflows.