High-Performance Sorting-Based k-mer Counting in Distributed Memory with Flexible Hybrid Parallelism

TL;DR

HySortK is a novel distributed memory k-mer counting tool that employs sorting-based algorithms and flexible hybrid parallelism, significantly improving speed and memory efficiency for large-scale DNA data analysis.

Contribution

This work introduces HySortK, a sorting-based distributed k-mer counter with a flexible parallelism model, outperforming existing hash-based methods in speed and memory usage.

Findings

Achieves 2-10x speedup over GPU baseline on multiple nodes.

Reduces peak memory usage by 30% compared to state-of-the-art CPU software.

Provides up to 1.8x speedup when integrated into genome assembly pipelines.

Abstract

In generating large quantities of DNA data, high-throughput sequencing technologies require advanced bioinformatics infrastructures for efficient data analysis. k-mer counting, the process of quantifying the frequency of fixed-length k DNA subsequences, is a fundamental step in various bioinformatics pipelines, including genome assembly and protein prediction. Due to the growing volume of data, the scaling of the counting process is critical. In the literature, distributed memory software uses hash tables, which exhibit poor cache friendliness and consume excessive memory. They often also lack support for flexible parallelism, which makes integration into existing bioinformatics pipelines difficult. In this work, we propose HySortK, a highly efficient sorting-based distributed memory k-mer counter. HySortK reduces the communication volume through a carefully designed communication scheme and domain-specific optimization strategies. Furthermore, we introduce an abstract task layer for flexible hybrid parallelism to address load imbalances in different scenarios. HySortK achieves a 2-10x speedup compared to the GPU baseline on 4 and 8 nodes. Compared to state-of-the-art CPU software, HySortK achieves up to 2x speedup while reducing peak memory usage by 30% on 16 nodes. Finally, we integrated HySortK into an existing genome assembly pipeline and achieved up to 1.8x speedup, proving its flexibility and practicality in real-world scenarios.