Checkpointing SPAdes for Metagenome Assembly: Transparency versus Performance in Production

TL;DR

This paper examines the challenges of checkpointing the SPAdes metagenome assembler using DMTCP, highlighting bugs, limitations, and the trade-off between transparency and performance in production environments.

Contribution

It identifies specific issues with DMTCP when applied to SPAdes and offers solutions, along with insights on balancing transparency and performance in checkpointing large, complex applications.

Findings

DMTCP had bugs and limitations with SPAdes' large memory and fragmented files.

Fixes were implemented to support SPAdes' resource demands.

An I/O bottleneck affects checkpointing performance, revealing a transparency-performance trade-off.

Abstract

The SPAdes assembler for metagenome assembly is a long-running application commonly used at the NERSC supercomputing site. However, NERSC, like many other sites, has a 48-hour limit on resource allocations. The solution is to chain together multiple resource allocations in a single run, using checkpoint-restart. This case study provides insights into the "pain points" in applying a well-known checkpointing package (DMTCP: Distributed MultiThreaded CheckPointing) to long-running production workloads of SPAdes. This work has exposed several bugs and limitations of DMTCP, which were fixed to support the large memory and fragmented intermediate files of SPAdes. But perhaps more interesting for other applications, this work reveals a tension between the transparency goals of DMTCP and performance concerns due to an I/O bottleneck during the checkpointing process when supporting large memory and many files. Suggestions are made for overcoming this I/O bottleneck, which provides important "lessons learned" for similar applications.