Improving scalability and reliability of MPI-agnostic transparent checkpointing for production workloads at NERSC

TL;DR

This paper discusses enhancing the MANA checkpointing tool to improve its scalability and reliability for diverse production workloads at NERSC, enabling fault tolerance for large-scale HPC applications.

Contribution

The work adapts and improves MANA for NERSC's HPC environment, addressing bugs, adding features, and evaluating performance for production workloads.

Findings

MANA was successfully adapted for NERSC's diverse HPC workloads.

Performance overhead of checkpointing was evaluated at scale.

Lessons learned inform future C/R tool development for supercomputing centers.

Abstract

Checkpoint/restart (C/R) provides fault-tolerant computing capability, enables long running applications, and provides scheduling flexibility for computing centers to support diverse workloads with different priority. It is therefore vital to get transparent C/R capability working at NERSC. MANA, by Garg et. al., is a transparent checkpointing tool that has been selected due to its MPI-agnostic and network-agnostic approach. However, originally written as a proof-of-concept code, MANA was not ready to use with NERSC's diverse production workloads, which are dominated by MPI and hybrid MPI+OpenMP applications. In this talk, we present ongoing work at NERSC to enable MANA for NERSC's production workloads, including fixing bugs that were exposed by the top applications at NERSC, adding new features to address system changes, evaluating C/R overhead at scale, etc. The lessons learned from making MANA production-ready for HPC applications will be useful for C/R tool developers, supercomputing centers and HPC end-users alike.