Shrink or Substitute: Handling Process Failures in HPC Systems using In-situ Recovery

TL;DR

This paper investigates fault tolerance strategies in HPC systems using ULFM MPI extensions, proposing in-memory checkpointing with process replacement or continuation to improve resilience against failures.

Contribution

It introduces two novel recovery strategies leveraging ULFM MPI and in-memory checkpointing for process failure handling in HPC applications.

Findings

Graceful degradation is effective when spares are unavailable.

In-memory checkpointing reduces recovery time.

Replacing failed processes maintains application progress.

Abstract

Efficient utilization of today's high-performance computing (HPC) systems with complex hardware and software components requires that the HPC applications are designed to tolerate process failures at runtime. With low mean time to failure (MTTF) of current and future HPC systems, long running simulations on these systems require capabilities for gracefully handling process failures by the applications themselves. In this paper, we explore the use of fault tolerance extensions to Message Passing Interface (MPI) called user-level failure mitigation (ULFM) for handling process failures without the need to discard the progress made by the application. We explore two alternative recovery strategies, which use ULFM along with application-driven in-memory checkpointing. In the first case, the application is recovered with only the surviving processes, and in the second case, spares are used to replace the failed processes, such that the original configuration of the application is restored. Our experimental results demonstrate that graceful degradation is a viable alternative for recovery in environments where spares may not be available.