# Checkpoint/restart approaches for a thread-based MPI runtime

## TL;DR

This paper presents checkpoint/restart strategies for a thread-based MPI runtime, focusing on maintaining high network performance and efficient data replication to improve fault tolerance at scale.

## Contribution

It introduces practical checkpointing methods tailored for thread-based MPI, emphasizing minimal performance impact and enhanced data replication through user-level scheduling.

## Key findings

- High-speed network performance preserved during checkpointing
- Over-subscription of checkpoint data improves fault tolerance
- Measured overheads and trade-offs on MPI benchmarks

## Abstract

Fault-tolerance has always been an important topic when it comes to running massively parallel programs at scale. Statistically, hardware and software failures are expected to occur more often on systems gathering millions of computing units. Moreover, the larger jobs are, the more computing hours would be wasted by a crash. In this paper, we describe the work done in our MPI runtime to enable both transparent and application-level checkpointing mechanisms. Unlike the MPI 4.0 User-Level Failure Mitigation (ULFM) interface, our work targets solely Checkpoint/Restart and ignores other features such as resiliency. We show how existing checkpointing methods can be practically applied to a thread-based MPI implementation given sufficient runtime collaboration. The two main contributions are the preservation of high-speed network performance during transparent C/R and the over-subscription of checkpoint data replication thanks to a dedicated user-level scheduler support. These techniques are measured on MPI benchmarks such as IMB, Lulesh and Heatdis, and associated overhead and trade-offs are discussed.