MATCH: An MPI Fault Tolerance Benchmark Suite

TL;DR

MATCH is a comprehensive benchmark suite designed to evaluate and compare various MPI fault tolerance techniques, providing insights into their performance and scalability for high-performance computing applications.

Contribution

The paper introduces MATCH, the first structured benchmark suite for systematically studying and comparing MPI fault tolerance methods across different scenarios.

Findings

Reinit recovery generally outperforms ULFM recovery.

Reinit recovery is scalable and problem-size independent.

Combining Reinit recovery with FTI checkpointing yields high efficiency.

Abstract

MPI has been ubiquitously deployed in flagship HPC systems aiming to accelerate distributed scientific applications running on tens of hundreds of processes and compute nodes. Maintaining the correctness and integrity of MPI application execution is critical, especially for safety-critical scientific applications. Therefore, a collection of effective MPI fault tolerance techniques have been proposed to enable MPI application execution to efficiently resume from system failures. However, there is no structured way to study and compare different MPI fault tolerance designs, so to guide the selection and development of efficient MPI fault tolerance techniques for distinct scenarios. To solve this problem, we design, develop, and evaluate a benchmark suite called MATCH to characterize, research, and comprehensively compare different combinations and configurations of MPI fault tolerance designs. Our investigation derives useful findings: (1) Reinit recovery in general performs better than ULFM recovery; (2) Reinit recovery is independent of the scaling size and the input problem size, whereas ULFM recovery is not; (3) Using Reinit recovery with FTI checkpointing is a highly efficient fault tolerance design. MATCH code is available at https://github.com/kakulo/MPI- FT- Bench.