To Repair or Not to Repair: Assessing Fault Resilience in MPI Stencil Applications

TL;DR

This paper explores fault resilience in MPI stencil applications, proposing a method that continues execution after faults with reduced overhead, sacrificing some accuracy for faster recovery.

Contribution

It introduces a fault resilience approach using User Level Fault Mitigation in MPI, enabling continued execution post-fault without checkpointing.

Findings

Fault resilience yields faster recovery times.

Results remain scientifically meaningful despite some inaccuracies.

Approach reduces overhead compared to checkpoint-restart.

Abstract

With the increasing size of HPC computations, faults are becoming more and more relevant in the HPC field. The MPI standard does not define the application behaviour after a fault, leaving the burden of fault management to the user, who usually resorts to checkpoint and restart mechanisms. This trend is especially true in stencil applications, as their regular pattern simplifies the selection of checkpoint locations. However, checkpoint and restart mechanisms introduce non-negligible overhead, disk load, and scalability concerns. In this paper, we show an alternative through fault resilience, enabled by the features provided by the User Level Fault Mitigation extension and shipped within the Legio fault resilience framework. Through fault resilience, we continue executing only the non-failed processes, thus sacrificing result accuracy for faster fault recovery. Our experiments on a specimen stencil application show that, despite the fault impact visible in the result, we produced meaningful values usable for scientific research, proving the possibilities of a fault resilience approach in a stencil scenario.