Implicit Actions and Non-blocking Failure Recovery with MPI

TL;DR

This paper advances MPI fault tolerance by enabling asynchronous, non-blocking failure recovery, allowing multiple components to recover simultaneously and overlap recovery with ongoing computations, thus improving resilience and efficiency.

Contribution

It introduces mechanisms for consistent fault reporting, scoped recovery, and overlapping recovery activities, enhancing MPI ULFM's ability to support asynchronous, non-blocking failure recovery.

Findings

Enables applications to assess computational success without performance penalties.

Allows independent recovery of application components and groups.

Overlaps system and application recovery activities for efficiency.

Abstract

Scientific applications have long embraced the MPI as the environment of choice to execute on large distributed systems. The User-Level Failure Mitigation (ULFM) specification extends the MPI standard to address resilience and enable MPI applications to restore their communication capability after a failure. This works builds upon the wide body of experience gained in the field to eliminate a gap between current practice and the ideal, more asynchronous, recovery model in which the fault tolerance activities of multiple components can be carried out simultaneously and overlap. This work proposes to: (1) provide the required consistency in fault reporting to applications (i.e., enable an application to assess the success of a computational phase without incurring an unacceptable performance hit); (2) bring forward the building blocks that permit the effective scoping of fault recovery in an application, so that independent components in an application can recover without interfering with each other, and separate groups of processes in the application can recover independently or in unison; and (3) overlap recovery activities necessary to restore the consistency of the system (e.g., eviction of faulty processes from the communication group) with application recovery activities (e.g., dataset restoration from checkpoints).