TeaMPI -- Replication-based Resilience without the (Performance) Pain

TL;DR

This paper introduces a novel replication-based resilience method for large-scale HPC simulations that reduces time-to-solution without major code changes, using a weakly consistent data model and shared task outcomes.

Contribution

It presents a new algorithmic approach where replication reduces computation time and maintains resilience without significant code modifications in the ExaHyPE engine.

Findings

Replication can be made affordable for large-scale simulations.

Shared outcomes among replicas improve efficiency.

The approach maintains resilience with minimal code changes.

Abstract

In an era where we can not afford to checkpoint frequently, replication is a generic way forward to construct numerical simulations that can continue to run even if hardware parts fail. Yet, replication often is not employed on larger scales, as na\"ively mirroring a computation once effectively halves the machine size, and as keeping replicated simulations consistent with each other is not trivial. We demonstrate for the ExaHyPE engine -- a task-based solver for hyperbolic equation systems -- that it is possible to realise resiliency without major code changes on the user side, while we introduce a novel algorithmic idea where replication reduces the time-to-solution. The redundant CPU cycles are not burned "for nothing". Our work employs a weakly consistent data model where replicas run independently yet inform each other through heartbeat messages whether they are still up and running. Our key performance idea is to let the tasks of the replicated simulations share some of their outcomes, while we shuffle the actual task execution order per replica. This way, replicated ranks can skip some local computations and automatically start to synchronise with each other. Our experiments with a production-level seismic wave-equation solver provide evidence that this novel concept has the potential to make replication affordable for large-scale simulations in high-performance computing.