Scalable Fault-Tolerant MapReduce

TL;DR

This paper introduces a scalable, fault-tolerant MapReduce approach that minimizes overhead by backing up network messages, enabling efficient recovery in large supercomputers with frequent hardware failures.

Contribution

It presents a novel low-overhead fault-tolerance technique for MapReduce that relies on message backup and local storage, surpassing traditional checkpointing methods.

Findings

Low overhead <4% during fault-free operation

Recovery time is approximately the time to process 1/p of data

Expected communication overhead of 1/(p-1) on p processing elements

Abstract

Supercomputers getting ever larger and energy-efficient is at odds with the reliability of the used hardware. Thus, the time intervals between component failures are decreasing. Contrarily, the latencies for individual operations of coarse-grained big-data tools grow with the number of processors. To overcome the resulting scalability limit, we need to go beyond the current practice of interoperation checkpointing. We give first results on how to achieve this for the popular MapReduce framework where huge multisets are processed by user-defined mapping and reducing functions. We observe that the full state of a MapReduce algorithm is described by its network communication. We present a low-overhead technique with no additional work during fault-free execution and the negligible expected relative communication overhead of $1/(p-1)$ on $p$ PEs. Recovery takes approximately the time of processing $1/p$ of the data on the surviving PEs. We achieve this by backing up self-messages and locally storing all messages sent through the network on the sending and receiving PEs until the next round of global communication. A prototypical implementation already indicates low overhead $<4\,\%$ during fault-free execution.