ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload

TL;DR

ReCoVer is a fault-tolerant pre-training system for large language models that maintains training consistency and efficiency despite hardware failures across large GPU clusters.

Contribution

It introduces a novel, parallelism-agnostic framework with three protocol layers that ensures resilient LLM pre-training without drifting from failure-free trajectories.

Findings

Successfully maintains training trajectory despite 256 GPUs failures.

Achieves 2.23x higher effective throughput compared to checkpointing.

Processes 74.9% more tokens at 234 GPU-hours with increased prolongation.

Abstract

Pre-training large language models on massive GPU clusters has made hardware faults routine rather than rare, driving the need for resilient training systems. Yet existing frameworks either focus on specific parallelism schemes or risk drifting away from a failure-free training trajectory. We propose ReCoVer, a resilient LLM pre-training system that upholds a single invariant: each iteration keeps the number of microbatches constant, ensuring per-iteration gradients remain stochastically equivalent to a failure-free run. The framework is organized as three decoupled protocol layers: (1) Fault-tolerant collectives that isolate faults from propagating across replicas; (2) in-step fine-grained recovery that preserves intra-iteration progress and prevents gradient corruption; (3) versatile-workload policy that dynamically redistributes microbatch quotas across the survivors. The design is parallelism-agnostic, integrating directly with both 3D parallelism and Hybrid Sharded Data Parallel (HSDP) as a drop-in substrate. We evaluate our implementation on end-to-end pre-training tasks for up to 512 GPUs, ReCoVer successfully preserves the training trajectory from a failure-free reference despite of 256 GPUs lost spread across the run. For comparison with checkpoint-and-restart baselines, ReCoVer demonstrates $2.23\times$ higher effective throughput after successive failures. This advantage results in ReCoVer processing 74.9% more tokens at 234 GPU-hours, with the gap widening as the training prolongs.