Decoupled DiLoCo for Resilient Distributed Pre-training

TL;DR

Decoupled DiLoCo introduces an asynchronous, resilient distributed training framework that improves efficiency and fault tolerance in large-scale language model pre-training by breaking synchronization barriers.

Contribution

It extends DiLoCo by enabling asynchronous communication among independent learners, enhancing fault tolerance and training efficiency in failure-prone environments.

Findings

Significantly improved training efficiency in failure-prone environments.

Maintains competitive model performance across various tasks and architectures.

Achieves zero global downtime during simulated failures.

Abstract

Modern large-scale language model pre-training relies heavily on the single program multiple data (SPMD) paradigm, which requires tight coupling across accelerators. Due to this coupling, transient slowdowns, hardware failures, and synchronization overhead stall the entire computation, wasting significant compute time at scale. While recent distributed methods like DiLoCo reduced communication bandwidth, they remained fundamentally synchronous and vulnerable to these system stalls. To address this, we introduce Decoupled DiLoCo, an evolution of the DiLoCo framework designed to break the lock-step synchronization barrier and go beyond SPMD to maximize training goodput. Decoupled DiLoCo partitions compute across multiple independent ``learners'' that execute local inner optimization steps. These learners asynchronously communicate parameter fragments to a central synchronizer, which circumvents failed or straggling learners by aggregating updates using a minimum quorum, an adaptive grace window, and dynamic token-weighted merging. Inspired by ``chaos engineering'', we achieve significantly improved training efficiency in failure-prone environments with millions of simulated chips with strictly zero global downtime, while maintaining competitive model performance across text and vision tasks, for both dense and mixture-of-expert architectures.