ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation

TL;DR

ReCycle is a system that enhances large DNN training resilience by dynamically rerouting micro-batches during failures, maintaining high throughput without spare servers through pipeline schedule optimizations.

Contribution

ReCycle introduces a novel failure-tolerance method for large-scale DNN training that leverages inherent redundancy and pipeline adjustments to sustain throughput.

Findings

ReCycle outperforms recent fault-tolerance methods by up to 1.64x.

It maintains high training throughput under multiple failures.

The system effectively minimizes throughput degradation during failures.

Abstract

Training large Deep Neural Network (DNN) models requires thousands of GPUs over the course of several days or weeks. At this scale, failures are frequent and can have a big impact on training throughput. Utilizing spare GPU servers to mitigate performance loss becomes increasingly costly as model sizes grow. ReCycle is a system designed for efficient DNN training in the presence of failures, without relying on spare servers. It exploits the inherent functional redundancy in distributed training systems -- where servers across data-parallel groups store the same model parameters -- and pipeline schedule bubbles within each data-parallel group. When servers fails, ReCycle dynamically re-routes micro-batches to data-parallel peers, allowing for uninterrupted training despite multiple failures. However, this re-routing can create imbalances across pipeline stages, leading to reduced training throughput. To address this, ReCycle introduces two key optimizations that ensure re-routed micro-batches are processed within the original pipeline schedule's bubbles. First, it decouples the backward pass into two phases: one for computing gradients for the input and another for calculating gradients for the parameters. Second, it avoids synchronization across pipeline stages by staggering the optimizer step. Together, these optimizations enable adaptive pipeline schedules that minimize or even eliminate training throughput degradation during failures. We describe a prototype for ReCycle and show that it achieves high training throughput under multiple failures, outperforming recent proposals for fault-tolerant training such as Oobleck and Bamboo by up to $1.46\times$ and $1.64\times$, respectively.