Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates

TL;DR

Oobleck introduces a fault-tolerant distributed training framework for large DNNs that guarantees resilience and high throughput by using heterogeneous pipeline templates and multiple replicas, outperforming existing solutions.

Contribution

It proposes a planning-execution co-design approach with heterogeneous pipeline templates and multiple replicas to ensure fault tolerance without resource idling during large model training.

Findings

Achieves up to 29.6x higher throughput than state-of-the-art solutions.

Provides guaranteed fault tolerance with minimal resource idling.

Successfully scales to large models with billions of parameters.

Abstract

Oobleck enables resilient distributed training of large DNN models with guaranteed fault tolerance. It takes a planning-execution co-design approach, where it first generates a set of heterogeneous pipeline templates and instantiates at least $f+1$ logically equivalent pipeline replicas to tolerate any $f$ simultaneous failures. During execution, it relies on already-replicated model states across the replicas to provide fast recovery. Oobleck provably guarantees that some combination of the initially created pipeline templates can be used to cover all available resources after $f$ or fewer simultaneous failures, thereby avoiding resource idling at all times. Evaluation on large DNN models with billions of parameters shows that Oobleck provides consistently high throughput, and it outperforms state-of-the-art fault tolerance solutions like Bamboo and Varuna by up to $29.6x$.