SWIFT: Expedited Failure Recovery for Large-scale DNN Training

TL;DR

SWIFT is a novel failure recovery method for large-scale distributed deep neural network training that reduces recovery time and overhead by leveraging model replicas and intermediate data logging, without sacrificing training speed or accuracy.

Contribution

SWIFT introduces a logging-based failure recovery approach that avoids copying model states, significantly reducing recovery overhead and accelerating training in large-scale DNNs.

Findings

Reduces failure recovery time significantly

Achieves up to 1.16x training speedup

Maintains model accuracy during recovery

Abstract

As the size of deep learning models gets larger and larger, training takes longer time and more resources, making fault tolerance more and more critical. Existing state-of-the-art methods like CheckFreq and Elastic Horovod need to back up a copy of the model state (i.e., parameters and optimizer states) in memory, which is costly for large models and leads to non-trivial overhead. This paper presents SWIFT, a novel recovery design for distributed deep neural network training that significantly reduces the failure recovery overhead without affecting training throughput and model accuracy. Instead of making an additional copy of the model state, SWIFT resolves the inconsistencies of the model state caused by the failure and exploits the replicas of the model state in data parallelism for failure recovery. We propose a logging-based approach when replicas are unavailable, which records intermediate data and replays the computation to recover the lost state upon a failure. The re-computation is distributed across multiple machines to accelerate failure recovery further. We also log intermediate data selectively, exploring the trade-off between recovery time and intermediate data storage overhead. Evaluations show that SWIFT significantly reduces the failure recovery time and achieves similar or better training throughput during failure-free execution compared to state-of-the-art methods without degrading final model accuracy. SWIFT can also achieve up to 1.16x speedup in total training time compared to state-of-the-art methods.