What does fault tolerant Deep Learning need from MPI?

TL;DR

This paper explores the requirements for fault tolerant MPI in deep learning, proposing extensions and evaluating them with large-scale experiments to improve resilience during long training runs.

Contribution

It analyzes fault tolerance needs for MPI in deep learning, evaluates existing MPI fault tolerance proposals, and extends MaTEx-Caffe with ULFM for improved fault resilience.

Findings

ULFM-based MPI effectively handles faults in large-scale DL training

Fault tolerance features are crucial for long-duration DL jobs on large datasets

Extended MaTEx-Caffe demonstrates improved robustness with ULFM

Abstract

Deep Learning (DL) algorithms have become the de facto Machine Learning (ML) algorithm for large scale data analysis. DL algorithms are computationally expensive - even distributed DL implementations which use MPI require days of training (model learning) time on commonly studied datasets. Long running DL applications become susceptible to faults - requiring development of a fault tolerant system infrastructure, in addition to fault tolerant DL algorithms. This raises an important question: What is needed from MPI for de- signing fault tolerant DL implementations? In this paper, we address this problem for permanent faults. We motivate the need for a fault tolerant MPI specification by an in-depth consideration of recent innovations in DL algorithms and their properties, which drive the need for specific fault tolerance features. We present an in-depth discussion on the suitability of different parallelism types (model, data and hybrid); a need (or lack thereof) for check-pointing of any critical data structures; and most importantly, consideration for several fault tolerance proposals (user-level fault mitigation (ULFM), Reinit) in MPI and their applicability to fault tolerant DL implementations. We leverage a distributed memory implementation of Caffe, currently available under the Machine Learning Toolkit for Extreme Scale (MaTEx). We implement our approaches by ex- tending MaTEx-Caffe for using ULFM-based implementation. Our evaluation using the ImageNet dataset and AlexNet, and GoogLeNet neural network topologies demonstrates the effectiveness of the proposed fault tolerant DL implementation using OpenMPI based ULFM.