Training Time Prediction for Mixed Precision-based Distributed Training

TL;DR

This paper introduces a precision-aware model for predicting distributed training times in deep learning, significantly improving accuracy by accounting for mixed precision effects.

Contribution

It presents the first prediction model that incorporates floating-point precision variations, including mixed precision, for more accurate training time estimation.

Findings

Prediction error reduced to 9.8% MAPE with the new model.

Static graph-based models can have up to 147.85% MAPE in prediction.

Precision-aware prediction improves resource planning and scheduling.

Abstract

Accurate prediction of training time in distributed deep learning is crucial for resource allocation, cost estimation, and job scheduling. We observe that the floating-point precision setting is a key determinant of training time, leading to training time variations of ~2.4x over its minimum. However, existing studies on distributed training time prediction rely on static model computation graphs that do not capture precision variations, including mixed precision. According to our experiments, training time prediction without considering precision results in significant prediction errors - reaching up to 147.85% in mean absolute percentage error (MAPE). To address this issue, we propose a precision-aware distributed training time predictor that achieves robust accuracy across diverse precision settings, including mixed precision, with 9.8% MAPE.