MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems

TL;DR

This paper introduces MAD-Max, a performance modeling framework that optimizes distributed training of large machine learning models, significantly reducing communication overhead and boosting throughput on GPU clusters.

Contribution

MAD-Max is a novel framework that models and optimizes parallelization strategies for large-scale ML training, enabling hardware-software co-design and substantial performance improvements.

Findings

Communication accounts for 14-32% of GPU hours in large model training.

MAD-Max achieves up to 2.24x throughput increase for pre-training.

MAD-Max achieves up to 5.2x throughput increase for inference.

Abstract

Training and deploying large-scale machine learning models is time-consuming, requires significant distributed computing infrastructures, and incurs high operational costs. Our analysis, grounded in real-world large model training on datacenter-scale infrastructures, reveals that 14~32% of all GPU hours are spent on communication with no overlapping computation. To minimize this outstanding communication latency and other inherent at-scale inefficiencies, we introduce an agile performance modeling framework, MAD-Max. This framework is designed to optimize parallelization strategies and facilitate hardware-software co-design opportunities. Through the application of MAD-Max to a suite of real-world large-scale ML models on state-of-the-art GPU clusters, we showcase potential throughput enhancements of up to 2.24x for pre-training and up to 5.2x for inference scenarios, respectively.