Demystifying the Communication Characteristics for Distributed Transformer Models

TL;DR

This paper analyzes the communication patterns of distributed transformer models, especially GPT-based, identifying bottlenecks and guiding future optimizations in communication and framework design.

Contribution

It provides an empirical and analytical study of transformer communication behavior, highlighting key bottlenecks and areas for optimization in distributed training.

Findings

Small message point-to-point communication is a significant bottleneck.

Correlations exist between sequence length, throughput, model size, and optimizations.

Guidance for future framework and HPC middleware improvements.

Abstract

Deep learning (DL) models based on the transformer architecture have revolutionized many DL applications such as large language models (LLMs), vision transformers, audio generation, and time series prediction. Much of this progress has been fueled by distributed training, yet distributed communication remains a substantial bottleneck to training progress. This paper examines the communication behavior of transformer models - that is, how different parallelism schemes used in multi-node/multi-GPU DL Training communicate data in the context of transformers. We use GPT-based language models as a case study of the transformer architecture due to their ubiquity. We validate the empirical results obtained from our communication logs using analytical models. At a high level, our analysis reveals a need to optimize small message point-to-point communication further, correlations between sequence length, per-GPU throughput, model size, and optimizations used, and where to potentially guide further optimizations in framework and HPC middleware design and optimization.