Collective Communication Profiling of Modern-day Machine Learning Workloads

TL;DR

This paper analyzes the communication patterns of modern machine learning workloads on distributed systems, revealing insights that could improve network resource provisioning and collective communication frameworks.

Contribution

It provides an extensive analysis of collective communication behavior across various ML models using Nvidia's library instrumentation, highlighting the need to rethink current network topologies.

Findings

Communication patterns vary significantly across models.

Network anomalies impact collective operations.

Insights suggest redesigning communication frameworks.

Abstract

Machine Learning jobs, carried out on large number of distributed high performance systems, involve periodic communication using operations like AllReduce, AllGather, and Broadcast. These operations may create high bandwidth and bursty traffic patterns, leading to network congestion and packet loss, thus impacting the performance of these jobs. Hence it is imperative to analyze these patterns, which can be helpful in provisioning network resources depending on the type of machine learning workloads. In this poster we carry out extensive analysis of the collective communication behavior seen in a wide variety of models (ex. DeepSeek, GPT, Llama, etc.) To achieve this we instrument Nvidia Collective Communication Library logging functionality for richer context about the collectives and workloads. We adjust configuration parameters that influence collective communication behavior, such as parallelism, number of nodes, and model type. This overview presents and discusses some of the results on the collective communication behavior for the open source DeepSeek V3 inferencing model, which includes operation type and count, transfer sizes per operation, and request size distribution. Our analysis shows that it makes sense to rethink current collective communication frameworks and network topologies so as to accommodate the effect of network anomalies on the mentioned workloads.