Benchmarking network fabrics for data distributed training of deep neural networks

TL;DR

This paper evaluates how different network fabrics and communication primitives impact the training efficiency of deep neural networks in distributed data parallel settings, finding minimal effect of Ethernet-based networking on training times.

Contribution

It provides a comparative analysis of network interconnects and communication methods like GPUDirect and NCCL in distributed deep learning training.

Findings

Ethernet-based networking shows negligible impact on training times.

GPUDirect and NCCL improve communication efficiency.

Network fabric choice has limited effect on training performance.

Abstract

Artificial Intelligence/Machine Learning applications require the training of complex models on large amounts of labelled data. The large computational requirements for training deep models have necessitated the development of new methods for faster training. One such approach is the data parallel approach, where the training data is distributed across multiple compute nodes. This approach is simple to implement and supported by most of the commonly used machine learning frameworks. The data parallel approach leverages MPI for communicating gradients across all nodes. In this paper, we examine the effects of using different physical hardware interconnects and network-related software primitives for enabling data distributed deep learning. We compare the effect of using GPUDirect and NCCL on Ethernet and OmniPath fabrics. Our results show that using Ethernet-based networking in shared HPC systems does not have a significant effect on the training times for commonly used deep neural network architectures or traditional HPC applications such as Computational Fluid Dynamics.