HetCCL: Accelerating LLM Training with Heterogeneous GPUs

TL;DR

HetCCL is a new communication library that enables efficient, high-performance training of large language models across heterogeneous GPU clusters from different vendors, without requiring driver modifications.

Contribution

HetCCL introduces a novel approach for collective communication across heterogeneous GPUs, unifying vendor-specific backends and enabling RDMA-based communication without driver changes.

Findings

Matches homogeneous GPU performance in single-vendor setups

Scales effectively in multi-vendor heterogeneous environments

Enables practical high-performance training with NVIDIA and AMD GPUs

Abstract

The rapid growth of large language models is driving organizations to expand their GPU clusters, often with GPUs from multiple vendors. However, current deep learning frameworks lack support for collective communication across heterogeneous GPUs, leading to inefficiency and higher costs. We present HetCCL, a collective communication library that unifies vendor-specific backends and enables RDMA-based communication across GPUs without requiring driver modifications. HetCCL introduces two novel mechanisms that enable cross-vendor communication while leveraging optimized vendor libraries, NVIDIA NCCL and AMD RCCL. Evaluations on a multi-vendor GPU cluster show that HetCCL matches NCCL and RCCL performance in homogeneous setups while uniquely scaling in heterogeneous environments, enabling practical, high-performance training with both NVIDIA and AMD GPUs without changes to existing deep learning applications.