HiCCL: A Hierarchical Collective Communication Library

TL;DR

HiCCL is a flexible, hierarchical collective communication library that significantly improves throughput and portability across diverse GPU systems by decoupling communication logic from hardware-specific optimizations.

Contribution

It introduces a compositional API for collective communication that adapts to various hardware hierarchies, enabling high performance and portability.

Findings

Achieves 17x higher throughput than specialized MPI implementations

Provides competitive performance with vendor-specific libraries

Demonstrates portability across Nvidia, AMD, and Intel GPU systems

Abstract

HiCCL (Hierarchical Collective Communication Library) addresses the growing complexity and diversity in high-performance network architectures. As GPU systems have envolved into networks of GPUs with different multilevel communication hierarchies, optimizing each collective function for a specific system has become a challenging task. Consequently, many collective libraries struggle to adapt to different hardware and software, especially across systems from different vendors. HiCCL's library design decouples the collective communication logic from network-specific optimizations through a compositional API. The communication logic is composed using multicast, reduction, and fence primitives, which are then factorized for a specified network hieararchy using only point-to-point operations within a level. Finally, striping and pipelining optimizations applied as specified for streamlining the execution. Performance evaluation of HiCCL across four different machines$\unicode{x2014}$two with Nvidia GPUs, one with AMD GPUs, and one with Intel GPUs$\unicode{x2014}$demonstrates an average 17$\times$ higher throughput than the collectives of highly specialized GPU-aware MPI implementations, and competitive throughput with those of vendor-specific libraries (NCCL, RCCL, and OneCCL), while providing portability across all four machines.