Co-Design and Evaluation of a CPU-Free MPI GPU Communication Abstraction and Implementation

TL;DR

This paper presents a novel MPI-based GPU communication API that eliminates CPU involvement, significantly reducing latency and improving scalability for GPU-based HPC and ML applications.

Contribution

It introduces a CPU-free MPI GPU communication API leveraging new network capabilities, enabling high-performance, easy-to-use GPU communication without CPU synchronization.

Findings

Up to 50% reduction in medium message latency.

28% speedup in halo-exchange benchmark on 8,192 GPUs.

Effective integration with Cabana/Kokkos framework.

Abstract

Removing the CPU from the communication fast path is essential to efficient GPU-based ML and HPC application performance. However, existing GPU communication APIs either continue to rely on the CPU for communication or rely on APIs that place significant synchronization burdens on programmers. In this paper we describe the design, implementation, and evaluation of an MPI-based GPU communication API enabling easy-to-use, high-performance, CPU-free communication. This API builds on previously proposed MPI extensions and leverages HPE Slingshot 11 network card capabilities. We demonstrate the utility and performance of the API by showing how the API naturally enables CPU-free gather/scatter halo exchange communication primitives in the Cabana/Kokkos performance portability framework, and through a performance comparison with Cray MPICH on the Frontier and Tuolumne supercomputers. Results from this evaluation show up to a 50% reduction in medium message latency in simple GPU ping-pong exchanges and a 28% speedup improvement when strong scaling a halo-exchange benchmark to 8,192 GPUs of the Frontier supercomputer.