Collectives in hybrid MPI+MPI code: design, practice and performance

TL;DR

This paper introduces a new design method for hybrid MPI+MPI collective communication that reduces on-node memory overhead and improves performance, validated through benchmarks and computational kernels.

Contribution

It proposes a novel design approach for MPI+MPI collective operations, including wrapper primitives and best practices, enhancing efficiency over traditional methods.

Findings

Micro-benchmarks show comparable or better performance than pure MPI.

Validated effectiveness in three computational kernels.

Reduces on-node communication overheads.

Abstract

The use of hybrid scheme combining the message passing programming models for inter-node parallelism and the shared memory programming models for node-level parallelism is widely spread. Existing extensive practices on hybrid Message Passing Interface (MPI) plus Open Multi-Processing (OpenMP) programming account for its popularity. Nevertheless, strong programming efforts are required to gain performance benefits from the MPI+OpenMP code. An emerging hybrid method that combines MPI and the MPI shared memory model (MPI+MPI) is promising. However, writing an efficient hybrid MPI+MPI program -- especially when the collective communication operations are involved -- is not to be taken for granted. In this paper, we propose a new design method to implement hybrid MPI+MPI context-based collective communication operations. Our method avoids on-node memory replications (on-node communication overheads) that are required by semantics in pure MPI. We also offer wrapper primitives hiding all the design details from users, which comes with practices on how to structure hybrid MPI+MPI code with these primitives. The micro-benchmarks show that our collectives are comparable or superior to those in pure MPI context. We have further validated the effectiveness of the hybrid MPI+MPI model (which uses our wrapper primitives) in three computational kernels, by comparison to the pure MPI and hybrid MPI+OpenMP models.