Lessons Learned on MPI+Threads Communication

TL;DR

This paper examines the challenges and recent advancements in MPI+threads communication, comparing MPI 4.0 options and MPI Endpoints, and discusses future directions for improving parallel efficiency and usability.

Contribution

It provides a comparative analysis of MPI 4.0 communication mechanisms and MPI Endpoints, highlighting usability, scope, and portability considerations for MPI+threads applications.

Findings

MPI 4.0 offers new performance options but has usability challenges.

MPI Endpoints provide an alternative design with different trade-offs.

Lessons learned inform future directions for MPI+threads communication.

Abstract

Hybrid MPI+threads programming is gaining prominence, but, in practice, applications perform slower with it compared to the MPI everywhere model. The most critical challenge to the parallel efficiency of MPI+threads applications is slow MPI_THREAD_MULTIPLE performance. MPI libraries have recently made significant strides on this front, but to exploit their capabilities, users must expose the communication parallelism in their MPI+threads applications. Recent studies show that MPI 4.0 provides users with new performance-oriented options to do so, but our evaluation of these new mechanisms shows that they pose several challenges. An alternative design is MPI Endpoints. In this paper, we present a comparison of the different designs from the perspective of MPI's end-users: domain scientists and application developers. We evaluate the mechanisms on metrics beyond performance such as usability, scope, and portability. Based on the lessons learned, we make a case for a future direction.