Hybrid Parallelization of Euler-Lagrange Simulations Based on MPI-3 Shared Memory

TL;DR

This paper introduces a novel MPI-3 shared memory-based algorithm for halo element search in Euler-Lagrange simulations, significantly improving scalability and performance on large parallel systems.

Contribution

It presents a communication-free halo search algorithm using MPI-3 shared memory, enabling better scalability and efficiency in unstructured grid simulations.

Findings

Achieved up to 80% efficiency on 131,000 cores

Reduced communication bottlenecks in halo search routines

Enhanced scalability for particle emission and coupling methods

Abstract

The use of Euler-Lagrange methods on unstructured grids extends their application area to more versatile setups. However, the lack of a regular topology limits the scalability of distributed parallel methods, especially for routines that perform a physical search in space. One of the most prominent slowdowns is the search for halo elements in physical space for the purpose of runtime communication avoidance. In this work, we present a new communication-free halo element search algorithm utilizing the MPI-3 shared memory model. This novel method eliminates the severe performance bottleneck of many-to-many communication during initialization compared to the distributed parallelization approach and extends the possible applications beyond those achievable with the previous approach. Building on these data structures, we then present methods for efficient particle emission, scalable deposition schemes for particle-field coupling, and latency hiding approaches. The scaling performance of the proposed algorithms is validated through plasma dynamics simulations of an open-source framework on a massively parallel system, demonstrating an efficiency of up to 80% on 131000 cores.