Enclave Tasking for Discontinuous Galerkin Methods on Dynamically Adaptive Meshes

TL;DR

This paper introduces enclave tasking, a novel approach for efficiently parallelizing high-order Discontinuous Galerkin methods on dynamically adaptive meshes, significantly improving MPI and shared memory performance.

Contribution

The paper presents a new enclave tasking method that manages dependencies and concurrency in DG methods on adaptive meshes, enhancing parallel efficiency.

Findings

MPI parallelization improves by a factor of three.

Shared memory parallelization adds an additional factor of two.

Enclave tasking effectively overlaps computation and communication.

Abstract

High-order Discontinuous Galerkin (DG) methods promise to be an excellent discretisation paradigm for partial differential equation solvers by combining high arithmetic intensity with localised data access. They also facilitate dynamic adaptivity without the need for conformal meshes. A parallel evaluation of DG's weak formulation within a mesh traversal is non-trivial, as dependency graphs over dynamically adaptive meshes change, as causal constraints along resolution transitions have to be preserved, and as data sends along MPI domain boundaries have to be triggered in the correct order. We propose to process mesh elements subject to constraints with high priority or, where needed, serially throughout a traversal. The remaining cells form enclaves and are spawned into a task system. This introduces concurrency, mixes memory-intensive DG integrations with compute-bound Riemann solves, and overlaps computation and communication. We discuss implications on MPI and show that MPI parallelisation improves by a factor of three through enclave tasking, while we obtain an additional factor of two from shared memory if grids are dynamically adaptive.