Directionally Unsplit Hydrodynamic Schemes with Hybrid MPI/OpenMP/GPU Parallelization in AMR

TL;DR

This paper introduces GPU-accelerated, directionally unsplit hydrodynamic schemes integrated into an AMR code, demonstrating significant speed-ups over CPU implementations and existing codes like Athena.

Contribution

The paper presents the implementation of GPU-based hydrodynamic schemes with hybrid MPI/OpenMP parallelization in an AMR framework, achieving substantial performance improvements.

Findings

GPU schemes achieve over 100x speed-up compared to CPU

Hybrid MPI/OpenMP enhances performance on heterogeneous clusters

GAMER outperforms Athena by two orders of magnitude in speed

Abstract

We present the implementation and performance of a class of directionally unsplit Riemann-solver-based hydrodynamic schemes on Graphic Processing Units (GPU). These schemes, including the MUSCL-Hancock method, a variant of the MUSCL-Hancock method, and the corner-transport-upwind method, are embedded into the adaptive-mesh-refinement (AMR) code GAMER. Furthermore, a hybrid MPI/OpenMP model is investigated, which enables the full exploitation of the computing power in a heterogeneous CPU/GPU cluster and significantly improves the overall performance. Performance benchmarks are conducted on the Dirac GPU cluster at NERSC/LBNL using up to 32 Tesla C2050 GPUs. A single GPU achieves speed-ups of 101(25) and 84(22) for uniform-mesh and AMR simulations, respectively, as compared with the performance using one(four) CPU core(s), and the excellent performance persists in multi-GPU tests. In addition, we make a direct comparison between GAMER and the widely-adopted CPU code Athena (Stone et al. 2008) in adiabatic hydrodynamic tests and demonstrate that, with the same accuracy, GAMER is able to achieve two orders of magnitude performance speed-up.