AsterX: a new open-source GPU-accelerated GRMHD code for dynamical spacetimes

TL;DR

AsterX is an open-source GPU-accelerated GRMHD code designed for dynamic spacetimes, utilizing adaptive mesh refinement and high-performance computing techniques to achieve high accuracy and scalability in 3D simulations.

Contribution

It introduces a modular, GPU-accelerated GRMHD code with adaptive mesh refinement for dynamic spacetimes, optimized for exascale supercomputing environments.

Findings

Achieves 2.5 to 4.5 times performance gain with subcycling

Demonstrates high accuracy and agreement with existing codes

Maintains 67%-77% scaling efficiency on 4096 nodes

Abstract

We present AsterX, a novel open-source, modular, GPU-accelerated, fully general relativistic magnetohydrodynamic (GRMHD) code designed for dynamic spacetimes in 3D Cartesian coordinates, and tailored for exascale computing. We utilize block-structured adaptive mesh refinement (AMR) through CarpetX, the new driver for the Einstein Toolkit, which is built on AMReX, a software framework for massively parallel applications. AsterX employs the Valencia formulation for GRMHD, coupled with the Z4c formalism for spacetime evolution, while incorporating high resolution shock capturing schemes to accurately handle the hydrodynamics. AsterX has undergone rigorous testing in both static and dynamic spacetime, demonstrating remarkable accuracy and agreement with other codes in literature. Using subcycling in time, we find an overall performance gain of factor 2.5 to 4.5. Benchmarking the code through scaling tests on OLCF's Frontier supercomputer, we demonstrate a weak scaling efficiency of about 67%-77% on 4096 nodes compared to an 8-node performance.