Cholla-MHD: An Exascale-Capable Magnetohydrodynamic Extension to the Cholla Astrophysical Simulation Code

TL;DR

Cholla-MHD extends the GPU-accelerated Cholla code to efficiently perform large-scale magnetohydrodynamics simulations on exascale supercomputers, achieving high performance, scalability, and accuracy.

Contribution

This work introduces a GPU-native MHD module for Cholla, enabling exascale-capable, high-resolution simulations with robust testing and continuous integration tools.

Findings

Achieves 260 million cell updates per GPU-second.

Supports simulations with ~500^3 cells on a single GPU.

Maintains zero magnetic divergence to round-off error.

Abstract

We present an extension of the massively parallel, GPU native, astrophysical hydrodynamics code Cholla to magnetohydrodynamics (MHD). Cholla solves the ideal MHD equations in their Eulerian form on a static Cartesian mesh utilizing the Van Leer + Constrained Transport integrator, the HLLD Riemann solver, and reconstruction methods at second and third order. Cholla's MHD module can perform $\approx260$ million cell updates per GPU-second on an NVIDIA A100 while using the HLLD Riemann solver and second order reconstruction. The inherently parallel nature of GPUs combined with increased memory in new hardware allows Cholla's MHD module to perform simulations with resolutions $\sim500^3$ cells on a single high end GPU (e.g. an NVIDIA A100 with 80GB of memory). We employ GPU direct MPI to attain excellent weak scaling on the exascale supercomputer \textit{Frontier}, while using 74,088 GPUs and simulating a total grid size of over 7.2 trillion cells. A suite of test problems highlights the accuracy of Cholla's MHD module and demonstrates that zero magnetic divergence in solutions is maintained to round off error. We also present new testing and continuous integration tools using GoogleTest, GitHub Actions, and Jenkins that have made development more robust and accurate and ensure reliability in the future.