Porting the grid-based 3D+3V hybrid-Vlasov kinetic plasma simulation Vlasiator to heterogeneous GPU architectures

TL;DR

This paper discusses porting the space plasma simulation code Vlasiator to heterogeneous GPU architectures, enhancing its performance and portability across different hardware for large-scale exascale simulations.

Contribution

It introduces a unified codebase and novel GPU porting strategies for Vlasiator, enabling efficient multi-GPU and multi-node exascale plasma simulations.

Findings

Achieved portability to NVIDIA and AMD GPUs using CUDA and HIP.

Developed a parallel block adjustment method for sparse velocity meshes.

Outlined optimization plans for future exascale supercomputing applications.

Abstract

Vlasiator is a space plasma simulation code which models near-Earth ion-kinetic dynamics in three spatial and three velocity dimensions. It is highly parallelized, modeling the Vlasov equation directly through the distribution function, discretized on a Cartesian grid, instead of the more common particle-in-cell approach. Modeling near-Earth space, plasma properties span several orders of magnitude in temperature, density, and magnetic field strength. In order to fit the required six-dimensional grids in memory, Vlasiator utilizes a sparse block-based velocity mesh, where chunks of velocity space are added or deleted based on the advection requirements of the Vlasov solver. In addition, the spatial mesh is adaptively refined through cell-based octree refinement. In this paper, we describe the design choices of porting Vlasiator to heterogeneous CPU/GPU architectures. We detail the memory management, algorithmic changes, and kernel construction as well as our unified codebase approach, resulting in portability to both NVIDIA and AMD hardware (CUDA and HIP languages, respectively). In particular, we showcase a highly parallel block adjustment approach allowing efficient re-ordering of a sparse velocity mesh. We detail pitfalls we have overcome and lay out a plan for optimization to facilitate future exascale simulations using multi-node GPU supercomputing.