Modern Gyrokinetic Particle-In-Cell Simulation of Fusion Plasmas on Top Supercomputers

TL;DR

This paper presents a highly scalable gyrokinetic particle-in-cell simulation code for fusion plasmas, optimized for modern supercomputers including GPUs and MICs, enabling advanced turbulence studies with unprecedented resolution.

Contribution

The paper introduces new parallelization and optimization techniques for PIC simulations on diverse supercomputing architectures, significantly enhancing scalability and performance.

Findings

Achieved extreme scalability on supercomputers with heterogeneous architectures.

Enabled high-resolution turbulence simulations for fusion energy research.

Provided performance insights across various high-end computing platforms.

Abstract

The Gyrokinetic Toroidal Code at Princeton (GTC-P) is a highly scalable and portable particle-in-cell (PIC) code. It solves the 5D Vlasov-Poisson equation featuring efficient utilization of modern parallel computer architectures at the petascale and beyond. Motivated by the goal of developing a modern code capable of dealing with the physics challenge of increasing problem size with sufficient resolution, new thread-level optimizations have been introduced as well as a key additional domain decomposition. GTC-P's multiple levels of parallelism, including inter-node 2D domain decomposition and particle decomposition, as well as intra-node shared memory partition and vectorization have enabled pushing the scalability of the PIC method to extreme computational scales. In this paper, we describe the methods developed to build a highly parallelized PIC code across a broad range of supercomputer designs. This particularly includes implementations on heterogeneous systems using NVIDIA GPU accelerators and Intel Xeon Phi (MIC) co-processors and performance comparisons with state-of-the-art homogeneous HPC systems such as Blue Gene/Q. New discovery science capabilities in the magnetic fusion energy application domain are enabled, including investigations of Ion-Temperature-Gradient (ITG) driven turbulence simulations with unprecedented spatial resolution and long temporal duration. Performance studies with realistic fusion experimental parameters are carried out on multiple supercomputing systems spanning a wide range of cache capacities, cache-sharing configurations, memory bandwidth, interconnects and network topologies. These performance comparisons using a realistic discovery-science-capable domain application code provide valuable insights on optimization techniques across one of the broadest sets of current high-end computing platforms worldwide.