Experience and Analysis of Scalable High-Fidelity Computational Fluid Dynamics on Modular Supercomputing Architectures

TL;DR

This paper evaluates the performance of high-fidelity spectral element CFD simulations on modular supercomputing architectures, analyzing communication, load balancing, and memory considerations to optimize scalability and efficiency.

Contribution

It provides an assessment of spectral element CFD on modular supercomputers, highlighting when multi-module deployment improves performance and how to manage associated challenges.

Findings

Communication overhead often outweighs benefits in multi-module setups.

Using CPUs alongside GPUs can increase memory capacity for large simulations.

A simple performance model helps determine when multi-module execution is advantageous.

Abstract

The never-ending computational demand from simulations of turbulence makes computational fluid dynamics (CFD) a prime application use case for current and future exascale systems. High-order finite element methods, such as the spectral element method, have been gaining traction as they offer high performance on both multicore CPUs and modern GPU-based accelerators. In this work, we assess how high-fidelity CFD using the spectral element method can exploit the modular supercomputing architecture at scale through domain partitioning, where the computational domain is split between a Booster module powered by GPUs and a Cluster module with conventional CPU nodes. We investigate several different flow cases and computer systems based on the modular supercomputing architecture (MSA). We observe that for our simulations, the communication overhead and load balancing issues incurred by incorporating different computing architectures are seldom worthwhile, especially when I/O is also considered, but when the simulation at hand requires more than the combined global memory on the GPUs, utilizing additional CPUs to increase the available memory can be fruitful. We support our results with a simple performance model to assess when running across modules might be beneficial. As MSA is becoming more widespread and efforts to increase system utilization are growing more important our results give insight into when and how a monolithic application can utilize and spread out to more than one module and obtain a faster time to solution.