Parallel high-order resolution of the Shallow-water equations on real large-scale meshes with complex bathymetries

TL;DR

This paper presents a parallel method for domain decomposition and ghost-layer generation in large-scale meshes for shallow-water equations, enabling efficient high-order simulations on supercomputers with GPUs.

Contribution

It introduces a parallel approach for domain decomposition and ghost-layer management, stored in a single CGNS file, and demonstrates its effectiveness on large meshes with GPU acceleration.

Findings

Parallel ghost-layer generation scales well with mesh size.

Using multiple GPUs significantly reduces simulation time.

High-order methods achieve accurate results on large river meshes.

Abstract

The resolution of the Shallow-water equations is of practical interest in the study of inundations and often requires very large and dense meshes to accurately simulate river flows. Those large meshes are often decomposed into multiple sub-domains to allow for parallel processing. When such a decomposition process is used in the context of distributed parallel computing, each sub-domain requires an exchange of one or more layers of ghost cells at each time step of the simulation due to the spatial dependency of numerical methods. In the first part of this paper, we show how the domain decomposition and ghost-layer generation process can be performed in a parallel manner for large meshes, and show a new way of storing the resulting sub-domains with all their send/receive information within a single CGNS mesh file. The performance of the ghost-layer generation process is studied both in terms of time and memory on 2D and 3D meshes containing up to 70 million cells. In the second part of the paper, the program developed in the first part is used to generate the domain decomposition of large meshes of practical interest in the study of natural free surface flows. We use our in-house multi-CPU multi-GPU (MPI+CUDA) solver to show the impact of multiple layers of ghost cells on the execution times of the first-order HLLC, and second-order WAF and MUSCL methods. Finally, the parallel solver is used to perform the second-order resolution on real large-scale meshes of rivers near Montr\'eal, using up to 32 GPUs on meshes of 13 million cells.