OpenACC offloading of the MFC compressible multiphase flow solver on AMD and NVIDIA GPUs

TL;DR

This paper demonstrates efficient acceleration of a multiphase flow solver on AMD and NVIDIA GPUs using OpenACC, achieving significant speedups and high scaling efficiencies through specific optimizations and parallelization techniques.

Contribution

It introduces optimized OpenACC implementation and techniques for multiphase flow simulation on AMD and NVIDIA GPUs, achieving high performance and scalability.

Findings

Achieved up to 30x speedup of kernels.

Maintained over 95% weak scaling efficiency on large systems.

Improved strong scaling efficiency with GPU-aware MPI.

Abstract

GPUs are the heart of the latest generations of supercomputers. We efficiently accelerate a compressible multiphase flow solver via OpenACC on NVIDIA and AMD Instinct GPUs. Optimization is accomplished by specifying the directive clauses 'gang vector' and 'collapse'. Further speedups of six and ten times are achieved by packing user-defined types into coalesced multidimensional arrays and manual inlining via metaprogramming. Additional optimizations yield seven-times speedup in array packing and thirty-times speedup of select kernels on Frontier. Weak scaling efficiencies of 97% and 95% are observed when scaling to 50% of Summit and 95% of Frontier. Strong scaling efficiencies of 84% and 81% are observed when increasing the device count by a factor of 8 and 16 on V100 and MI250X hardware. The strong scaling efficiency of AMD's MI250X increases to 92% when increasing the device count by a factor of 16 when GPU-aware MPI is used for communication.