Metrics and Design of an Instruction Roofline Model for AMD GPUs

TL;DR

This paper develops an instruction roofline model for AMD GPUs to evaluate application performance, addressing profiling tool limitations and comparing AMD and NVIDIA GPU architectures using a scientific application.

Contribution

It introduces an instruction roofline model for AMD GPUs utilizing AMD's ROCProfiler and BabelStream, enabling performance measurement and comparison with NVIDIA GPUs.

Findings

AMD MI100 achieves similar or better execution time than NVIDIA V100.

Profiling tool differences complicate direct performance comparisons.

AMD MI60 shows the worst performance among the tested GPUs.

Abstract

Due to the recent announcement of the Frontier supercomputer, many scientific application developers are working to make their applications compatible with AMD architectures (CPU-GPU), which means moving away from the traditional CPU and NVIDIA-GPU systems. Due to the current limitations of profiling tools for AMD GPUs, this shift leaves a void in how to measure application performance on AMD GPUs. In this paper, we design an instruction roofline model for AMD GPUs using AMD's ROCProfiler and a benchmarking tool, BabelStream (the HIP implementation), as a way to measure an application's performance in instructions and memory transactions on new AMD hardware. Specifically, we create instruction roofline models for a case study scientific application, PIConGPU, an open source particle-in-cell (PIC) simulations application used for plasma and laser-plasma physics on the NVIDIA V100, AMD Radeon Instinct MI60, and AMD Instinct MI100 GPUs. When looking at the performance of multiple kernels of interest in PIConGPU we find that although the AMD MI100 GPU achieves a similar, or better, execution time compared to the NVIDIA V100 GPU, profiling tool differences make comparing performance of these two architectures hard. When looking at execution time, GIPS, and instruction intensity, the AMD MI60 achieves the worst performance out of the three GPUs used in this work.