Measurement and Analysis of GPU-accelerated Applications with HPCToolkit

TL;DR

This paper presents extensions to HPCToolkit for performance measurement and analysis of GPU-accelerated applications on exascale supercomputers, including novel data structures, call path profiling, and hardware counter analysis.

Contribution

The paper introduces new techniques and tools in HPCToolkit for efficient GPU performance measurement, attribution, and visualization in complex, high-performance computing environments.

Findings

Effective GPU performance attribution using novel wait-free data structures.

Ability to analyze GPU code generated from high-level programming models.

Successful application to exascale computing codes demonstrating new capabilities.

Abstract

To address the challenge of performance analysis on the US DOE's forthcoming exascale supercomputers, Rice University has been extending its HPCToolkit performance tools to support measurement and analysis of GPU-accelerated applications. To help developers understand the performance of accelerated applications as a whole, HPCToolkit's measurement and analysis tools attribute metrics to calling contexts that span both CPUs and GPUs. To measure GPU-accelerated applications efficiently, HPCToolkit employs a novel wait-free data structure to coordinate monitoring and attribution of GPU performance. To help developers understand the performance of complex GPU code generated from high-level programming models, HPCToolkit constructs sophisticated approximations of call path profiles for GPU computations. To support fine-grained analysis and tuning, HPCToolkit uses PC sampling and instrumentation to measure and attribute GPU performance metrics to source lines, loops, and inlined code. To supplement fine-grained measurements, HPCToolkit can measure GPU kernel executions using hardware performance counters. To provide a view of how an execution evolves over time, HPCToolkit can collect, analyze, and visualize call path traces within and across nodes. Finally, on NVIDIA GPUs, HPCToolkit can derive and attribute a collection of useful performance metrics based on measurements using GPU PC samples. We illustrate HPCToolkit's new capabilities for analyzing GPU-accelerated applications with several codes developed as part of the Exascale Computing Project.