Scalable GPU Performance Variability Analysis framework

TL;DR

This paper presents a distributed GPU performance analysis framework that efficiently processes large-scale trace data, enabling faster diagnosis of variability and insights into GPU kernel behavior in HPC and AI workloads.

Contribution

We introduce a scalable, distributed analysis system that partitions trace data for concurrent processing, reducing memory usage and analysis time compared to traditional sequential tools.

Findings

Effective diagnosis of performance variability.

Uncovered impact of memory transfer latency on GPU kernels.

Demonstrated scalability on real HPC and AI workloads.

Abstract

Analyzing large-scale performance logs from GPU profilers often requires terabytes of memory and hours of runtime, even for basic summaries. These constraints prevent timely insight and hinder the integration of performance analytics into automated workflows. Existing analysis tools typically process data sequentially, making them ill-suited for HPC workflows with growing trace complexity and volume. We introduce a distributed data analysis framework that scales with dataset size and compute availability. Rather than treating the dataset as a single entity, our system partitions it into independently analyzable shards and processes them concurrently across MPI ranks. This design reduces per-node memory pressure, avoids central bottlenecks, and enables low-latency exploration of high-dimensional trace data. We apply the framework to end-to-end Nsight Compute traces from real HPC and AI workloads, demonstrate its ability to diagnose performance variability, and uncover the impact of memory transfer latency on GPU kernel behavior.