Scalable GPU Performance Variability Analysis framework
Ankur Lahiry, Ayush Pokharel, Seth Ockerman, Amal Gueroudji, Line Pouchard, Tanzima Z. Islam

TL;DR
This paper presents a distributed GPU performance analysis framework that efficiently processes large-scale trace data, enabling faster diagnosis of variability and insights into GPU kernel behavior in HPC and AI workloads.
Contribution
We introduce a scalable, distributed analysis system that partitions trace data for concurrent processing, reducing memory usage and analysis time compared to traditional sequential tools.
Findings
Effective diagnosis of performance variability.
Uncovered impact of memory transfer latency on GPU kernels.
Demonstrated scalability on real HPC and AI workloads.
Abstract
Analyzing large-scale performance logs from GPU profilers often requires terabytes of memory and hours of runtime, even for basic summaries. These constraints prevent timely insight and hinder the integration of performance analytics into automated workflows. Existing analysis tools typically process data sequentially, making them ill-suited for HPC workflows with growing trace complexity and volume. We introduce a distributed data analysis framework that scales with dataset size and compute availability. Rather than treating the dataset as a single entity, our system partitions it into independently analyzable shards and processes them concurrently across MPI ranks. This design reduces per-node memory pressure, avoids central bottlenecks, and enables low-latency exploration of high-dimensional trace data. We apply the framework to end-to-end Nsight Compute traces from real HPC and AI…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDistributed and Parallel Computing Systems · Parallel Computing and Optimization Techniques
