A Distributed Framework for Causal Modeling of Performance Variability in GPU Traces

TL;DR

This paper introduces a parallel framework for analyzing large-scale GPU traces to efficiently identify performance bottlenecks and variability in heterogeneous HPC systems, significantly improving scalability.

Contribution

The paper presents a novel distributed framework that processes multiple GPU traces concurrently using causal graph methods, enhancing analysis efficiency and scalability.

Findings

Achieved 67% scalability improvement in trace analysis

Effectively exposed performance variability and dependencies

Demonstrated efficiency in handling large-scale GPU data

Abstract

Large-scale GPU traces play a critical role in identifying performance bottlenecks within heterogeneous High-Performance Computing (HPC) architectures. However, the sheer volume and complexity of a single trace of data make performance analysis both computationally expensive and time-consuming. To address this challenge, we present an end-to-end parallel performance analysis framework designed to handle multiple large-scale GPU traces efficiently. Our proposed framework partitions and processes trace data concurrently and employs causal graph methods and parallel coordinating chart to expose performance variability and dependencies across execution flows. Experimental results demonstrate a 67% improvement in terms of scalability, highlighting the effectiveness of our pipeline for analyzing multiple traces independently.