Trace-based, time-resolved analysis of MPI application performance using standard metrics

TL;DR

This paper introduces a method for detailed, time-resolved analysis of MPI application performance metrics from execution traces, revealing transient bottlenecks often hidden in aggregated data.

Contribution

It presents a novel approach to compute standard MPI metrics over fixed or adaptive time segments, enabling detailed performance analysis even with large trace sizes.

Findings

Time-resolved metrics uncover localized performance bottlenecks.

Method is scalable and effective on real-world applications.

Handles common trace anomalies robustly.

Abstract

Detailed trace analysis of MPI applications is essential for performance engineering, but growing trace sizes and complex communication behaviour often render comprehensive visual inspection impractical. This work presents a trace-based calculation of time-resolved values of standard MPI performance metrics, load balance, serialisation, and transfer efficiency, by discretising execution traces into fixed or adaptive time segments. The implementation processes Paraver traces postmortem, reconstructing critical execution paths and handling common event anomalies, such as clock inconsistencies and unmatched MPI events, to robustly calculate metrics for each segment. The calculated per-window metric values expose transient performance bottlenecks that the timeaggregated metrics from existing tools may conceal. Evaluations on a synthetic benchmark and real-world applications (LaMEM and ls1-MarDyn) demonstrate how time-resolved metrics reveal localised performance bottlenecks obscured by global aggregates, offering a lightweight and scalable alternative even when trace visualisation is impractical.