The Ghost of Performance Reproducibility Past

TL;DR

This paper investigates performance fluctuations in ensemble computing, especially for GROMACS workflows, highlighting challenges in understanding and reproducing performance results at scale.

Contribution

It provides an in-depth analysis of performance variability in ensemble applications and discusses the implications for performance measurement and reproducibility.

Findings

Performance fluctuations are complex and not fully understood.

Unsuccessful attempts to identify causes highlight the challenge.

Discussion suggests a need to rethink performance reproducibility concepts.

Abstract

The importance of ensemble computing is well established. However, executing ensembles at scale introduces interesting performance fluctuations that have not been well investigated. In this paper, we trace our experience uncovering performance fluctuations of ensemble applications (primarily constituting a workflow of GROMACS tasks), and unsuccessful attempts, so far, at trying to discern the underlying cause(s) of performance fluctuations. Is the failure to discern the causative or contributing factors a failure of capability? Or imagination? Do the fluctuations have their genesis in some inscrutable aspect of the system or software? Does it warrant a fundamental reassessment and rethinking of how we assume and conceptualize performance reproducibility? Answers to these questions are not straightforward, nor are they immediate or obvious. We conclude with a discussion about the performance of ensemble applications and ruminate over the implications for how we define and measure application performance.