# Propagation and Decay of Injected One-Off Delays on Clusters: A Case Study

## TL;DR

This paper investigates how injected delays propagate and decay in distributed-memory systems, analyzing their dependence on application properties and noise levels to improve understanding of performance disturbances.

## Contribution

It provides a detailed analysis of delay propagation mechanisms and how noise influences their decay, offering insights into performance stability in parallel applications.

## Key findings

- Delay propagation speed depends on application communication patterns.
- Increased noise levels can accelerate delay decay.
- Fine-grained noise can mitigate adverse effects of delay propagation.

## Abstract

Analytic, first-principles performance modeling of distributed-memory applications is difficult due to a wide spectrum of random disturbances caused by the application and the system. These disturbances (commonly called "noise") destroy the assumptions of regularity that one usually employs when constructing simple analytic models. Despite numerous efforts to quantify, categorize, and reduce such effects, a comprehensive quantitative understanding of their performance impact is not available, especially for long delays that have global consequences for the parallel application. In this work, we investigate various traces collected from synthetic benchmarks that mimic real applications on simulated and real message-passing systems in order to pinpoint the mechanisms behind delay propagation. We analyze the dependence of the propagation speed of idle waves emanating from injected delays with respect to the execution and communication properties of the application, study how such delays decay under increased noise levels, and how they interact with each other. We also show how fine-grained noise can make a system immune against the adverse effects of propagating idle waves. Our results contribute to a better understanding of the collective phenomena that manifest themselves in distributed-memory parallel applications.