Automated Calibration of Parallel and Distributed Computing Simulators: A Case Study

TL;DR

This paper demonstrates that automating the calibration of parallel and distributed system simulators using simple algorithms can match or surpass manual calibration, improving efficiency and enabling better trade-offs between accuracy and speed.

Contribution

The paper presents a case study showing that automated calibration methods can effectively replace manual calibration in simulator configuration, enhancing accuracy and efficiency.

Findings

Automated calibration matches or outperforms manual calibration.

Automation simplifies the calibration process and enables trade-offs.

Automated methods improve simulation accuracy and speed management.

Abstract

Many parallel and distributed computing research results are obtained in simulation, using simulators that mimic real-world executions on some target system. Each such simulator is configured by picking values for parameters that define the behavior of the underlying simulation models it implements. The main concern for a simulator is accuracy: simulated behaviors should be as close as possible to those observed in the real-world target system. This requires that values for each of the simulator's parameters be carefully picked, or "calibrated," based on ground-truth real-world executions. Examining the current state of the art shows that simulator calibration, at least in the field of parallel and distributed computing, is often undocumented (and thus perhaps often not performed) and, when documented, is described as a labor-intensive, manual process. In this work we evaluate the benefit of automating simulation calibration using simple algorithms. Specifically, we use a real-world case study from the field of High Energy Physics and compare automated calibration to calibration performed by a domain scientist. Our main finding is that automated calibration is on par with or significantly outperforms the calibration performed by the domain scientist. Furthermore, automated calibration makes it straightforward to operate desirable trade-offs between simulation accuracy and simulation speed.