Reliability of Computational Experiments on Virtualised Hardware

TL;DR

This paper investigates the reliability and reproducibility of computational experiments on virtualised hardware, especially clouds, finding that virtualisation does not inherently cause variation compared to non-virtualised hardware.

Contribution

It provides preliminary evidence that virtualised hardware can be reliable for computational experiments, addressing concerns about variability in cloud environments.

Findings

Virtualised hardware does not inherently introduce variation.

Reliability depends on the number of VMs on the same hardware.

Preliminary results support using clouds for reproducible experiments.

Abstract

We present preliminary results of an investigation into the suitability of virtualised hardware -- in particular clouds -- for running computational experiments. Our main concern was that the reported CPU time would not be reliable and reproducible. The results demonstrate that while this is true in cases where many virtual machines are running on the same physical hardware, there is no inherent variation introduced by using virtualised hardware compared to non-virtualised hardware.