Continuous evaluation of the performance of cloud infrastructure for scientific applications

TL;DR

This paper introduces an online ecosystem for continuous, collaborative evaluation of cloud infrastructure performance tailored for scientific applications, enabling comparison, traceability, and sharing of benchmark results across vendors.

Contribution

It presents a novel online platform and tools for ongoing, collaborative benchmarking of cloud hardware for scientific workflows, with a shared database for results and case contributions.

Findings

Benchmark results for multiple cloud vendors are available.

The ecosystem facilitates comparison and traceability of performance data.

It supports contribution of new benchmark cases and results.

Abstract

Cloud computing recently developed into a viable alternative to on-premises systems for executing high-performance computing (HPC) applications. With the emergence of new vendors and hardware options, there is now a growing need to continuously evaluate the performance of the infrastructure with respect to the most commonly-used simulation workflows. We present an online ecosystem and the corresponding tools aimed at providing a collaborative and repeatable way to assess the performance of the underlying hardware for multiple real-world application-specific benchmark cases. The ecosystem allows for the benchmark results to be stored and shared online in a centrally accessible database in order to facilitate their comparison, traceability, and curation. We include the current up-to-date example results for multiple cloud vendors and explain how to contribute new results and benchmark cases.