Cloud Benchmarking For Maximising Performance of Scientific Applications

TL;DR

This paper presents a six-step benchmarking methodology that helps users select optimal cloud VMs for scientific applications by considering performance and cost, based on application-specific weights.

Contribution

It introduces a novel benchmarking approach that incorporates user-defined weights to rank cloud VMs for maximizing application performance and cost-efficiency.

Findings

Top-ranked VMs achieve maximum application performance.

At least one of the top three VMs offers cost-effective performance.

Method validated on three case study applications.

Abstract

How can applications be deployed on the cloud to achieve maximum performance? This question is challenging to address with the availability of a wide variety of cloud Virtual Machines (VMs) with different performance capabilities. The research reported in this paper addresses the above question by proposing a six step benchmarking methodology in which a user provides a set of weights that indicate how important memory, local communication, computation and storage related operations are to an application. The user can either provide a set of four abstract weights or eight fine grain weights based on the knowledge of the application. The weights along with benchmarking data collected from the cloud are used to generate a set of two rankings - one based only on the performance of the VMs and the other takes both performance and costs into account. The rankings are validated on three case study applications using two validation techniques. The case studies on a set of experimental VMs highlight that maximum performance can be achieved by the three top ranked VMs and maximum performance in a cost-effective manner is achieved by at least one of the top three ranked VMs produced by the methodology.