Towards a Taxonomy of Performance Evaluation of Commercial Cloud Services

TL;DR

This paper proposes a taxonomy to standardize and analyze performance evaluation practices of commercial Cloud services, aiming to improve clarity and consistency in evaluation methods.

Contribution

It introduces a systematic taxonomy based on literature review to profile and design performance evaluations of commercial Cloud services.

Findings

Taxonomy helps analyze existing evaluation practices.

Facilitates designing new performance experiments.

Can be expanded to broader Cloud Computing evaluations.

Abstract

Cloud Computing, as one of the most promising computing paradigms, has become increasingly accepted in industry. Numerous commercial providers have started to supply public Cloud services, and corresponding performance evaluation is then inevitably required for Cloud provider selection or cost-benefit analysis. Unfortunately, inaccurate and confusing evaluation implementations can be often seen in the context of commercial Cloud Computing, which could severely interfere and spoil evaluation-related comprehension and communication. This paper introduces a taxonomy to help profile and standardize the details of performance evaluation of commercial Cloud services. Through a systematic literature review, we constructed the taxonomy along two dimensions by arranging the atomic elements of Cloud-related performance evaluation. As such, this proposed taxonomy can be employed both to analyze existing evaluation practices through decomposition into elements and to design new experiments through composing elements for evaluating performance of commercial Cloud services. Moreover, through smooth expansion, we can continually adapt this taxonomy to the more general area of evaluation of Cloud Computing.