On a Catalogue of Metrics for Evaluating Commercial Cloud Services

TL;DR

This paper systematically reviews and catalogs metrics used for evaluating commercial Cloud services, providing a structured reference to aid future research and benchmarking efforts in Cloud computing evaluation.

Contribution

It introduces a comprehensive evaluation metrics catalogue for Cloud services based on a systematic literature review, filling a gap in structured evaluation frameworks.

Findings

Collected and organized metrics for Cloud service evaluation

Arranged metrics according to Cloud service features

Supports future benchmarking and research in Cloud evaluation

Abstract

Given the continually increasing amount of commercial Cloud services in the market, evaluation of different services plays a significant role in cost-benefit analysis or decision making for choosing Cloud Computing. In particular, employing suitable metrics is essential in evaluation implementations. However, to the best of our knowledge, there is not any systematic discussion about metrics for evaluating Cloud services. By using the method of Systematic Literature Review (SLR), we have collected the de facto metrics adopted in the existing Cloud services evaluation work. The collected metrics were arranged following different Cloud service features to be evaluated, which essentially constructed an evaluation metrics catalogue, as shown in this paper. This metrics catalogue can be used to facilitate the future practice and research in the area of Cloud services evaluation. Moreover, considering metrics selection is a prerequisite of benchmark selection in evaluation implementations, this work also supplements the existing research in benchmarking the commercial Cloud services.