A Factor Framework for Experimental Design for Performance Evaluation of Commercial Cloud Services

TL;DR

This paper introduces a systematic factor framework to guide the design of performance evaluation experiments for commercial Cloud services, addressing the lack of structured approaches in current practices.

Contribution

It proposes a comprehensive factor framework based on prior taxonomy and modeling work to improve experimental design in Cloud service performance evaluation.

Findings

Capsules current evaluation factors in Cloud computing

Facilitates systematic experimental design

Supports better decision-making for Cloud service evaluation

Abstract

Given the diversity of commercial Cloud services, performance evaluations of candidate services would be crucial and beneficial for both service customers (e.g. cost-benefit analysis) and providers (e.g. direction of service improvement). Before an evaluation implementation, the selection of suitable factors (also called parameters or variables) plays a prerequisite role in designing evaluation experiments. However, there seems a lack of systematic approaches to factor selection for Cloud services performance evaluation. In other words, evaluators randomly and intuitively concerned experimental factors in most of the existing evaluation studies. Based on our previous taxonomy and modeling work, this paper proposes a factor framework for experimental design for performance evaluation of commercial Cloud services. This framework capsules the state-of-the-practice of performance evaluation factors that people currently take into account in the Cloud Computing domain, and in turn can help facilitate designing new experiments for evaluating Cloud services.