Boosting Metrics for Cloud Services Evaluation -- The Last Mile of Using Benchmark Suites

TL;DR

This paper introduces Boosting Metrics, a novel approach to synthesize multiple benchmarking results into summary measures, enhancing the holistic evaluation of Cloud services and aiding decision-making.

Contribution

It proposes the concept of Boosting Metrics and demonstrates their application to improve the summarization of Cloud service evaluations.

Findings

Boosting metrics can supplement primary evaluation measures.

Boosting metrics help provide a global perspective of Cloud services.

The approach is adaptable to other computing paradigms.

Abstract

Benchmark suites are significant for evaluating various aspects of Cloud services from a holistic view. However, there is still a gap between using benchmark suites and achieving holistic impression of the evaluated Cloud services. Most Cloud service evaluation work intended to report individual benchmarking results without delivering summary measures. As a result, it could be still hard for customers with such evaluation reports to understand an evaluated Cloud service from a global perspective. Inspired by the boosting approaches to machine learning, we proposed the concept Boosting Metrics to represent all the potential approaches that are able to integrate a suite of benchmarking results. This paper introduces two types of preliminary boosting metrics, and demonstrates how the boosting metrics can be used to supplement primary measures of individual Cloud service features. In particular, boosting metrics can play a summary Response role in applying experimental design to Cloud services evaluation. Although the concept Boosting Metrics was refined based on our work in the Cloud Computing domain, we believe it can be easily adapted to the evaluation work of other computing paradigms.