A Multi-faceted Analysis of the Performance Variability of Virtual Machines

TL;DR

This paper provides a comprehensive analysis of performance variability in virtual machines across multiple cloud providers, introducing new benchmarks, indicators, and predictive models to better understand and forecast VM performance fluctuations.

Contribution

It introduces a new benchmark suite, a variability indicator, and machine learning models for performance prediction, offering the most extensive study on VM variability to date.

Findings

Performance variability differs significantly across providers.

The Variability Indicator effectively measures performance fluctuations.

Predictive models can forecast VM performance patterns.

Abstract

Cloud computing and virtualization solutions allow one to rent the virtual machines (VMs) needed to run applications on a pay-per-use basis, but rented VMs do not offer any guarantee on their performance. Cloud platforms are known to be affected by performance variability, but a better understanding is still required. This paper moves in that direction and presents an in-depth, multi-faceted study on the performance variability of VMs. Unlike previous studies, our assessment covers a wide range of factors: 16 VM types from 4 well-known cloud providers, 10 benchmarks, and 28 different metrics. We present four new contributions. First, we introduce a new benchmark suite (VMBS) that let researchers and practitioners systematically collect a diverse set of performance data. Second, we present a new indicator, called Variability Indicator, that allows for measuring variability in the performance of VMs. Third, we illustrate an analysis of the collected data across four different dimensions: resources, isolation, time, and cost. Fourth, we present multiple predictive models based on Machine Learning that aim to forecast future performance and detect time patterns. Our experiments provide important insights on the resource variability of VMs, highlighting differences and similarities between various cloud providers. To the best of our knowledge, this is the widest analysis ever conducted on the topic.