KEWS: A KPIs-Based Evaluation Framework of Workload Simulation On Microservice System

TL;DR

This paper presents KEWS, a comprehensive framework utilizing a novel KPI similarity metric, ESBD, to effectively evaluate workload simulations in microservice systems, ensuring realistic and privacy-preserving testing.

Contribution

The paper introduces the ESBD metric for KPI similarity and the KEWS framework with preprocessing, compression, and evaluation modules for workload simulation assessment.

Findings

ESBD accurately measures KPI similarity in shape and intensity.

KEWS effectively evaluates workload simulation quality.

Experimental results validate the proposed methods.

Abstract

Simulating the workload is an essential procedure in microservice systems as it helps augment realistic workloads whilst safeguarding user privacy. The efficacy of such simulation depends on its dynamic assessment. The straightforward and most efficient approach to this is comparing the original workload with the simulated one using Key Performance Indicators (KPIs), which capture the state of the system. Nonetheless, due to the extensive volume and complexity of KPIs, fully evaluating them is not feasible, and measuring their similarity poses a significant challenge. This paper introduces a similarity metric algorithm for KPIs, the Extended Shape-Based Distance (ESBD), which gauges similarity in both shape and intensity. Additionally, we propose a KPI-based Evaluation Framework for Workload Simulations (KEWS), comprising three modules: preprocessing, compression, and evaluation. These methodologies effectively counteract the adverse effects of KPIs' characteristics and offer a holistic evaluation. Experimental results substantiate the effectiveness of both ESBD and KEWS.