HybridTune: Spatio-temporal Data and Model Driven Performance Diagnosis for Big Data Systems

TL;DR

HybridTune is a novel tool that uses spatio-temporal correlation analysis and data-driven algorithms to diagnose performance bottlenecks in Big Data systems like Spark and Hadoop, improving efficiency and accuracy.

Contribution

The paper introduces HybridTune, a lightweight, extensible performance diagnosis tool that combines correlation analysis with data and model-driven algorithms for Big Data systems.

Findings

Achieves about 80% accuracy in abnormal/outlier detection.

Effectively identifies bottlenecks like workload imbalance and data skew.

Supports performance analysis on Spark and Hadoop applications.

Abstract

With tremendous growing interests in Big Data systems, analyzing and facilitating their performance improvement become increasingly important. Although there have much research efforts for improving Big Data systems performance, efficiently analysing and diagnosing performance bottlenecks over these massively distributed systems remain a major challenge. In this paper, we propose a spatio-temporal correlation analysis approach based on stage characteristic and distribution characteristic of Big Data applications, which can associate the multi-level performance data fine-grained. On the basis of correlation data, we define some priori rules, select features and vectorize the corresponding datasets for different performance bottlenecks, such as, workload imbalance, data skew, abnormal node and outlier metrics. And then, we utilize the data and model driven algorithms for bottlenecks detection and diagnosis. In addition, we design and develop a lightweight, extensible tool HybridTune, and validate the diagnosis effectiveness of our tool with BigDataBench on several benchmark experiments in which the outperform state-of-the-art methods. Our experiments show that the accuracy of abnormal/outlier detection we obtained reaches about 80%. At last, we report several Spark and Hadoop use cases, which are demonstrated how HybridTune supports users to carry out the performance analysis and diagnosis efficiently on the Spark and Hadoop applications, and our experiences demonstrate HybridTune can help users find the performance bottlenecks and provide optimization recommendations.