BigOP: Generating Comprehensive Big Data Workloads as a Benchmarking Framework

TL;DR

BigOP is a comprehensive benchmarking framework for big data systems that abstracts representative operations and workload patterns, enabling automatic test generation and evaluation across multiple platforms like Hadoop, Spark, and MySQL.

Contribution

Introduces BigOP, an end-to-end big data benchmarking framework with an abstraction model for operations and workloads, supporting automatic test generation for diverse systems.

Findings

Successfully implemented an automatic test generation tool.

Benchmarking results for Hadoop, Spark, and MySQL Cluster.

Demonstrated comprehensive workload coverage across different data types.

Abstract

Big Data is considered proprietary asset of companies, organizations, and even nations. Turning big data into real treasure requires the support of big data systems. A variety of commercial and open source products have been unleashed for big data storage and processing. While big data users are facing the choice of which system best suits their needs, big data system developers are facing the question of how to evaluate their systems with regard to general big data processing needs. System benchmarking is the classic way of meeting the above demands. However, existent big data benchmarks either fail to represent the variety of big data processing requirements, or target only one specific platform, e.g. Hadoop. In this paper, with our industrial partners, we present BigOP, an end-to-end system benchmarking framework, featuring the abstraction of representative Operation sets, workload Patterns, and prescribed tests. BigOP is part of an open-source big data benchmarking project, BigDataBench. BigOP's abstraction model not only guides the development of BigDataBench, but also enables automatic generation of tests with comprehensive workloads. We illustrate the feasibility of BigOP by implementing an automatic test generation tool and benchmarking against three widely used big data processing systems, i.e. Hadoop, Spark and MySQL Cluster. Three tests targeting three different application scenarios are prescribed. The tests involve relational data, text data and graph data, as well as all operations and workload patterns. We report results following test specifications.