Boosting Cloud Data Analytics using Multi-Objective Optimization

TL;DR

This paper introduces a multi-objective optimization approach for cloud data analytics that automatically configures system parameters to meet user performance and budget goals, significantly improving efficiency and tradeoff exploration.

Contribution

It presents a novel Pareto-based multi-objective optimization method with efficient algorithms and a Spark prototype to optimize cloud analytics configurations.

Findings

2-50x speedup over existing MOO methods

26%-49% reduction in TPCx-BB benchmark runtime

Good coverage of Pareto frontiers with fast recommendations

Abstract

Data analytics in the cloud has become an integral part of enterprise businesses. Big data analytics systems, however, still lack the ability to take user performance goals and budgetary constraints for a task, collectively referred to as task objectives, and automatically configure an analytic job to achieve these objectives. This paper presents a data analytics optimizer that can automatically determine a cluster configuration with a suitable number of cores as well as other system parameters that best meet the task objectives. At a core of our work is a principled multi-objective optimization (MOO) approach that computes a Pareto optimal set of job configurations to reveal tradeoffs between different user objectives, recommends a new job configuration that best explores such tradeoffs, and employs novel optimizations to enable such recommendations within a few seconds. We present efficient incremental algorithms based on the notion of a Progressive Frontier for realizing our MOO approach and implement them into a Spark-based prototype. Detailed experiments using benchmark workloads show that our MOO techniques provide a 2-50x speedup over existing MOO methods, while offering good coverage of the Pareto frontier. When compared to Ottertune, a state-of-the-art performance tuning system, our approach recommends configurations that yield 26\%-49\% reduction of running time of the TPCx-BB benchmark while adapting to different application preferences on multiple objectives.