Boosting Cloud Data Analytics using Multi-Objective Optimization
Fei Song, Khaled Zaouk, Chenghao Lyu, Arnab Sinha, Qi Fan, Yanlei, Diao, Prashant Shenoy

TL;DR
This paper introduces a multi-objective optimization approach for cloud data analytics that automatically configures system parameters to meet user performance and budget goals, significantly improving efficiency and tradeoff exploration.
Contribution
It presents a novel Pareto-based multi-objective optimization method with efficient algorithms and a Spark prototype to optimize cloud analytics configurations.
Findings
2-50x speedup over existing MOO methods
26%-49% reduction in TPCx-BB benchmark runtime
Good coverage of Pareto frontiers with fast recommendations
Abstract
Data analytics in the cloud has become an integral part of enterprise businesses. Big data analytics systems, however, still lack the ability to take user performance goals and budgetary constraints for a task, collectively referred to as task objectives, and automatically configure an analytic job to achieve these objectives. This paper presents a data analytics optimizer that can automatically determine a cluster configuration with a suitable number of cores as well as other system parameters that best meet the task objectives. At a core of our work is a principled multi-objective optimization (MOO) approach that computes a Pareto optimal set of job configurations to reveal tradeoffs between different user objectives, recommends a new job configuration that best explores such tradeoffs, and employs novel optimizations to enable such recommendations within a few seconds. We present…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsCloud Computing and Resource Management · IoT and Edge/Fog Computing · Data Stream Mining Techniques
