Towards General and Efficient Online Tuning for Spark

TL;DR

This paper introduces a general, efficient online tuning framework for Spark that employs Bayesian optimization, online evaluations, and acceleration techniques, significantly improving performance and resource efficiency in large-scale data processing.

Contribution

It presents a novel online tuning framework for Spark that supports multiple goals, reduces overhead, and accelerates search using innovative techniques like adaptive sub-space generation and meta-learning.

Findings

Saves 57% memory cost on average

Reduces CPU cost by 34.93%

Demonstrates effectiveness on large-scale production tasks

Abstract

The distributed data analytic system -- Spark is a common choice for processing massive volumes of heterogeneous data, while it is challenging to tune its parameters to achieve high performance. Recent studies try to employ auto-tuning techniques to solve this problem but suffer from three issues: limited functionality, high overhead, and inefficient search. In this paper, we present a general and efficient Spark tuning framework that can deal with the three issues simultaneously. First, we introduce a generalized tuning formulation, which can support multiple tuning goals and constraints conveniently, and a Bayesian optimization (BO) based solution to solve this generalized optimization problem. Second, to avoid high overhead from additional offline evaluations in existing methods, we propose to tune parameters along with the actual periodic executions of each job (i.e., online evaluations). To ensure safety during online job executions, we design a safe configuration acquisition method that models the safe region. Finally, three innovative techniques are leveraged to further accelerate the search process: adaptive sub-space generation, approximate gradient descent, and meta-learning method. We have implemented this framework as an independent cloud service, and applied it to the data platform in Tencent. The empirical results on both public benchmarks and large-scale production tasks demonstrate its superiority in terms of practicality, generality, and efficiency. Notably, this service saves an average of 57.00% memory cost and 34.93% CPU cost on 25K in-production tasks within 20 iterations, respectively.