Rover: An online Spark SQL tuning service via generalized transfer learning

TL;DR

Rover is an online Spark SQL tuning service that leverages generalized transfer learning, combining expert knowledge and controlled history transfer, to efficiently optimize configurations and significantly reduce memory costs in industrial workloads.

Contribution

The paper introduces Rover, a novel online tuning system that integrates expert-assisted Bayesian optimization and controlled transfer learning for improved Spark SQL performance.

Findings

Rover reduces memory costs by 50.1% on average across 12,000 tasks.

76.2% of tasks achieve over 60% memory reduction.

Outperforms existing tuning baselines in efficiency and safety.

Abstract

Distributed data analytic engines like Spark are common choices to process massive data in industry. However, the performance of Spark SQL highly depends on the choice of configurations, where the optimal ones vary with the executed workloads. Among various alternatives for Spark SQL tuning, Bayesian optimization (BO) is a popular framework that finds near-optimal configurations given sufficient budget, but it suffers from the re-optimization issue and is not practical in real production. When applying transfer learning to accelerate the tuning process, we notice two domain-specific challenges: 1) most previous work focus on transferring tuning history, while expert knowledge from Spark engineers is of great potential to improve the tuning performance but is not well studied so far; 2) history tasks should be carefully utilized, where using dissimilar ones lead to a deteriorated performance in production. In this paper, we present Rover, a deployed online Spark SQL tuning service for efficient and safe search on industrial workloads. To address the challenges, we propose generalized transfer learning to boost the tuning performance based on external knowledge, including expert-assisted Bayesian optimization and controlled history transfer. Experiments on public benchmarks and real-world tasks show the superiority of Rover over competitive baselines. Notably, Rover saves an average of 50.1% of the memory cost on 12k real-world Spark SQL tasks in 20 iterations, among which 76.2% of the tasks achieve a significant memory reduction of over 60%.