Auto Tuning of Hadoop and Spark parameters

TL;DR

This paper introduces two automatic parameter tuning algorithms for Hadoop and Spark that significantly reduce execution time, improving efficiency in big data processing.

Contribution

It proposes two novel algorithms, Grid Search with Finer Tuning and Controlled Random Search, for automatic tuning of Hadoop and Spark parameters.

Findings

70% reduction in Hadoop execution time with Grid Search

81.19% reduction in Spark execution time with Grid Search

Controlled Random Search also significantly reduces execution time

Abstract

Data of the order of terabytes, petabytes, or beyond is known as Big Data. This data cannot be processed using the traditional database software, and hence there comes the need for Big Data Platforms. By combining the capabilities and features of various big data applications and utilities, Big Data Platforms form a single solution. It is a platform that helps to develop, deploy and manage the big data environment. Hadoop and Spark are the two open-source Big Data Platforms provided by Apache. Both these platforms have many configurational parameters, which can have unforeseen effects on the execution time, accuracy, etc. Manual tuning of these parameters can be tiresome, and hence automatic ways should be needed to tune them. After studying and analyzing various previous works in automating the tuning of these parameters, this paper proposes two algorithms - Grid Search with Finer Tuning and Controlled Random Search. The performance indicator studied in this paper is Execution Time. These algorithms help to tune the parameters automatically. Experimental results have shown a reduction in execution time of about 70% and 50% for Hadoop and 81.19% and 77.77% for Spark by Grid Search with Finer Tuning and Controlled Random Search, respectively.