On data skewness, stragglers, and MapReduce progress indicators

TL;DR

This paper introduces NearestFit, a novel progress indicator for MapReduce that accurately predicts job completion times despite data skewness and load imbalance, outperforming existing methods.

Contribution

The paper presents NearestFit, a new profile-guided progress indicator that avoids linear assumptions and uses efficient algorithms to improve accuracy in MapReduce performance prediction.

Findings

NearestFit achieves high accuracy in diverse scenarios.

It operates with low space and time overheads.

It outperforms existing progress indicators like Hadoop's in accuracy.

Abstract

We tackle the problem of predicting the performance of MapReduce applications, designing accurate progress indicators that keep programmers informed on the percentage of completed computation time during the execution of a job. Through extensive experiments, we show that state-of-the-art progress indicators (including the one provided by Hadoop) can be seriously harmed by data skewness, load unbalancing, and straggling tasks. This is mainly due to their implicit assumption that the running time depends linearly on the input size. We thus design a novel profile-guided progress indicator, called NearestFit, that operates without the linear hypothesis assumption and exploits a careful combination of nearest neighbor regression and statistical curve fitting techniques. Our theoretical progress model requires fine-grained profile data, that can be very difficult to manage in practice. To overcome this issue, we resort to computing accurate approximations for some of the quantities used in our model through space- and time-efficient data streaming algorithms. We implemented NearestFit on top of Hadoop 2.6.0. An extensive empirical assessment over the Amazon EC2 platform on a variety of real-world benchmarks shows that NearestFit is practical w.r.t. space and time overheads and that its accuracy is generally very good, even in scenarios where competitors incur non-negligible errors and wide prediction fluctuations. Overall, NearestFit significantly improves the current state-of-art on progress analysis for MapReduce.