Scheduling MapReduce Jobs under Multi-Round Precedences

TL;DR

This paper develops approximation algorithms for scheduling multi-round MapReduce jobs on parallel processors, minimizing average weighted completion time, with strong theoretical guarantees and promising practical performance.

Contribution

It introduces LP-based approximation algorithms for multi-round MapReduce scheduling on identical and unrelated processors, improving guarantees especially for single-round cases.

Findings

LP-based algorithms achieve constant-factor approximation for identical processors.

Algorithms perform well in practice compared to heuristics and lower bounds.

Improved approximation guarantees for single-round MapReduce scheduling.

Abstract

We consider non-preemptive scheduling of MapReduce jobs with multiple tasks in the practical scenario where each job requires several map-reduce rounds. We seek to minimize the average weighted completion time and consider scheduling on identical and unrelated parallel processors. For identical processors, we present LP-based O(1)-approximation algorithms. For unrelated processors, the approximation ratio naturally depends on the maximum number of rounds of any job. Since the number of rounds per job in typical MapReduce algorithms is a small constant, our scheduling algorithms achieve a small approximation ratio in practice. For the single-round case, we substantially improve on previously best known approximation guarantees for both identical and unrelated processors. Moreover, we conduct an experimental analysis and compare the performance of our algorithms against a fast heuristic and a lower bound on the optimal solution, thus demonstrating their promising practical performance.